Broward Health Finds High Availability Disaster Recovery Capabilities

Presidio Expertise Helps Broward Health Integrate Technologies and Ensure Critical Application High-Availability

The Organization: Broward Health
Broward Health, one of the largest non-governmental public health systems in America, has provided healthcare services for more than 70 years. The organization is a nationally-recognized system offering world-class healthcare services in southern Florida at four medical centers and over 30 ambulatory healthcare facilities. Across the entire system, Broward Health provides more than 1,500 beds for patients who receive care from close to 2,000 physicians and an employee staff that numbers nearly 8,000.

The Challenge: Establish High Availability Disaster Recovery Capabilities
For Ronaldo Möntmann, the Director of Technology and Infrastructure for Broward Health, the IT initiatives his staff undertakes are driven largely by the five pillars of federal healthcare regulations: providing healthcare to every citizen, improving quality and patient outcomes, lowering healthcare costs, privacy, and safeguarding patient personal information. Meeting the requirements of these pillars presents a major financial challenge for the Broward Healthcare IT team.

As Doris Peek, the CIO for Broward Health, explains, “Keeping the IT strategy in alignment with the rapidly-changing requirements of healthcare reform while still being fiscally responsible and ever mindful of the tax paying citizens of Broward County, the Broward Health IT department has maintained high levels of technical competencies and a very sharp pencil (with a big eraser). Therefore, we have to find innovative and creative ways to meet the demand of changing regulations with fewer and fewer resources.”

“The HITECH Act put teeth into the HIPAA laws, so any healthcare organizations that don’t comply will now face severe penalties,” Möntmann said. “One of the key aspects we needed to address, and consistent with our overall strategic plan, involved establishing a business continuity/disaster recovery site to ensure our physicians can always access patient information when providing services. We also needed to make sure the site properly protected patient information.”

Broward Health previously relied on a single, centralized data center for all its servers and enterprise applications servicing thousands of computer users. The organization also relies on hosted application service providers for key critical systems, such as the Electronic Medical Record (EMR), patient registration & accounting, and scheduling systems.

“Our disaster recovery strategy previously involved backing up files to online storage systems and tapes each day and then storing the tapes at an off-site facility,” Möntmann said. “But if we faced a catastrophic disaster, it would likely have taken us days to completely restore all operations. To make sure we don’t hamper the process for delivering patient care, we needed to create a high-availability disaster recovery plan.”

Broward Health first worked with an analyst firm to estimate the cost of building a back-up data center and replicating the same infrastructure of the primary data center. The cost estimates were significant. As part of the healthcare system’s mission to keep costs down, Broward Health strives for fiscal responsibility. Möntmann thus looked for another way to provide high availability for critical data.

The internal Broward Health IT team*, with its vast technical experience, designed an alternative high-availability configuration that relies on a hosted data center as the second site. The plan called for replicating all critical systems to a Tier-4 data center in Miami to complement the primary site in Ft. Lauderdale.

“We devised an active-active data center configuration where we have two load-balanced data centers that split the production environment of our users and serve as failover sites for each other,” Möntmann explained. “We can also access our remote hosted EMR solution through both data centers.”

The active-active data center design required a wide range of technologies. These included Cisco networking gear, EMC storage, VMware server and desktop virtualization, Citrix desktop virtualization, Microsoft and Oracle databases, F5 load balancing/traffic management, and Infoblox directory-network- service high availability. “Given all of these high-end, complex technologies and the need to configure circuits between the two data centers, we knew we would need to depend heavily on a solution provider with deep expertise across the board,” Möntmann said.

After Broward Hospital created the data center designs—with validation provided by EMC and the aforementioned hardware/software vendors, Möntmann then issued a formal RFP to which Presidio responded. “We chose Presidio for multiple reasons,” Möntmann revealed. “In addition to demonstrating expertise in all the technologies we planned to deploy, Presidio showed us how we could leverage much of our existing data center hardware and software to keep our costs down while remaining true to the design. The other solution provider wanted to alter the design and add new components, but Presidio demonstrated its flexibility to take advantage of what we already planned to do.”

The Deployment: Project Coordination Across All Technologies Plays Key Role

Presidio coordinated the overall deployment of the new data center technologies while working closely with the internal Broward Health IT team and all of the hardware/software vendors. In addition to helping with new routers at each data center, Presidio also installed the Cisco Nexus 7000 router that connects the two data centers and configured the circuits between the data centers so users can easily failover from one data center to the other.

“A lot of resources had to be coordinated to make the deployment a success, and Presidio came through in a big way—in terms of both project management and systems integration,” said Möntmann. “Each vendor knew what their components could do, but Presidio knew how to make them all work well together. They served as the ‘glue’ of the project, making sure each component performed as designed. Presidio turned in one of the best coordinating efforts I have seen in my entire IT career.”

In addition to installing the entirely new data center infrastructure in Miami, Presidio system engineers also re-engineered the Ft. Lauderdale infrastructure to enable to the two data centers to work together in active-active mode. Broward Health now has full redundancy across the two data centers.

The Benefits: Internal IT Team Design Avoids Unnecessary Costs
Instead of building a completely new data center and replicating all the systems, Broward Health ended up spending significantly less to set up the hosted data center in Miami (with the most critical applications and infrastructure) and then reconfiguring the existing Ft. Lauderdale data center in high-availability mode. “The combination of newer technologies and Presidio’s ability to configure all the new technologies to work well together made a tremendous cost savings possible,” Möntmann said.

With the two data centers, clinical end users who function through one of the data centers automatically fail over to the other data center in the event their primary data center goes down. In addition to providing users with immediate access to on-premise applications, the failover site also provides immediate access to the hosted EMR and patient registration applications.

“Gaining immediate failover is critical, because in the event of a power outage that lasts just a few minutes, our physicians, clinicians and nurses need to resort to paper-driven documentation processes,” Möntmann said. “When the power comes back on, the support staff would then need to spend additional time to catch up on entering manually-recorded data into the system. Even a short delay can create havoc for us, but with automatic and immediate failover, this is no longer a major concern.”

In addition to the cost savings and the ability to keep end-user devices functioning even if one data center experiences a disaster, Broward Health has gained additional capabilities from the active- active data center architecture. “Now that we have load balancing across the two data centers on our critical systems, we can more easily address application and server performance issues that might arise as well as the times we need to perform system maintenance,” Möntmann said. “During any of these occurrences, we no longer have to take users offline—we can simply move them over to the other data center temporarily and then move them back after we work through the issue. This means less downtime for our clinical staff.”

During all the data center changes, end-users never experienced any downtime. “With Presidio’s help, we essentially performed ‘brain surgery’ on our network core without our users ever noticing anything was going on,” Möntmann concluded. “Thanks to the expertise and project management Presidio provided, we fundamentally changed our network core by stretching it across two data centers. This helps us ensure our doctors and nurses can always access the critical information they need to take care of our patients.”

* Broward Health extends special thanks to all of its IT management and staff and the following individuals for their special contributions to this project: Mitchell Aronson, Manager of Network and Telecommunications; Robert Glazer, Manager of Enterprise Virtualization Technologies; Shalom Maimon, Manager of Programming and Database Administration; James Orr, Manager of Onsite Support, PACS, and Storage Area Network; Sony Fenelon, Manager of Service Center; and Gisela Diaz, Senior Project Manager.