Enterprise Resiliency & Application Recovery Real Life Chronicle of Two IT Departments During Hurricane Sandy Superstorm Sandy, a hurricane combined with a powerful nor'easter, struck the East Coast in late October 2012. It was the deadliest and most destructive hurricane of 2012, as well as the second-costliest hurricane in U.S. history. This is the story of how productivity and preparedness paid-off for a prominent healthcare provider and
N
ew
EAST
AST UNIVE O C
ITY RS
CAL CEN I D
R TE
ME
East Coast university, who were able to weather the storm and maintain IT availability in spite of the odds.
ty Yo r k C i
A world class medical center located in New York City evaluated its IT landscape and began a two-year transformation. The initial strategy was designed to help refresh their technology, set priorities and determine the
A prominent East Coast university with a complex hybrid environment implements SunGard’s Managed Recovery Program (MRP) for comprehensive life cycle management of IT disaster recovery.
strategy. While the hospital has its own on-site datacenter, it also hosts production applications in SunGard’s Carlstadt, New Jersey facility.
During the first 5 months of onboarding, SunGard observes the university’s annual IT application testing and gathers information about its complex environment
data is a high priority for this healthcare customer- and while they use cloud technology internally, they decided not to go that route for backing up mission-critical data.
the university would send 40 people to their colocation facility to exercise its recovery plans losing an entire IT sprint cycle in the process.
According to a recent survey conducted by Aberdeen Group, the top 2 pressures leading to BCDR initiatives are:
Risk of Business Interruption
72%
Cost of Downtime
44%
According to the same survey, the average cost per hour of downtime was over:
$163,000
MAY 2012 NOAA predicts 2012 Atlantic hurricane season:
“A less active season compared to recent years”
- NOAA Administrator Jane Lubchenco, Ph.D.
OCTOBER 2012
TROPICAL DEPRESSION EIGHTEEN
OCTOBER 22, 2012
Senior vice president, vice dean and CIO of the healthcare provider is getting reports about the approaching storm and puts his IT team on high alert.
forms south of Kingston, Jamaica.
SunGard deploys
1/3
of its staff
OCTOBER 26, 2012
Scheduled to review its newly prepared IT recovery runbooks with the university on November 1, SunGard is prepared to deliver them when Sandy hits.
342
Hurricane Sandy takes aim at Eastern seaboard of U.S. and has shut down businesses from North Carolina to Maine and cut off power to 8 million homes in 10 states.
“Very scary sight. New York's gone black.”
Twitter Feed
customers put SunGard on alert
OCTOBER 29, 2012 knocked out emergency generators, the medical center is forced to shut down and evacuate and declares a disaster.
Across town, due to rising water levels, the university is also forced to shut down its generators and is unable to run its mission-critcal applications.
117
DISASTERS DECLARED
Nearly the same number of declarations the company had following 9/11
Sandy is causing severe flooding in many states along the mid-Altantic coast.
Bridges and subways close.
OCTOBER 30, 2012
The healthcare provider engages full disaster recovery plan to restore email, critical applications, and telecom. Key IT personnel are able to operate from SunGard’s Carlstadt, NJ facility. The university puts SunGard on alert and within hours declares a disaster.
76
evacuation shelters open around New York City.
National Guard troops are deployed.
SunGard advises the university to have their tapes delivered to its Carlstadt, NJ recovery facility. mission-critical applications that need recovering- one of them a data warehouse which they have never been able to successfully self-recover in any previous tests.
Mandatory evacuations are underway.
Built on high ground, SunGard’s Carlstadt, NJ datacenter is serving as an impromptu triage, command, and evacuation center by local government for the town of Moonachie, NJ. Referred to as “a beacon of light on a very dark night”- 100+ emergency responders and local residents
http://www.datacenterknowledge.com/archives/2012/11/06/ sungard-data-center-takes-in-flood-evacuees/
TAPES
Hurricane Sandy affects some 1.5 million businesses located across 19 counties in the tri-state region.
OCTOBER 31, 2012 medical facility is completed – service is gradually restored.
These businesses employ 9.3 million individuals.
All told, these businesses represent nearly 1,000 different industries.
1,500
9 of SunGard’s workgroup facilities are in use housing 1,500 customer employees.
University tapes are delivered to SunGard’s facility. Drawing upon previous months of information gathering and knowledge, the SunGard team works through the night to stand up the various platforms and operating systems needed to perform the recovery.
TAPES
5 Mobile Recovery Units are deployed including 2 that will eventually be sent to the medical facility to help run telecom and hospital infrastructure.
NOVEMBER 2012 NOVEMBER 1, 2012
Transportation Authority says that the destruction is the worst disaster in the 108-year history of New York City.
Healthcare provider recovery plan is in full execution.
The university's IT team arrives at Carlstadt facility with a skeleton crew of 8 people. Shocked to learn that their AIX, Windows, Linux, and backup environments are all up and running, the teams are ready to begin the data recovery process.
100% AIX Windows Linux
“SunGard didn’t just wait around for [us] to show up before beginning the recovery process.” - University VP for Information Technology and Chief IT Officer
Following the storm, as displaced customer employees stream in, SunGard’s Recovery teams build 15 additional workgroup recovery 750 more seats.
NOVEMBER 2, 2012
The Red Cross mobilizes 1,700 disaster workers from all over the country and activates 167 response vehicles.
The hospitals switch board and telephony are back online, public website is fully restored and functional, and critical application access restored for 600 users. By evening, all university systems are back together and the database team begins the restore.
An IDC study states, enterprises that performed IT recovery in-house lost an average of $4 million per disaster incident across a variety of business functions while enterprises that partnered with a managed recovery service provider lost an average of $1.1 million per incident. http://searchdisasterrecovery. techtarget.com/news/1359893/Out sourcing-disaster-recovery-servicesvs-in-house-disaster-recovery
NOVEMBER 4, 2012
IT fully restored at medical facility critical office complex.
By midnight, the university database is restored.
“The most important thing is to understand the business requirement—what are the critical systems. Then have a clear policy of data retention and make sure everyone understands what that policy is.” - Medical Center Vice President of Infrastructure and Engineering
AFTERMATH
Medical Center 43% of businesses that close after a natural disaster never reopen, and an additional 29% permanently close within two years. http://www.information-management.com/ news/ten-reasons-disaster-recovery-plansfail-10023118-1.html
Several months later, total economic impact of Hurricane Sandy estimated at $65M. http://www.usatoday.com/story/weather/2013/01/
In looking back over the recovery efforts following Sandy, Medical Center senior vice president, vice dean and CIO stresses the importance of team work.
“In a crisis, people work together. We learned we can do things faster than we thought we could.”
East Coast University
“Hurricane Sandy brought many gaps in our DR program to light, and we were fortunate to have SunGard as our recovery partner. Without them, we would not have been able to recover.” - University VP for Information Technology and Chief IT Officer
SunGard maintains 100% uptime through crisis. None of its facilities in the affected areas or its managed customer environments ever went down.
“Champions do not become champions when they win the event, but in the hours, weeks, months and years they spend preparing for it.” - Alan Armstrong To learn more about SunGard Availability Services, a leading provider of disaster recovery, cloud and managed hosting solutions services, visit us on the web at www.sungardas.com. To see how the SunGard Managed Recovery Program can help provide the level of assurance and resiliency to help your complex IT operations remain available when the unexpected happens, view this short demo. http://sungardas.com/KnowledgeCenter/VideosAndDemos/ Pages/disaster-recovery-program-still-on-the-todos-list.aspx