CSTC2018: Disaster Recovery_Phil Miller

Disaster Preparedness & Recovery 2018

Phil Miller President – Integrity IT

Background in DRP (Disaster Recovery Planning) • • • • •

Prior to starting Integrity IT, worked for 2 HealthCare Organizations - Catholic Health Initiatives, and before that Columbia HCA Columbia HCA – Developed IT Systems Technical Recovery Plans for all clinical systems at multiple sites CHI - IT Analyst responsible for leading Emergency Ops & Disaster Recovery Planning team CHI - Y2K Project Lead (Prep for anticipated year 2000 impact on all computer systems) Lead Team on DR and Contingency Planning at Integrity IT

What does DRP have to do with Cybersecurity? • Backups (a key element of your DRP) are often the last line of defense in Ransomware Attacks • Over 80% of the time, data is restored in Ransomware scenarios

Cyber Threat Sources Greatly ^ Risk of Disaster • Ransomware, Hacking & Social Engineering are all capable of producing disaster scenarios • Nearly 20% of organizations report having been victims of Ransomware (even though this study was of larger organizations, I believe it to be true of SMB’s as well) • Actual number will be higher due to reluctance to report • What other Disaster Threats are that likely to happen?

Are You Ready?

Is your organization ready to respond to a Disaster Scenario?

Hurricane Florence – Sept 2018 3 or 4 out of 5 of our organizations are not prepared to respond to a Disaster

Why Not? • • • • • •

I just haven’t gotten around to it We don’t have many natural disasters around here It will never happen to us I’ll take my chances Too Costly I am fairly sure these are words we will all regret when faced with a real disaster

My Goals for this presentation 1. Convince you that having a well thought out and tested Disaster Recovery plan (DRP) is essential to your organization 2. Provide you with a framework you can use to develop an effective DRP

Where to Begin – Standards for DRP • National Institute for Standards in Technology (NIST) > NIST SP 80034, R1 • Standards designed for Government Agencies, but most are very applicable for the public sector as well • International Standards Organization (ISO) > ISO/IEC 24762:2008 • British Standards Institution (BS) > BS 25777:2008

Where does Disaster Recovery Fit in the Bigger Picture of EOP

NIST positions Disaster Recovery Planning as a component of EOP Emergency Operations Planning >>>>>

NIST - EMERGENCY OPERATION PLANS Business Continuity Plan (BCP) Continuity of Operations (COOP) Plan Crisis Communications Plan Critical Infrastructure Protection (CIP) Plan Cyber Incident Response Plan Disaster Recovery Plan (DRP) ^^ Information System Contingency ^^ Plan (ISCP) Occupant Emergency Plan (OEP)

NIST Emergency Operations Plans 1 PLAN

PURPOSE

SCOPE

Business Continuity Plan (BCP)

Provides procedures for sustaining business operations while recovering from a significant disruption.

Addresses business processes at a lower or expanded level from COOP mission essential functions

Continuity of Operations (COOP) Plan

Provides procedures and guidance to sustain an organization’s mission essential functions at an alternate site for up to 30 days; mandated by federal directives.

Addresses the mission essential functions; facility- based plan; information systems are addressed based only on their support to the mission essential functions.

PLAN RELATIONSHIP Mission/business process focused plan that may be activated in coordination with a COOP plan to sustain non mission essential functions .

Mission essential function focused plan that may also activate several business unit level BCPs, ISCPs, or DRPs, as appropriate.

NIST Emergency Operations Plans 2 PLAN

PURPOSE SCOPE Provides procedures for disseminating internal and Addresses communications with Crisis Communications Plan external communications; means personnel and the public; not to provide critical status information system focused. information and control rumors.

Provides policies and procedures for protection of Addresses critical infrastructure national critical infrastructure components that are supported Critical Infrastructure Protection components, as defined in the (CIP) Plan or operated by an agency or National Infrastructure Protection organization.

Plan.

PLAN RELATIONSHIP Incident-based plan often activated with a COOP or BCP, but may be used alone during a public exposure event.

Risk management plan that supports COOP plans for organizations with CI/KR assets.

NIST Emergency Operations Plans 3

PLAN Cyber Incident Response Plan

Disaster Recovery Plan (DRP)

PURPOSE Provides procedures for mitigating and correcting a system cyber attack, such as a virus, worm, or Trojan horse.

Provides procedures for relocating information systems operations to an alternate location.

SCOPE Addresses mitigation and isolation of affected systems, cleanup, and minimizing loss of information.

Activated after major system disruptions with long-term effects.

PLAN RELATIONSHIP Information system focused plan that may activate an ISCP or DRP, depending on the extent of the attack.

Information system focused plan that activates one or more ISCPs for recovery of individual systems..

NIST Emergency Operations Plans 4 PLAN Information System Contingency Plan (ISCP)

Occupant Emergency Plan (OEP)

PURPOSE

SCOPE

Provides procedures and capabilities for recovering an information system.

Location-independent plan that focuses on the procedures needed to recovery a system at the current or an alternate location.

Provides coordinated procedures Focuses on personnel and for minimizing loss of life or property particular to the specific injury and protecting property facility; not business process or damage in response to a information system-based. physical threat.

PLAN RELATIONSHIP Information system focused plan that may be activated independent from other plans or as part of a larger recovery effort coordinated with a DRP, COOP, and/or BCP. Incident-based plan that is initiated immediately after an event, preceding a COOP or DRP activation.

Today’s Focus PLAN Disaster Recovery Plan (DRP)

Information System Contingency Plan (ISCP)

PURPOSE Provides procedures for relocating information systems operations to an alternate location.

Provides procedures and capabilities for recovering an information system.

SCOPE Activated after major system disruptions with long-term effects.

Location-independent plan that focuses on the procedures needed to recovery a system at the current or an alternate location.

PLAN RELATIONSHIP Information system focused plan that activates one or more ISCPs for recovery of individual systems..

Information system focused plan that may be activated independent from other plans or as part of a larger recovery effort coordinated with a DRP, COOP, and/or BCP.

Disaster Recovery Plan (DRP) – what’s unique •

Applies to major, usually physical disruptions to service that deny access to the primary facility infrastructure for an extended period

•

IT system-focused plan designed to restore operability of the target systems, applications, and computer facility infrastructure at an alternate site after an emergency

Disaster Recovery Plan (DRP) •

May be supported by multiple information system contingency plans to address recovery of impacted individual systems once the alternate facility has been established.

•

May support a BCP or COOP plan by recovering supporting systems for mission/business processes or mission essential functions at an alternate location

•

Only addresses information system disruptions that require relocation

NIST - 7 Key Steps • • • • • • •

Develop the contingency policy objective statement Conduct a Business Impact Analysis (BIA) Identify preventive controls Develop recovery strategies Create the contingency plan Conduct testing and training Review and maintenance

Disaster Recovery Plan – Small/Med Business Reality •

The NIST guidelines are designed for government organizations. While very helpful point of reference, some of them are inapplicable.

•

Most SMB’s will not have separate EOPs for each area of focus

•

It’s OK to combine into a single plan as long as you cover all the areas

Contingency Policy Objectives Statement • The contingency planning policy statement should define the organization’s overall contingency objectives and establish the organizational framework and responsibilities for system contingency planning.

Contingency Policy Objectives Statement • Roles and responsibilities • Scope as applies to common platform types and organization functions (i.e., telecommunications • Legal, media relations subject to contingency planning • Resource requirements • Training requirements • Exercise and testing schedules • Plan maintenance schedule • Minimum frequency of backups and storage of backup media

Business Impact Analysis Potential Disaster

Probability: 1=Very Low, 5 = Very High Impact 1=Minor Annoyance, 5=Total Destruction

Probability Rating

Impact Rating

Electrical power failure

5

2

Electrical storms / Lightening Strike

3

1-4

Fire

3

1-5

Tornado / Earthquake Loss of communications network services

2 5

1-5 2

Flood / Water

2

1-4

Act of sabotage / Including System Wide Virus Malware Outbreak Act of terrorism

4

1-4

1

1-5

Brief Description Of Impact on Operations & Remedial Actions

Monitored UPS Systems with auto standby generator that is tested monthly & monitored 24/7. Natural Gas Generator to power infrastructure.

Potential Surge Damage to equipment Fire suppression system (sprinklers directly above server infrastructure) would likely destroy the core network and server infrastructure. Fire and smoke detectors on all floors may prevent spread to Server Room Area Two diversely routed Internet trunks (two ISPs). No WAN/MPLS redundancy, voice network resilience. Cellular Comm. Backup Building where equipment is located is not in an area where flood is likely. However, is in basement.

Potential Widespread Server / Network Failure.

Business Systems Classification Classification

Mission Critical

Critical

Non-Critical

Definition Restoration of business functions must be completed within 3 Days of an emergency or a declared disaster Restoration of existing business functions can be delayed up to 5 days following an emergency or a declared disaster Restoration of existing business functions can be delayed until restoration becomes practical following an emergency or a declared disaster

RTO

48 – 72 Hours

72 – 120 Hours

120+ Hours

Restoration Objectives Application Domain Controller Electronic Medical Record Practice Management Pharmacy Citrix Environment Email Financial System Fund Raising System Document Management Print Servers* File Sharing*

Technical Recovery

Classification RTO (Hrs) RPO (Hrs) Date & Test Results Mission Critical

24-48

24

Mission Critical

24 - 48

1

Mission Critical Mission Critical Mission Critical Critical Critical Critical

24 24 24 48 48 48

-

48 48 48 72 72 72

4 1 24 8 1 24

Critical

48 - 72

4

Non-Critical Non-Critical

72 - 120 72 – 120

72 4

* Services in support of Mission Critical and Critical Applications will be restored as needed

Successful Successful With Minor Issues Unsuccessful

Backup Schedule Server Name DC1 Allscripts DB Allscripts EMR Allscripts PM McKesson Pharmacy Citrix Environment MS Exchange Great Plains Raisers Edge Alfresco Print1 FS1

Backup Schedule Local Hourly Hourly Hourly Every 4 Hours Hourly Daily Every 8 Hours Hourly Daily Every 4 Hours Every 3 Days Every 4 Hours

Offsite Replication Cloud Continuous Continuous Continuous Continuous Continuous Continuous Continuous Continuous Continuous Continuous Continuous Continuous

Preventative Controls • Backup all information in accordance with the Backup Policy • Test integrity of backups periodically - Boot from image, recover to login screen • Protect by uninterruptible power supplies (UPS) all servers and other critical equipment from damage in the event of an electrical outage • Training in disaster preparation and recovery, and knowledge of responsibilities in the event of a disaster (share Emergency Operations Plan with employees)

System Information & Primary Contacts Workarounds / Primary Contact Alternatives List any other method that would allow your List the primary Explain what would happen List name of system or Describe the function or business to continue if the system was contact for the application purpose of the system. to access the data or listed system. unavailable. use the system during a disruption.

James Houston in Use downtime IT / Integrity IT PM & EMR Database Unable to access patient procedures & paper Server Name 859 253-4284 & server records and rely on patient EMR Vendor self report for hx data Helpdesk System / Application

Function

Consequences of Disruption

Server Name

Monica Shane in PM & EMR Application Remote Site Users and Use downtime Informatics / Server for remote users / Specialty Clinics staff would procedures & paper Integrity IT 859 (Citrix / Remote Desktop not be able to access and rely on patient 253-4284 & EMR Servers) patient records self report for hx data Vendor Helpdesk

Server Name

Domain Controller, Files Server, Email Server

Unable to login to computers on domain, email would not work

Integrity IT 859 253-4284

Disaster Recovery Team Roles & Responsibilities Name

Role/Title

Work Phone Number

Mobile/Home Phone Number/s

Client IT Disaster Recover Lead

Title

Work Number

Cell Phone

(859) 253-4284

(859) 489-6820

(859) 899-1924

(859)Â 539-6974

(859) 899-1916

(859) 983-8838

?

Management Finance IT Vendor Backups Victor Kalinyuk Administrator IT Vendor, Backups Resource Maximo Bredfeldt & vCIO Phil Miller

IT Vendor President – manage additional IT Resources needed

Disaster Plan Trigger Events The following Disaster Plan Event Triggers are only applicable if the outage / local recovery time is expected to exceed 48 hours.

• • • • •

Total loss of all LAN/WAN network communications Total loss of power Flooding of the Data Processing Facilities Destruction of the Data Processing Facilities Security Breach or Sabotage that has compromised the data integrity and/or security of the network / infrastructure

Disaster Recovery Team Roles & Responsibilities Disaster Recovery Lead(s) • Manage all processes of the disaster recovery plan Disaster Recovery Team • Support the DR lead • Communicate the disaster to workforce members

Disaster Recovery Team Roles & Responsibilities IT Department • Handle all IT related processes of the DR plan • If multiple servers are impacted, the team will prioritize the recovery of servers in the manner and order that has the least business impact • Install and implement any tools, hardware, and systems required for recovery

Disaster Recovery Team Roles & Responsibilities Management • Ensure that the Disaster Recovery Team Lead is held accountable for his/her role • Assist the Disaster Recovery Team Lead in his/her role as required

Disaster Recovery Team Roles & Responsibilities Finance Department •

Ensure there is sufficient cash on-hand or accessible to deal with small-scale expenses caused by the disaster

•

Ensure there is sufficient credit available or accessible to deal with large-scale expenses caused by the disaster. These can include paying for new equipment, repairs for primary facilities, etc.

•

Review and approve Disaster Teams’ finances and spending

I’m told we’ve got good backups – isn’t that enough?

No, it isn’t – you have to have: • • • • • • • • •

An alternate network that your users can connect to Methods to route inbound traffic to your new data location Secure Remote Access Methods (VPN/Hosted Desktops/Other?)

Sufficient Bandwidth Access Control & Data Security An execution platform to run your systems on Plan for licensing server/s Methods for recovering your systems & restoring system functionality Methods for backing up data during the disaster period, etc …………

The necessary spare equipment or a service agreement with a provider to be able to run your systems

Disaster Recovery Steps • Determine the extent of the damage and whether additional equipment/supplies are needed.

• Determine how long it will be before service can be restored, and notify required personnel.

• Set the DRP into motion after the Disaster Recovery Lead has declared a disaster.

• Keep a log for the system outage/s, failure/s (capture dates and times), and data loss to critical systems. The log should also include a chronological diary of the steps taken during the disaster response period through closure and restoration to normal production processing.

Disaster Recovery Steps Replace hardware as necessary to restore service.

Retrieve and upload backup files if necessary to restore service.

Ensure that restoration procedures are followed.

Verify the integrity of data restored and the ability for workforce members to access.

• Coordinate activities to ensure that the most critical tasks are being supported as needed.

• Keep administration, management, staff, and others informed of the status of the emergency mode operations.

• • • •

Training & Testing • Train your staff on you contingency & disaster plans • Establish a schedule for testing Technical Restoration • Periodically conduct table top exercises involving all areas of your business • Address the areas where you were not sufficiently prepared

Review & Maintenance • • • •

Review your contingency and DR plans annually and whenever there are major organizational changes Make updates to systems, primary contacts, prioritization, etc Adjust the plans as necessary Be sure and include IT and ensure that the Technical Recovery plans are reviewed and updated

NIST - 7 Key Steps • • • • • • •

Develop the contingency policy objective statement Conduct a Business Impact Analysis (BIA) Identify preventive controls Develop recovery strategies Create the contingency plan Conduct testing and training Review and maintenance

https://www.integrityky.com/blog https://www.integrityky.com/victor-asks-are-you-sure-you-have-a-backup-of-your-data/

integrity

Be a Champion in October!

National Cyber Security Alliance and our support of their Stop. Think. Connect. initiative staysafeonline.org/ncsam dhs.gov/national-cyber-security-awareness-month

https://www.integrityky.com/ Free Dark Web Scan

https://www.integrityky.com/free-dark-web-scan/

Free Backup Consult

https://www.integrityky.com/free-backup-consult/

Free vCIO Consult

https://www.integrityky.com/free-vcio-consult/

Free Cyber Security Tips

https://www.integrityky.com/my-security-tips/

Free SRA Consult

https://www.integrityky.com/security-risk-assessment-andanalysis/

Free Hacker Report

https://www.integrityky.com/top-10-ways-hackers/