Security Metrics The Useful and Not-So-Useful Doug Copley CISO – Beaumont Health Chairman Emeritus – Michigan Healthcare Cybersecurity Council 17-DEC-2015
Why Measure At All? »
» » » » » »
Integral to a program’s governance Necessary to determine process effectiveness Can show resource gaps or shortages Manage service provider SLAs Supports continuous improvement Provides assurance to executives & the Board of Directors Provides basis for comparative benchmarking “You can’t manage what you can’t measure.” - W. Edwards Deming
Why Measure your InfoSec Program? Are our vulnerability management efforts effective? Are we educating our workforce? And changing behavior?
How resilient are our information systems? Is our risk posture improving or degrading?
Effective Security Metric Characteristics » » » » » » »
»
Be quantitative, or at least ordinal NOT be measured one time only Measure breadth of activities/involvement Be presented in context Measure against established baselines/targets Demonstrate control effectiveness Should induce action and/or influence behavior Can be used to make plan/strategy changes
What Should I Measure? »
Depends on Audience Board/C-level execs Management team CISO/Security Mgmt Info security staff
»
Dependent on maturity of the security program Low maturity – sheer counts may show progress Moderate maturity – % of target effectiveness High maturity – defect rate, duration of attack
Metrics Categories InfoSec Management Access Control Human Resource Security Risk Management Security Policy Security Organization Audit & Compliance Asset Management Threat Intelligence
Physical Security Communications Systems Acquisition, Development & Mgmt Incident Management Cryptography Business Resilience Operations Mgmt Supplier Relationships
Metrics Categories Access Control Human Resource Security Security Policy Security Organization Audit & Compliance Asset Management
Physical Security Communications Systems Acquisition, Development & Mgmt Incident Management Cryptography Business Resilience Operations Mgmt Supplier Relationships
What Should I Measure? »
Metrics must be meaningful to the audience
This is NOT the look you want from your audience:
What are you trying to say?
OR
They will be different by audience There is no one metric that will fit all of them
How to Present Metrics
Creating Meaningful Diagrams »
A good security metric can provide the following information: Current value – normal? Current value – satisfactory? Trend – Increase or decrease Trend – Improving or degrading?
»
Use colors and arrows to represent them in a compact and concise way Ideally, it will clearly show required actions
»
Do not make your audience search for meaning
»
Making Them Visual Âť
Make the slides visually appealing and informative to the target audience
Show Metrics in a Time Series Attempts to Access Malware 120,000
106,971
100,000
80,000
60,000
56,602 48,953
62,262
57,801
52,210 44,808
52,210 45,256
39,460
36,297
40,000
21,111 20,000
0 0
AUG 14
SEP 14
OCT 14 NOV 14 DEC 14
Jan-15
Feb-15
Mar-15
Apr-15
15-May
15-Jun
15-Jul
15-Aug
Improving or Worsening? Data Loss Prevention Security Events (n=193) 600 547
500 418 400 346 300
274
230
219 196
200 172
117
183
184 68
100 116
89
64
58
45
48
Mar-14 Only IronKey
Apr-14 SSN
May-14 US Credit Cards
Jun-14 Linear (Only IronKey)
Jul-14 Linear (SSN)
Aug-14 Linear (US Credit Cards)
Does Downward Slope Mean Good? IT Audit and Compliance Items - 24 Month Trend 140 127 121
120 120
113 108
105
106
107
106
105
114
114
122
124
112 104
103
100
80 65
61
61
67
66
New Closed
60
Open 38
40 20
19
11
0
21
20
0
0
2014-03
2014-05
2014-06
2014-07
2014-08
10
2014-02
0
02
2014-01
01
2013-11
01
2013-10
10
0
9
02
2013-09
10
2013-07
0
2013-06
1
3
2013-05
11
10
7
2013-03
0
2012-11
0
7
2012-10
3
4
2013-08
20
2013-02
2013-01
2012-09
2012-08
0
Pulling It All Together »
Do NOT make metrics slideware that is shared via email
»
Make sure you can explain them in person
Make sure the slides not only show the data, but explain the significance Help the reader understand what to look for Don’t assume they know what it means
Examples (focused on CISO/leadership)
Example Metric #1
Not So Not Good Good? So Good?
Improved Metric #1 % Workforce Completed Security Training 100% 90%
Target range
80% 70% 60% 50% 40% 30% 20%
Phishing campaign click rate
10% 0% Oct'14
Nov'14
Dec'14
Better
Jan'15
Feb'15
Mar'15
Apr'15
May'15
Jun'15
Jul'15
Aug'15
Sep'15
Key Message: Training is progressing well and we’re recognizing the intended results
Oct'15
Example Metric #2 # Open Vulnerabilities 250
200
150
100
50
0
Oct'14 Nov'14 Dec'14 Jan'15 Feb'15 Mar'15 Apr'15 May'15 Jun'15 Exploitable Sev 4-5
Jul'15
Aug'15 Sep'15 Oct'15
Non-Exploitable Sev 4-5
Not So Not Good Good? So Good?
Improved Metric #2 Vulnerability Management Efforts 100 90
% of Target Systems Scanned
80 70 60 50
# Critical, Exploitable Unpatched Vulnerabilities
40 30 20 10 0 Oct'14
Nov'14
Dec'14
Jan'15
Feb'15
Mar'15
>90 days
Better
Apr'15 46-90 days
May'15
Jun'15
Jul'15
Aug'15
Sep'15
Oct'15
0-45 days
Key Messages: 1. There has been steady progress, but we’re still not scanning all target systems 2. Patching/Remediation requires improvement. Unacceptable number open past 90 days.
Example Metric #3
USB
Not So Not Good Good? So Good?
Improved Metric #3 Percent DLP Events Resolved Within Target Timeframes 120
100
80
60
40
20
What’s the exposure? Like vulnerabilities, remediation timing – how long Are they open. Maybe % events closed within 5 days? Trending of critical events isrisk important identify A measure of aggregate would betoeven better. Broken business processes and their remediation What % of alerts are being closed monthly? 14
20
16
12
13
15
20
19
13
0 Oct'14 Nov'14 Dec'14
# Open
Jan'15
Feb'15 Mar'15 Apr'15 May'15 Jun'15
Critical <3 Days
Better
High <6 days
9 Jul'15
13
11
Aug'15 Sep'15
9 Oct'15
Med-Low <30 days
Key Messages: 1. Remediation on critical events is lacking 2. Need more focus on resolving open items Valuable addition could be reaction time (i.e. is it taking minutes/hours/days to escalate to an analyst?).
Example Metric #4 Managed PCs and Servers 5000 4500 4000 3500 3000 2500 2000 1500 1000 500 0 Oct'14 Nov'14 Dec'14 Jan'15 Feb'15 Mar'15 Apr'15 May'15 Jun'15
# Managed PCs
Jul'15
Aug'15 Sep'15 Oct'15
# Managed Servers
Not So Not Good Good? So Good?
Improved Metric #4 Managed PCs and Servers 6000
5000
4000
3000
2000
1000
0 Oct'14
Nov'14
# Managed PCs
Better
Dec'14
Jan'15
Feb'15
Unmanaged PCs
Mar'15
Apr'15
May'15
# Managed Servers
Jun'15
Jul'15
PCs on Gold Build
Aug'15
Sep'15
Oct'15
Servers on Gold Build
Key Messages: 1. Still not actively managing all PCs â&#x20AC;&#x201C; requires attention 2. Too many systems not configured to company standards
Example Metric #5 Weekly URL Filtering Porn Access Attempts
Minutes on Social Media
Ann Smith Joe Box Cliff Jones John Doe Jane Clemens Buford Horen Henry Isaac Jeff King Sue Beggs Nick Molar Kyle Bicuspid Iggy Incisor
Bob Jan Fred Feb Mary Mar Alex Apr Max May Jack Jun Justin Jul Andy Aug Sally Sep Oliver Oct Ned Nov Doug Dec 0
100
200
300
400
500
600
700
800
900
0
100
200
300
400
500
600
Not So Not Good Good? So Good?
700
800
Improved Metric #5 2500
Attempts to Access a Malicious Site
2000
1500
1000
500
0 Oct'14 Nov'14 Dec'14 Jan'15 Feb'15 Mar'15 Apr'15 May'15 Jun'15
Better
Jul'15 Aug'15 Sep'15 Oct'15
Key Messages: 1. Good progress, but still consistently see indicators of compromise 2. Need to strengthen anti-malware controls
Example Metric #6 Identified Risk Items 40 36 35 30 25
22
20 15 10
6 5
2
0 External Assessment
External Audit
Internal Assessment
Internal Audit
Not So Not Good Good? So Good?
Improved Metric #6 Risk Register Activity 80
Total Identified Risks
70 60
Risks with Action Plans 50 40
Aggregate Relative Risk
30 20 10 2
0
1
2
0
0
2
4
7 3
1
2
3
0 Oct'14 Nov'14 Dec'14
Better
Jan'15
Feb'15 Mar'15 Apr'15 May'15 Jun'15
Jul'15
Aug'15 Sep'15
Oct'15
Key Messages: 1. Risk posture moving in the right direction 2. Should add temporary staffing to inject a sizeable reduction in overall risk posture
Example Metric #7 40
37
35
32
37
30
30 25
18
20
15 10
14
17
12 8
7
Tier 0
Tier 1
9
7
5
0
Total Critical Apps
Tier 2
App Plans Updated 2014/2015
Tier 3 Apps Not Tested in 2014/2015
Good?NotNot BadSo Good?
Improved Metric #7 Resiliency / Preparedness 40 35 30 25 20 15 10 5 0 Tier 0
Tier 1 Critical Systems
Better
Tier 2 Plans Updated in 2015
Tier 3 Plans Tested in 2015
Key Messages: 1. Need significant improvement in system disaster recovery planning and testing
Executive Dashboard » »
Intended to convey a high-level status of the program to C-level executives and the Board Security Dashboard should convey: Status of regulatory compliance Capability, Maturity and Implementation level of program elements Key areas of information risk to the organization Current initiatives and future state posture External ties and intelligence information
»
Must answer the question “Is our Information Security program effective?”
Example Program Scorecard InfoSec Management Program (IS)
Human Resources Security (HR)
Access Control (AC) 22
4
3
Physical Security (PS)
Communications Security (CS)
Cryptography (CR)
5
2
11
Systems Acquisition, Development, and Maintenance (SD)
Incident Management (IM)
2
Modified ISO 27001 Scorecard
1
0
Operations Management (OM)
Business Continuity (BC) 1
Asset Management (AM)
Compliance (CO) 3
0
4
2
Organization of Information Security (OI)
Security Policy (SP)
Risk Management (RM)
7
Supplier Relationships (SR) 0
HIPAA Compliance
Sample Dashboard #1
34
Sample Dashboard #2
35
Sample Dashboard #3
Sample Dashboard #4
Questions?
Thank You! Doug.Copley@Beaumont.org Doug.Copley@mihcc.org