Getting to a Service Catalog Joel A. Manfredo
June 14, 2012
1
Agenda Background
Timeline
Phases 1 - 5
Results
Next Steps
Conclusions and Lessons Learned
County of Orange
$5.6 Billion Budget 7 Programs 32 Agencies / Departments
CEO IT Services Services are delivered through an Internal Service Fund “Running a Service Delivery Business�
Pricing a Service What does internet access cost? Hardware: •Redundant Firewalls •Redundant Routers •Redundant Core Switches •IPS / IDS Hardware •Monitoring (Orion) Servers •Content Filtering Appliances •Content Filtering Report Servers •DNS Servers
Software: •Firewall Security Software licensing •Content Filtering Software licensing & Report Services •Orion Monitoring Software licensing •DNS Software (Unix and Wintel)
Data Center Facilities Contracts: •Redundant Internet Service Providers: 1.TW Telecom 2.VPLS (this is a fail-over service) •IPS / IDS monitoring Service •Redundant WAN Circuit Contracts •Redundant Firewall Maintenance Contract (HW and SW) •Redundant Routers Maintenance Contract HW and SW)
•Redundant Core Switches Maintenance Contract HW and SW) •Content Filtering Maintenance Contract •County ARIN IP and BGP AS Services
Depreciation: •Firewalls - Hardware and software Depreciation associated •Routers - Hardware and software Depreciation associated •Core Switches - Hardware and software Depreciation associated •Monitoring - Hardware and software Depreciation associated •Content Filtering - Hardware and software Depreciation associated
Other CEOIT Services: •Contracts & Procurement Management •Resource Management •Financial and Cost, Budget Analysis •Billing Processing
Labor: •Network Engineering Labor •Security Engineering Labor •Availability Management Analysis •Capacity Management Analysis •Incident Management (Service Desk) •Change Management •Management
Pricing a Service What does internet access cost? Hardware: •Redundant Firewalls •Redundant Routers •Redundant Core Switches •IPS / IDS Hardware •Monitoring (Orion) Servers •Content Filtering Appliances •Content Filtering Report Servers •DNS Servers
Software: •Firewall Security Software licensing •Content Filtering Software licensing & Report Services •Orion Monitoring Software licensing •DNS Software (Unix and Wintel)
Data Center Facilities Contracts: •Redundant Internet Service Providers: 1.TW Telecom 2.VPLS (this is a fail-over service) •IPS / IDS monitoring Service •Redundant WAN Circuit Contracts •Redundant Firewall Maintenance Contract (HW and SW) •Redundant Routers Maintenance Contract HW and SW)
•Redundant Core Switches Maintenance Contract HW and SW) •Content Filtering Maintenance Contract •County ARIN IP and BGP AS Services
Depreciation: •Firewalls - Hardware and software Depreciation associated •Routers - Hardware and software Depreciation associated •Core Switches - Hardware and software Depreciation associated •Monitoring - Hardware and software Depreciation associated •Content Filtering - Hardware and software Depreciation associated
Other CEOIT Services: •Contracts & Procurement Management •Resource Management •Financial and Cost, Budget Analysis •Billing Processing
Labor: •Network Engineering Labor •Security Engineering Labor •Availability Management Analysis •Capacity Management Analysis •Incident Management (Service Desk) •Change Management •Management
The Story… Operational Transformation in the Face of Financial Distress Environment and Results: Organization: 1. 2. 3. 3. 4. 5.
A culture built on Heroics Bad relationship w/ vendor (expiring K) Siloed organization Assigning blame Resistance to change No priorities
How: Combination: 1. 2. 3. 4. 5.
Leadership Soft skills introduction Sustained “drumbeat” The persuader ITSM principles
Method: Results: 1. Moved to top 10% against benchmark 2. Reduced Budget 27% 3. Increased customer satisfaction 16% 4. Reduced SLA exceptions by 77% 5. Reduced incidents by 38% 6. Improved incident resolution from 5 days to 2 days
1. 2. 3. 4.
Plan Implement Absorb Measure
“Be transparent, or be gone”
Phases of the Journey Method - Plan, Implement, Absorb, Measure Phase 1:
Phase 2:
Phase 3: Phase 4: Phase 5:
ITSM Training DISC (Soft Skills) Training Capacity Management Availability Management Resource Planning Project Prioritization Network & Platform Services Reorganization Service Design Package Monthly MAP meetings
April 2009 May 2009 May 2009 June 2009 June 2009 June 2009 October 2009 November 2009 March 2010
Marketing Brochure (Mini Service Catalog - BRAND) Service Desk re-implementation Network Operations Center ITSM Full Suite (BMC – “AMP”)
October 2010 December 2010 January 2011 March 2011
Re-benchmark AIMM Service Catalog – Version 1 Customer Ranking of Services Service Catalog – Version 2 ITSM Reorganization Service Level Management Further Automate Capacity & Availability
July 2011 October 2011 January 2012 February 2012 Future Future Future
Driving Continuous Improvements
Phase 1
2009 Jan
Feb
Mar Apr
May Jun
Jul
2010 Aug
Sep
Oct
Nov
Dec
NPS Reorganized:
Phase 2
Jan
Feb
Mar Apr
“One Team”
Jul
Aug
Sep
Oct
Incident Mgt Process
SDP Process Capacity Reporting
May Jun
Nov
Dec
Customer Survey (Jul – Dec 09)
-Plan & Design -Build & Transition -Operate & Maintain
Training
Chaos
Phase 3
ITSM Integrated Suite Service Desk
Customer Survey (Jan – Jun 10)
“A journey of a thousand miles begins with a single step”
CTO Portal Certifications: 95
IT Service Management Path to Maturity Availability Reporting ACS Program Executive hired
Ticket Aging Process
AIMM (HP) Report
46
-Technology & Architecture -Mgt Tools & Process Maturity -Culture & IT Staff Maturity -Demand, Supply & IT Governance
Resource Planning
Marketing Brochure “Agency Success is AIMM (HP) Report 34
Communication Team Formed
Performance Audit Report County Connection: OCDC Project Prioritization
Service request turnaround improvements (100%) Incidents – from over 5 days to 1
(DISC)
Not a single failed non-standard RFC Not a single Service Level Breach
Data Center Tour Tech Fest
Our Mission”
Phase 4
Driving Continuous Improvements 2011
Jan
Feb
Mar Apr
May Jun
Jul
Method 2012 Aug
ITSM Integrated Suite Launch – FULL SUITE (CMDB, Change, Release…)
Sep
Oct
Nov
Service Catalog v 1
Dec
Jan
Feb
Mar Apr
May Jun
Jul
Aug
Service Catalog v 2
Customer Survey (Jul – Dec 10)
IT Service Management Path to Maturity AIMM (HP) Report 10 Network Operations Center Launched
Sep
ITSM based Reorganization
Application Monitoring
MSB Planning Success
Phase 5
Service Level Management Customer Grading by Service Virtual Server Farm Replacement
Oct
Nov
Dec
Phase 1
DISC Assessment Phase 1: Dominance, Influence, Steadiness, Compliance
Management Action Plan Results through Present 400 350 300 250 200 150 100 50 0 Nov Mar April May June Jul
Aug Sep Oct Nov Dec Jan Feb Mar Apr May
2010
Cumulative
2011
Monthly Accomplishments
Jul
Aug Sep
Capacity & Availability Reporting Phase 1 Capacity Report Capacity Report Table of Contents Executive Summary............................................................................................................ 4 Dashboard Summary........................................................................................................... 4 Risks.................................................................................................................................... 4 Warnings ............................................................................................................................. 4 Scope................................................................................................................................... 5 Capacity Measurements ...................................................................................................... 5 Network .......................................................................................................................... 5 County Wide Area Network ....................................................................................... 5 Time Warner Internet.................................................................................................. 6 Security ........................................................................................................................... 7 Storage ............................................................................................................................ 8 Tier-1........................................................................................................................... 8 Tier-2........................................................................................................................... 8 E-Mail ........................................................................................................................... 10 Virtual Environment ..................................................................................................... 13 Analysis Additional Capacity ................................................................................... 14 Conclusion ................................................................................................................ 14 Recommendation ...................................................................................................... 14 Power5 Systems............................................................................................................ 15 Mainframe..................................................................................................................... 18 Power ............................................................................................................................ 19 Data Center Space......................................................................................................... 22 Floor Space ............................................................................................................... 22 Office Space.............................................................................................................. 24 Telephone Services ....................................................................................................... 25 APPENDIX – A – Report Data Source and Contributors ................................................ 26 APPENDIX – B – Virtual Environment Cluster Information .......................................... 27
Availability Report
Capacity & Availability Reporting Phase 1
Resource Planning Key Metrics
3 Week Summary (by organization unit): Values Row Labels
ETC Hours
3 Week FTE
Enterprise Infrastructure Srvcs
13.92
0.12
Networks and Platform Services
523.87
4.37
Security & Bus Continuity Oper
15.50
0.13
553.28
4.61
Grand Total
Resource Planning Key Metrics 3 Week Summary (by project):
Row Labels Disaster Recovery Implementation (FY11-12)
Sum of ETC 336.00
Enterprise Secure File Transfer Migration
33.20
ePages Replacement
22.00
Exchange 2010 Production Build out HR & FS Access Definition Project Schedule Secure FTP Infrastructure Upgrade
Grand Total
112.00 1.50 48.58
553.28
Projects with late tasks: Sum of ETC Row Labels
Grand Total
ePages Replacement
10.00
Grand Total
10.00
Phase 2
Driving Continuous Improvements 2009 Jan
Feb
Mar Apr
May Jun
Jul
2010 Aug
Sep
Oct
Nov
Dec
NPS Reorganized:
Phase 2
Feb
Mar Apr
Chaos
“One Team”
Jul
Aug
Sep
Oct
Incident Mgt Process
SDP Process Capacity Reporting
May Jun
Nov
Dec
Customer Survey (Jul – Dec 09)
-Plan & Design -Build & Transition -Operate & Maintain
Training
Jan
ITSM Integrated Suite Service Desk
Customer Survey (Jan – Jun 10)
“A journey of a thousand miles begins with a single step”
CTO Portal Certifications: 95
IT Service Management Path to Maturity Availability Reporting ACS Program Executive hired
Ticket Aging Process
AIMM (HP) Report
46
-Technology & Architecture -Mgt Tools & Process Maturity -Culture & IT Staff Maturity -Demand, Supply & IT Governance
Resource Planning
Marketing Brochure “Agency Success is AIMM (HP) Report 34
Communication Team Formed
Performance Audit Report County Connection: OCDC Project Prioritization
Service request turnaround improvements (100%) Incidents – from over 5 days to 1
(DISC)
Not a single failed non-standard RFC Not a single Service Level Breach
Data Center Tour Tech Fest
Our Mission”
ITSM Organizational Design Phase 2: Reorganization of NPS
ASG
IRM
Process
Security
NPS
OPS
Plan & Design
Build & Transition
Operate & Maintain
TSD
PMO
BA
Design Validation / Review and Acceptance Procurements Configuration and Installation Update and Audit of Asset Lists/CMDB RFC Execution and Coordination/Communication Rollout Plan Backout Plan Risk Assessment for RFC Ops Training and Handoff QA and Validation of Systems Feedback to Design Team Approval of Rollout from Ops Approval of Rollout from customer Approval of Rollout from Design group Support and monitoring during cool down/warranty period
Operate and Maintain
Overall Architecture and Technologies / Standards* Capacity Requirements Availability Requirements Training Requirements POC requirements Design Diagrams and Requirements MOUs/SLRs Success Criteria TCO Calculations Alternatives Backup requirements Process / Procedure requirements Monitoring requirements Access Requirements Standard Requests Engineering and Ops review and acceptance Continuous improvement or scalability plans Approval of the Design from Peer Groups Approval of the design from Chief Architect Approval of the design from Customer
Build and Transition
Plan and Design
ITSM Organizational Design System acceptance Registering application in ops (contact information, service levels, monitoring and backup information, new or changed procedures, Access levels) Update procedures System lockdown QA and Validation of Monitoring, Backups and Access Assist in Negotiating SLRs/SLAs Freeze and Warranty Period Management
SDR SDP – Issues identified through the process
Phase 3
Driving Continuous Improvements 2009 Jan
Feb
Mar Apr
May Jun
Jul
2010 Aug
Sep
Oct
Nov
Dec
NPS Reorganized:
Jan
Feb
Mar Apr
“One Team”
Jul
Aug
Sep
Oct
Incident Mgt Process
SDP Process Capacity Reporting
May Jun
Nov
Dec
Customer Survey (Jul – Dec 09)
-Plan & Design -Build & Transition -Operate & Maintain
Training
Chaos
Phase 3
ITSM Integrated Suite – Service Desk
Customer Survey (Jan – Jun 10)
“A journey of a thousand miles begins with a single step”
CTO Portal Certifications: 95
IT Service Management Path to Maturity Availability Reporting ACS Program Executive hired
Ticket Aging Process
AIMM (HP) Report
46
-Technology & Architecture -Mgt Tools & Process Maturity -Culture & IT Staff Maturity -Demand, Supply & IT Governance
Resource Planning
Marketing Brochure “Agency Success is AIMM (HP) Report 34
Communication Team Formed
Performance Audit Report County Connection: OCDC Project Prioritization
Service request turnaround improvements (100%) Incidents – from over 5 days to 1
(DISC)
Not a single failed non-standard RFC Not a single Service Level Breach
Data Center Tour Tech Fest
Our Mission”
It’s working... Phase 3: “Mojo”
- Marshall Goldsmith
November 2009
Not a single Service Level Breech
Incidents – from over 5 days to 1
Not a single failed non-standard RFC
Service request turnaround improvements (100%)
Marketing Brochure Phase 3: “Mini Service Catalog�
Marketing Brochure Phase 3: “Mini Service Catalog�
Phase 4
Network Operations Center Phase 4 Network Monitoring
Server Monitoring
Handoff/Review Logs
Handoff/Review Logs
Orion Network Dashboard
Orion Server Dashboard
Daily Walkthrough AM/PM
VM Console
IP Monitor Int/Ext/DR
IP Monitor Probation/Int
INMON
HP SIM
Integrated ITSM Suite Phase 4: BMC Suite through the Cloud
Results Custom Written
More Results
Sep-11
Aug-11
Jul-11
Jun-11
May-11
Apr-11
Mar-11
Feb-11
Jan-11
Dec-10
Nov-10
Oct-10
Sep-10
Aug-10
Jul-10
Jun-10
May-10
Apr-10
Mar-10
Feb-10
Jan-10
Dec-09
Nov-09
Oct-09
Sep-09
Aug-09
Jul-09
Jun-09
May-09
Apr-09
Mar-09
Feb-09
Jan-09
Evidence of Improvements
Results
800
Incident Tickets
700
600
500
400
300
200
100
0
Percent Resolved
100%
90%
80%
70%
60%
50%
Evidence of Improvements Results SLA Exceptions Reduced by 77% 500 400 300 200 100 0
CY2009
CY2010
Evidence of Improvements Results Average Days to Resolve Tickets Reduced 75% 14
12 10 8 6 4 2 0 CY2009
CY2010
Evidence of Improvements Results: HP’s Converged Infrastructure Maturity Model
Evidence of Improvements Results: HP’s Converged Infrastructure Maturity Model
Evidence of Improvements Results: HP’s Converged Infrastructure Maturity Model Orange County versus US Public Sector Benchmark July 2011
4.00 3.50 3.00
Orange County US Public Sector
2.50 2.00 1.50 1.00 Technology & Architecture
Tools & Processes
Culture & IT Staff Maturity
Demand, Supply & IT Governance
Evidence of Improvements Results: HP’s Converged Infrastructure Maturity Model Orange County Percent of Characteristics "At or Better" July 2011 100
Head count
Top 10%
90 80
- 27%
70
Better
60
Equal 50 40 30 20 10 0 Dec-09
Jul-10
Jul-11
Evidence of Improvements Customer Survey
Scale: 1 = worst; 4 = best
Phase 5
Driving Continuous Improvements 2011 Jan
Feb
Mar Apr
May Jun
Jul
Method 2012 Aug
ITSM Integrated Suite Launch – FULL SUITE (CMDB, Change, Release…)
Sep
Oct
Nov
Service Catalog v 1
Dec
Jan
Feb
Mar Apr
May Jun
Jul
Aug
Service Catalog v 2
Customer Survey (Jul – Dec 10)
IT Service Management Path to Maturity AIMM (HP) Report 10 Network Operations Center Launched
Sep
ITSM based Reorganization
Application Monitoring
MSB Planning Success
Phase 5
Service Level Management Customer Grading by Service Virtual Server Farm Replacement
Oct
Nov
Dec
Sorted Top to Bottom (and vice-versa) Sorted by:
Customer
Sorted by:
Ave
A
B
1 Enterprise Time Synchronization 2 Midrange Platform (AIX Platform) 3 Mainframe Platform 4 Enterprise Network Connectivity (WAN) 5 Enterprise Secure File Transfer Services 6 Enterprise RightFax Services 7 SLA1 – Unmanaged Server Hosting 8 Database Mgt – Information Resource Mgt 9 Enterprise Domain Name Resolution 10 Blackberry Services 11 OnBase 12 VTL Replication Service 13 Standard Backup Service 14 Enterprise Email Services 15 Finance Services 16 Enterprise Storage Services (SAN) 17 Enterprise Internet Service Provider (ISP) 18 Technical Consultation 19 Applications Services Group 20 Desktop Support 21 Enterprise Global User Directory Services 22 Publishing Services 23 Enterprise Network Security 24 DeltaCopy Data Protection Servce (DCDP) 25 Business Recovery Center 26 SLA3 Service 27 Enterprise Remote Access Infrastructure 28 OCTNET 29 Project Management 30 Service Desk 31 eGov Services 32 Enterprise Secure Web Filtering 33 Disaster Recovery
96.50 96.50 96.00 94.71 94.20 94.00 94.00 93.67 93.20 92.67 92.60 91.67 91.50 91.33 91.33 91.25 90.86 90.80 90.50 90.00 88.67 88.67 88.50 88.33 88.33 87.80 86.86 86.39 85.00 84.78 83.33 82.50 66.50
Customer Ave
1 Disaster Recovery 2 Enterprise Secure Web Filtering 3 eGov Services 4 Service Desk 5 Project Management 6 OCTNET 7 Enterprise Remote Access Infrastructure 8 SLA3 Service 9 DeltaCopy Data Protection Servce (DCDP) 10 Business Recovery Center 11 Enterprise Network Security 12 Enterprise Global User Directory Services 13 Publishing Services 14 Desktop Support 15 Applications Services Group 16 Technical Consultation 17 Enterprise Internet Service Provider (ISP) 18 Enterprise Storage Services (SAN) 19 Enterprise Email Services 20 Finance Services 21 Standard Backup Service 22 VTL Replication Service 23 OnBase 24 Blackberry Services 25 Enterprise Domain Name Resolution 26 Database Mgt – Information Resource Mgt 27 Enterprise RightFax Services 28 SLA1 – Unmanaged Server Hosting 29 Enterprise Secure File Transfer Services 30 Enterprise Network Connectivity (WAN) 31 Mainframe Platform 32 Enterprise Time Synchronization 33 Midrange Platform (AIX Platform)
66.50 82.50 83.33 84.78 85.00 86.39 86.86 87.80 88.33 88.33 88.50 88.67 88.67 90.00 90.50 90.80 90.86 91.25 91.33 91.33 91.50 91.67 92.60 92.67 93.20 93.67 94.00 94.00 94.20 94.71 96.00 96.50 96.50
Service Catalog ISP
Service Catalog Storage Area Network
Extend ITSM Based Organization Move the Plan & Design Group and the Build & Transition Group “up”
Conclusions and Lessons Learned
Conclusions & Lessons Learned 1. Financial distress clouds have their silver linings in IT environments. a. b.
Working together in the face of a single galvanizing event Speak with “one voice”
2. It Takes an IT mojo-driven strategy to implement sustained continued improvements in financial distress. a. b.
Courage and consistency; buy-in Incremental changes (process, organizational), planning and soft skills provided results
3. New leadership can ignite IT Mojo. a. b.
Common goals Common messages
4. Establish a drumbeat to pace the changes and measure visibly and consistently. a. b.
5.
Messaging consistency Allow the organization to digest changes
Establish key measurements of the categories shown in the customer surveys. a. b.
Report on them Make sure everyone knows what’s being measured
6. Establish a consistent program of accountability . a. b.
Report on them Make sure everyone knows what’s being measured
7. Benchmark the organization before commencing the journey.
Questions and Answers