Scalable and Tailorable Appraisal Methods SCAMPI B&C Presented to the Silicon Valley SPIN April 26, 2006 by Diane McDonald
http://www.stepupsolutions.com This material was adapted from the SEI’s Handbook for Conducting SCAMPI B and C Appraisals, version 1.1 and overview material ® CMMI and Capability Maturity Model Integration are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. ® StepUp Solutions is registered in the U.S. Patent and Trademark Office. sm SCAMPI is a service mark of Carnegie Mellon University.
PDF created with pdfFactory trial version www.pdffactory.com
Topics • Standard CMMI Appraisal Method for Process Improvement (SCAMPI) Family of Appraisals • SCAMPI B and C Characteristics • Tailoring Considerations • Usage Modes • Core Concepts • Appraisal Phases • Summary
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 2
SCAMPI Appraisals • SCAMPI - Standard CMMI Appraisal Method for Process Improvement - SEI’s official methodology for conducting CMMI-based appraisals - “SCAMPI” B & C are distinguished from “Class” B and C appraisals, which are defined by Lead Appraisers using their own techniques • The SCAMPI method is embodied in 3 standard variants based on the class structure defined in the Appraisal Requirements for CMMI (ARC V1.1) • The SCAMPI family of appraisals provides a variety of solutions to accommodate various appraiser and organizational needs, e.g., - Self-assessments, initial appraisals, quick-look or mini-appraisals, incremental appraisals, benchmarking appraisals - Shared definitions of inputs, activities, outputs (upward compatible results) - Wide variety of possible tailorings to fit different situations
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 3
SCAMPI A, B, & C Comparison SCAMPI C - provides broad flexibility in addressing the approach planned (or taken) to implement processes. SCAMPI B - provides flexibility in model & organizational scope and is performed on deployed (implemented) practices. SCAMPI A - the most comprehensive and rigorous method, providing ratings (goal, maturity level rating, capability level ratings).
range of tailoring
depth of investigation
The tailoring and scaling permitted in each class is so wide that the 3 methods can overlap in many respects.
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 4
SCAMPI A, B, & C Comparison (cont.)
Class C
Class B
Class A
Resource requirements (organization/ appraisal team)
Low
Medium
High
Amount objective evidence
Low
Medium
High
Characterizations generated
Optional
Yes
Yes
Ratings generated
No
No
Yes
Characteristics
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 5
SCAMPI B & C Tailoring Examples • Model scope/focus - Number of Process Areas (PAs), coverage of PAs (e.g., specific practices only, single goal only) - Level of granularity (SCAMPI C), e.g., practice level, goal level • Organizational unit scope - 1 project, several projects - Disciplines (e.g., SW Engineering only, SW and Systems Engineering) • Appraisal team vs. individual; team size • Approach vs. deployment - Review of processes only, review of processes and implementation • Appraisal data sources - documentation, interviews, both • Appraisal output options - For SCAMPI C, findings only, or findings and characterizations (color-coded charts) - Findings/charts at project level or rollup at organizational level SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 6
Example Usage Modes Gap Analysis • Initial detailed appraisal to baseline an improvement plan • Appraisal used to bridge to new model, or new PAs, or new parts of an organization • Detailed look at target areas found weak in a previous appraisal anticipating a future benchmark appraisal
Mini-Appraisal • Interview-intensive first-time experience for new group learning about process improvement and appraisals • Full coverage dress rehearsal for benchmarking appraisal in a group that has a past history of process improvement
Improvement Monitoring • Internal progress appraisal using EPG staff to monitor deployment of new practices • Contract incentive award determination based on external auditor • Consultants may perform incremental appraisals building PIIDs for a future benchmark appraisal
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 7
Core Concept 1: SCAMPI Family Architecture SCAMPI family of appraisals can cover the following range: • SCAMPI C “approach,” - looking at the approach planned to satisfy process improvement goals, to • SCAMPI B “deployment” - examining deployment of processes in selected projects in an organizational unit, to • SCAMPI A “institutionalization” - benchmarking the institutionalization of CMMI in an organizational unit
Approach
Deployment
Institutionalization
A Reliability, rigor & cost
Accuracy of results
B C
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 8
Core Concept 1: SCAMPI Family Architecture (cont.) Example Scenario • Phase 1 - SCAMPI C is conducted on the proposed processes => Approach - Using the results from the SCAMPI C, the organization could then make adjustments to these processes to increase their confidence of fidelity to the CMMI. - Question being addressed: If these proposed processes are implemented, would they satisfy the goals and meet the intent of the CMMI? • Phase 2 - Implement the processes in pilot projects => Deployment - A SCAMPI B could then be conducted to examine the results of executing these processes. - Question being addressed: If the processes examined in the pilot projects were broadly implemented, would they satisfy the goals and meet the intent of the CMMI? • Phase 3 - The organization could then, with a higher level of confidence, broadly implement the processes => Institutionalization - A SCAMPI A could then be used to validate institutionalization and establish a benchmark for the organization. SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 9
Core Concept 2: Objective Evidence • Objective Evidence (OE) includes Direct Artifacts, Indirect Artifacts, and Affirmations - Direct Artifacts - tangible outputs resulting directly from performance of a CMMI practice (e.g., CMMI typical work products) - Indirect Artifacts - a secondary consequence of performing a CMMI practice (e.g., meeting minutes, reviews, logs, and reports) - Affirmations - oral or written statements confirming practice implementation • The organization can build a set of OE gradually, through a series of SCAMPI C and/or B appraisals - Practice Implementation Indicator Descriptions (PIIDs) provide an organized structure for storing and presenting the objective evidence needed for the appraisal • OE examined during an appraisal are related to the CMMI practices (or other components, such as goals, for a SCAMPI C) SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 10
Core Concept 3: Data Sources • Primary data sources: - Documentation - in hardcopy or electronic provide evidence for a wide range of practices - Interviews - used to corroborate the documentation and gain insight into how the work is actually performed • Note: For a SCAMPI B, evidence of implementation (not just intended approach, as in SCAMPI C) must be examined
OE
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 11
Core Concept 4: Practice Characterizations • Each SCAMPI class has a unique characterization scale designed to match the minimum standards of data sufficiency for that method - SCAMPI B - uses red, yellow, green - Each practice within scope is characterized indicating the risk of CMMI goal satisfaction if the observed practices were deployed across the organizational unit. - Model scope is not limited to entire Process Areas, but could include sets of related practices (e.g., associated with 1 goal)
- SCAMPI C - uses low, medium, high - The intent of the model practice is characterized indicating the risk of CMMI goal satisfaction if the proposed processes were implemented. - Data may be examined at a higher level of granularity than an individual practice (e.g., goal, process area, maturity level)
- Note: Rating of goals are not permitted in SCAMPI B or C appraisals
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 12
Core Concept 5: Appraisal Outputs • Outputs associated with each class of method are matched to the conclusiveness supported by the appraisal process. Outputs: - Findings (weaknesses, strengths) - Characterizations - Standard for SCAMPI B; Optional for SCAMPI C
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 13
SCAMPI B & C Phases Three phases involved in all SCAMPI appraisals: • Plan and Prepare for Appraisal • Conduct Appraisal • Report Results
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 14
Phase 1 - Plan and Prepare for Appraisal Requirement
SCAMPI C
SCAMPI B
Led by SEI-authorized Team Lead
Yes
Yes
Communication between Team Lead and sponsor
Yes
Yes
Sponsor approves Appraisal Input and Plan and any changes
Yes
Yes
1
2
Yes, if a team is used
Yes
Criteria for team experience in engineering and management
No
Yes
Objective evidence mapped to CMMI
Yes
Yes
Data Collection Plan
Yes
Yes
Readiness Review
Yes
Yes
Minimum team size Team members complete Intro to CMMI and appraisal training
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 15
Primary Contents of Appraisal Input • Sponsor identity • Appraisal purpose, including alignment with business objectives • CMMI model used and scope • Organizational unit appraised and process context • Appraisal constraints • Identity and affiliation of the Team Leader, appraisal team members, participants and support staff, and their responsibilities • Planned appraisal activities and outputs • Appraisal resources, schedule, and logistics • Risks and mitigation plans • Anticipated follow-on activities • Planned tailoring of SCAMPI and associated tradeoffs, including sample size or coverage of the organizational unit
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 16
Example SCAMPI C Readiness Review Document Count Chart RM-D # Complete # Incomplete # Missing # Not Reviewed
RM-I 25 19 7 0
RD-D 0 0 0 51
RD-I 31 37 4 0
TS-D 4 0 0 68
TS-I 34 23 12 0
PI-D 0 0 0 69
PI-I 25 29 9 0
VER-D 0 1 0 62
Engineering Category Results
VER-I 26 30 4 0
VAL-D 1 0 0 59
VAL-I 19 23 9 0
# Not Review ed # Missing # Incomplete # Complete
100% 90%
% Practices
80% 70% 60% 50% 40% 30% 20% 10% 0% RM-D
RM-I
RD-D
RD-I
TS-D
TS-I
PI-D
PI-I
VER-D
VER-I
VAL-D
VAL-I
Process Areas
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 17
0 0 0 51
Phase 2 - Conduct Appraisal Requirement
SCAMPI C SCAMPI B
Direct artifacts, indirect artifacts, affirmations identified
Yes
Yes
Objective evidence mapped to CMMI & projects/org
Yes
Yes
either
both
1
2
Interviews conducted by at least 2 team members
No
Yes
Observations corroborated
No
Yes
Validation of preliminary findings
No
Yes
Final Findings include strengths & weaknesses (gaps wrt CMMI)
Yes
Yes
Direct artifacts reviewed, interviews conducted Sources of objective evidence (interviews, documentation, instruments)
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 18
Phase 2 Steps - Conduct Appraisal •Direct Artifacts •Indirect Artifacts •Affirmations
Examine Objective Evidence
Validate Prelim. Appraisal Outputs
•notebooks •spreadsheets
Document Objective Evidence
Generate Appraisal Outputs
•accuracy •model relevance •consistency •corroboration
Verify Objective Evidence
•Findings •Characterizations*
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
•feedback •optional for SCAMPI C
*Optional for SCAMPI C
StepUp Solutions, Inc.
page 19
Characterization Options - SCAMPI B • Every practice in the appraisal scope is characterized on a 3 point scale (red, yellow, green) indicating risk of CMMI goal satisfaction if the observed practices were deployed across the Organizational Unit (OU) • Model scope is not limited to entire PAs, but could include sets of related practices
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 20
Practice Characterization - SCAMPI B Label
Meaning
Red
The intent of the model practice is judged to be absent or poorly addressed in the set of implemented practices – gaps or issues that will prevent goal achievement, if the deployment occurred in this way across the organizational unit, were identified
Yellow
The intent of the model practice is judged to be partially addressed in the set of implemented practices – some gaps or issues were identified, which might threaten goal achievement if the deployment occurred in this way across the organizational unit.
Green
The intent of the model practice is judged to be adequately addressed in the implemented set of practices examined – in a manner that would support goal achievement, if the practice were deployed across the organizational unit.
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 21
Aggregation of Practice Characterizations - SCAMPI B **OU Characterization Red Yellow Green Team judgment subject to below *
Instance Characterizations All instantiations characterized Red All instantiations characterized Yellow All instantiations characterized Green All conditions not included above
* The OU characterization shall not be Red unless at least one instance characterization is Red * The OU characterization shall not be Green unless at least one instance characterization is Green
**OU = Organizational Unit SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 22
Data Sufficiency Criteria - SCAMPI B For each practice within the scope of the appraisal: • Rule 1: At least one Direct Artifact or at least one Oral Affirmation for each instantiation • Rule 2: In addition, at least one Direct Artifact and at least one Oral Affirmation from the set of instantiations mapped to a goal For SCAMPI B appraisals focused on a single instance: • A Direct Artifact for each practice • An Oral Affirmation for at least one practice within each set of practices mapped to a goal
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 23
SCAMPI B Coverage: Example Acceptable Combinations project A
project B project C
project D
Direct Artifacts
Oral Affirmations
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 24
Characterization Options - SCAMPI C • The characterization must be a 3-point scale • Every characterization generated must be supported by relevant objective evidence • The characterization scale must not overstate the conclusiveness of the outcome (e.g., referencing goal satisfaction) • Can be scoped at any level of granularity, and the scale can be tailored to the appraisal objectives, e.g., - Fidelity of observed practices to model goal achievement - Return on Investment to the organization from implementing practices
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 25
Example Practice Characterization for SCAMPI C: Fidelity to Model Label Low
Medium
High
Meaning The intent of the model practice is judged absent, or inadequately addressed in the approach – goal achievement is judged unlikely because of this absence or inadequacy. The intent of the model practice is judged to be partially addressed in the approach – and only limited support for goal achievement is evident. The intent of the model practice is judged to be adequately addressed in the set of practices (planned or deployed) – in a manner that supports achievement of the goal in the given process context.
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 26
Example SCAMPI C Characterization Chart Appraisal: Project Name:
Specific Goal 1 SP 1.1 SP 1.2 SP 1.3 SP 1.4 SP 1.5 SP 1.6 SP 1.7
ACME ORGANIZATION PLUTO REQM
PP
PMC
SAM
M&A
PPQA
CM
H M L H H
L M H H
L H H M M M M
H H H
H H H M
L H
H H M
H H M H H H H
M H H
H H M L
L M H H
H H
H H
Specific Goal 2 SP 2.1 SP 2.2 SP 2.3 SP 2.4 SP 2.5 SP 2.6 SP 2.7 Specfic Goal 3 SP 3.1 SP 3.2 SP 3.3 SP 3.4 SP 3.5
H M M
H H
Specific Goal 4 SP 4.1 SP 4.2 SP 4.3 Generic Goal 2 GP 2.1 GP 2.2 GP 2.3 GP 2.4 GP 2.5 GP 2.6 GP 2.7 GP 2.8 GP 2.9 GP 2.10
L M H H M H H H M M
H M H H M H H H M M
H M H H M H H H M M
L M L H M M H H H M
H M H H M H H H M M
L M L H M M H H H M
H M H H M H H H M M
Generic Goal 3 GP 3.1 GP 3.2
H H
H L
H L
H H
H L
L H
L L
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 27
Data Sufficiency Criteria - SCAMPI C At least one item of Objective Evidence must be examined for every model component in the scope of the appraisal • If the appraisal is scoped at the practice level, then this means one item of OE for each practice
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 28
Other Characterization Values - SCAMPI B & C • Not Yet - The practice under consideration has not been implemented yet, because the project has not reached the appropriate point in the life cycle. • Out of Scope - No characterization was assigned because the appraisal did not gather data to support characterization of this practice
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 29
Phase 3 - Report Results Requirement
SCAMPI B
SCAMPI C
Designated appraisal results provided to the stakeholders identified in the appraisal plan
Yes
Yes
Appraisal record delivered to the appraisal sponsor
Yes
Yes
Appraisal data package submitted to the CMMI Steward
Yes
Yes
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 30
Contents of the CMMI Steward Data Package • Record of Entry - information related to appraisal entity, participants, team, etc. • Appraisal input and plan • Appraisal findings – strength and/or weakness statements • Appraisal disclosure statement - summary statements regarding the appraisal results, conditions, and constraints • Team member and sponsor feedback
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 31
Summary • SCAMPI B and C provide a wise range of choices and customizations to be made by the organization and Lead Appraiser - Broad flexibility to define an appraisal event to match business drivers • Used in a progression, all 3 methods in the SCAMPI family can be tailored to enhance integration of data from one appraisal event to another - Supports focused individual applications, as well as use in a multi-event, on-going program of improvement
SCAMPI B & C Overview - SV SPIN April, 2006
PDF created with pdfFactory trial version www.pdffactory.com
StepUp Solutions, Inc.
page 32