Creating Sanity Amidst Test Methodology Madness Right-sizing your test methodologies
CONFIDENTIAL: For limited circulation only
Š 2011 MindTree Limited
Slide 1
Agenda
Rightsizing?
hen Identify the symptoms
hat Key aspects to right-size
ow MindTree Tools & Frameworks
Processes should work for you. You should not work for processes. CONFIDENTIAL: For limited circulation only
Š 2011 MindTree Limited
Slide 2
Poll Question #1
● Are your processes taking too much effort and time? ● Yes
● No ● Not able to measure
CONFIDENTIAL: For limited circulation only
© 2011 MindTree Limited
Slide 3
When to Rightsize?
hen Identify the symptoms
• • • •
Test processes taking too much effort and time?
•
Difficulty in quantifying the quality in your test processes?
Metrics not helping you take decisions? Not able to justify the investments in testing? Test process not helping you identify the critical defects early?
Acknowledge the need. Don’t wait until you loose your sleep. CONFIDENTIAL: For limited circulation only
© 2011 MindTree Limited
Slide 4
What to Rightsize?
hat Key aspects to right-size
• • • •
Quality goals / objectives
•
Tools and frameworks
Software life cycle methodologies
Engagement models Governance mechanism (metrics, reviews)
Tailor your processes for right fit. CONFIDENTIAL: For limited circulation only
© 2011 MindTree Limited
Slide 5
Poll Question # 2
● Do you know what metrics should you be tracking for your testing needs?
● Yes ● No ● I do, but not sure if they are exhaustive
CONFIDENTIAL: For limited circulation only
© 2011 MindTree Limited
Slide 6
Test Metrics Analysis and Decisions (TMAD) Model
Test Metrics Analysis and Decisions (TMAD) Model
ow
Metrics Based Governance
MindTree Tools & Frameworks
• • •
• •
Business goals Quality goals Governance • Metrics, reviews, baselines Forecast • Risks, best practices Corrective & preventive actions Enablers
Statistical Models
CONFIDENTIAL: For limited circulation only
© 2011 MindTree Limited
Historical Data
Org / Industry Baselines
Slide 7
Test Metrics Analysis and Decisions (TMAD) Model Metrics Analysis Proposed Metrics
Defect Containment Effectiveness
Goal
1. Improve testing quality 2. Prevent defect leakage
Associated Queries (RCA)
Undesired Behaviour (Risks)
1. Was the criteria to classify defect detection phase clear from a leakage perspective? 2. Are there clear entry and exit criteria between phases 3. Was the team's knowledge adequate to uncover leaked defects? 4. Was the scope of testing for each phase clear? 5. Do we have requirement traceability matrix?
1. Blame game 2. Manipulation of defect classification criteria
Metrics Decision Table Decision Support (Scope) 1. 1. Improve test case review effectiveness 2. Improve process compliance 3. Improve product and test design knowledge
CONFIDENTIAL: For limited circulation only
2. 3. 4. 5.
Corrective Actions (feedback) Clear entry and exit criteria between phases Process adherence Effective knowledge management Update test suite Get test strategy reviewed by customers
Š 2011 MindTree Limited
Raw Data Requirements
1. Defects detected in test phase 2. Leaked defects
Suggestions
Do RCA for every release and identify preventive actions
Slide 8
Test Quality Index (TQI) Model
● Bivalent questionnaire; Y or N
1
Identify areas of improvement
responses
● “Productivity per tester” vs.
“quality of deployed resources and resource deployment”
● Contextual metrics vs.
comprehensive score; across 30 process areas
3
Improve scores with each deliverable
2
Customize the scored checklist
● Scoring based on self assessment
Winner of the Testing Leadership Award at QAI Software Testing Conference CONFIDENTIAL: For limited circulation only
© 2011 MindTree Limited
Slide 9
Poll Question # 3
● Do you have an integrated framework which helps you rightsize the test processes and methodologies?
● Yes ● No ● We have point solutions, but not an integrated solution
CONFIDENTIAL: For limited circulation only
© 2011 MindTree Limited
Slide 10
MindTest™ - Integrated Test Methodology Processes
Lifecycle Processes
Templates
Guidelines
Checklists
Best Practices
Type of Testing
Stage of Testing
Customization Tools
Technologies
Analysis
Continuous Processes
Domain
Audits
TQI
TMAD
Test Estimates
Feedback
Metrics & Measurement
ProjectSpace
ProcessNet
Test Dashboard
Radar
TestLink
Knowledge Management
Lifecycle Processes
Test Measurement and Metrics - ROI
Defect Management
Test Management
CONFIDENTIAL: For limited circulation only
Š 2011 MindTree Limited
Slide 11
Case Study The Customer
Business Impact • 90% coverage on test cases executed, a significant improvement over the earlier metric
The customer is the world's leading specialist in air transport communications and information technology (IT) solutions.
The Business Challenge • No testing guidelines • Inadequate process • Lack of measures for impact of testing • No single snapshot of engagement level governance
•
•
•
100% achievement on acceptable open defects for severity 1 and 2
• •
Customer overhead of test case reviews reduced by 60% Constant review of the test case execution helped us to evaluate and report how far/near is the application from the acceptance criteria at different phases of testing
•
Enabled development team to prioritize the defects that needs to be fixed for the immediate build
•
Customer is now able to ascertain the quality of the application
•
Reduced risks and enabled effective decision making through test dashboard
MindTree Solution Executed in-depth analysis to understand
•
Quality objectives and alignment between test process and application's business goals
•
Used Test Metrics Analysis and Decisions (TMAD) Model and Test Quality Index (TQI) to come up with appropriate metrics and processes
“MindTree's expertise in metrics is paramount in managing the engagement and seeing if process changes have made the hoped for improvement and if the application is ready for release. The metrics are also a great way to communicate progress across all the groups in this engagement.”
Collected key metrics as governance enablers to measure and drive decisions based on the health of the projects
CONFIDENTIAL: For limited circulation only
© 2011 MindTree Limited
Senior Manager, Quality Integration
Slide 12
In Short…
You don’t have to search for clues in the labyrinth
Identify the need
CONFIDENTIAL: For limited circulation only
Evaluate the key aspects
Optimize the process
© 2011 MindTree Limited
Measure and improve
Slide 13
MindTree Testing Services Overview
Testing Size and Focus
Testing Depth and Breadth
• Dedicated unit of 2200+ specialists focused on testing
• Tested over 2000 applications & products from chip to cloud
• Contributes 25% of the MindTree’s global business
• Services ranging from point solutions to long term CoEs
• Domain specific testing services and solutions
1500+
Testing Process and Delivery Excellence
Thought Leadership
• Testing pillars of excellence - MindTest™, TestLabs, TestingAcademy
• Part of ISO working group in defining new testing standard – ISO / IEC 29119
• Virtual practices and CoEs for effective delivery
• President of India’s leading testing forum STeP-IN
700+
20+
95%
delivering PREDICTABLE QUALITY through INDEPENDENT TESTING CONFIDENTIAL: For limited circulation only
© 2011 MindTree Limited
Slide 14
MindTree Workshop
ARE YOU TRACKING THE RIGHT TEST METRICS FOR YOUR BUSINESS DECISIONS? Complimentary Workshop at your Doorstep REGISTER NOW www.mindtree.com/events/testingwebinar/index.html or email to Balaji_Thesma@mindtree.com
CONFIDENTIAL: For limited circulation only
Š 2011 MindTree Limited
Slide 15
Sylvia Veeraraghavan Sylvia_Veeraraghavan@mindtree.com www.mindtree.com CONFIDENTIAL: For limited circulation only
Š 2011 MindTree Limited
Slide 16