Testing induction
Testing Essentials Unit 9 & 10 Test Process Management QSIT- Quality Solutions for Information Technology Pvt. Ltd., A 806 & 807, Mittal Tower, BANGALORE- 560 001,INDIA URL: www.qsitglobal.com PH: +91 80 41134334 Fax: +91 80 25597445
Software Testing Essentials
1
www.qsitglobal.com
Test Management •
Overview
•
Test Organization
•
Test Planning and Estimation
•
Test Progress monitoring and control
•
Incident Management
•
Configuration Management
•
Risk and Testing
Project Risk, Product Risk
Software Testing Essentials
QSIT Copyright
2
www.qsitglobal.com
Page:1
Testing induction Levels of Independent Testing
The Developer Independent Testers ceded to the development team Independent permanent test team with in the organization (IVS) Independent testers or test team provided by the operational business units Specialist testers such as usability, security and performance testers Outsourced test team or testers Eg: Contractors or other organization
Low
High
Software Testing Essentials
3
www.qsitglobal.com
Test Management Benefits
Isolation from the development team (if treated as totally independent on the test basis to understand what it is the tester is testing
The tester is unbiased
Tester may be seen as bottleneck as independent test execution is normally the last stage and affected by any delays earlier in the process of testing
The tester can see what has been built rather than what the developer thought had been built
Developers lose a sense of responsibility for quality as it may be assumed that they need not worry about errors because test team exists
The tester makes no assumptions regarding quality
Fully independent view sets developers and testers on either side of an invisible fence. This can hinder communication. It can also mean that developers are seen to throw the software over the fence
Software Testing Essentials
QSIT Copyright
Drawbacks
The tester sees other and different defects too than the author
4
www.qsitglobal.com
Page:2
Testing induction Typical tasks of a Tester may include • Reviewing and contributing to the development of test plans • Analyzing, reviewing and assessing user requirement and models for testability • Creating test specification for test basis • Setting up test environment (often coordinating with system administrator) • In some organization setting up and management of test environment could be centrally controlled in this situation tester would directly liaise with environment management to ensure test environment is delivered in time • Preparing and acquiring/creating test data • Implementing tests on test level, executing and logging the tests • Evaluating the results and documenting the deviation in the form of defects • Automating tests (may be supported by a developer or test automation expert) • Reviewing tests developed by other testers
Software Testing Essentials
5
www.qsitglobal.com
Typical tasks of a Test Leader may include • Coordinating the development of test strategy and plan with project manager and others • Writing/Reviewing test strategy produced for the project and test policies produced for the organization •Planning the development of required tests. •Managing the specification, preparation, implementation and execution of tests, including monitoring and control of all specification and execution • Taking required action including adopting the planning, based on test results and progress • Ensuring the adequate configuration management of test ware is in place and that the test ware is fully traceable • Putting in place suitable metrics for measuring test progress and evaluating the quality of testing •Agreeing what should be automated, to what degree and how •Scheduling all testing activity •End of the project writing a test summary report based on the information gathered during testing
Software Testing Essentials
QSIT Copyright
6
www.qsitglobal.com
Page:3
Testing induction Testing Strategies Introduction to Strategy Approach to Software Testing Strategy – Defect Prevention Strategy – Defect Detection Strategy Developing Strategy for New Project Developing Strategy for Maintenance project Inputs and Outputs of Strategy Issues and Risks to be considered while doing Strategy
Software Testing Essentials
7
www.qsitglobal.com
Testing Process Testing Process has following steps: Planning and Control Analysis and Design Implementation and execution Evaluating exit Criteria and reporting Test Closure activities
Software Testing Essentials
QSIT Copyright
8
www.qsitglobal.com
Page:4
Testing induction Introduction Test Strategy
Defines the scope and general direction for testing in the project
The intention of Test Strategy is to establish framework for testing the system and work products.
Aim is to find most important errors as early as possible hence reducing the cost of fixing the errors
To define the level of participation for the relevant stakeholders.
Should identify types of testing to address client’s requirements.
Test strategy should be considered as a prerequisite and overview to the testing design. Software Testing Essentials
9
www.qsitglobal.com
Approach to Software Test Strategy
Defect Prevention Strategies – Examination of constraints –
Early test involvement
–
Use of Standards
–
–
Defect Detection Strategies – Inspections and Walkthroughs –
Testing Product Deliverables
–
Design Testability in to the application
–
Use of Automated tools
–
Adherence to a test process
Inspection and Walkthroughs Quality Gates
Software Testing Essentials
QSIT Copyright
10
www.qsitglobal.com
Page:5
Testing induction Developing Strategy for New Project
To Clarify expectations with the user, sponsor and bidders
To Describe the details of how the testing team will evaluate the work products, system, and testing activities and results
To describe approach to all testing phases and the activities for which they are responsible
Software Testing Essentials
11
www.qsitglobal.com
Developing Strategy for Maintenance Project
Includes a greater focus on regression testing an keeping the users informed of specific fixes or changes that were requested
Test process should be described in terms of the periodic release cycles that are part of the change control process
Also describe a set of minimum tests to be performed when emergency fixes are needed (for instance, due to failed hardware or recovering from a database crash)
Software Testing Essentials
QSIT Copyright
12
www.qsitglobal.com
Page:6
Testing induction Test Strategy: Inputs & Deliverables
Time Required for Testing
Priority & Criticality
No. & Levels of Resources
Types of Application
Rounds of Testing
Test Deliverables Strategy
Exit Criteria Test Suspension Criteria
Project Success Criteria
Resumption Criteria
Software Testing Essentials
13
www.qsitglobal.com
Test Risks Poor Requirements Stakeholder Participation
Test Staffing
Requirement Testability Schedule Compression
Test Related & Risks & Considerations
Performance and Stress Testing
Testing of COTS
External Interfaces
Acceptance
Insufficient Resources Software Testing Essentials
QSIT Copyright
14
www.qsitglobal.com
Page:7
Testing induction Typical Test Issues
Test Participation Performances and Capacity Requirement/Testing
Test Environments
Approach of Testing COTS products
Test Issues
Scope of Acceptance Testing
Pilot of Field Testing Criteria for Acceptance of the System
Software Testing Essentials
Approach to Testing External Interfaces
Verification of Un-testable Requirements
15
www.qsitglobal.com
Summary
Test Strategy Must answer the following: – – – – – – – – –
When will testing occur? What kind of testing will occur? What are the risks? What are the critical success factors? What are the testing objectives? What are the trade-off? Who will conduct the testing? How much testing will be done? What tools will be used?
Software Testing Essentials
QSIT Copyright
16
www.qsitglobal.com
Page:8
Testing induction Purpose of Test Plan
Provides guidelines to Management to understand technical effort needed to support test program
Establishes nature and extent of tests necessary to achieve test goals and objectives
Outlines the orderly schedule of events and activities testing
Outlines efficient use of resources – Personal, financial, equipment and facility resources
Software Testing Essentials
17
www.qsitglobal.com
Purpose of Test Plan
Provides assurance of level of test coverage through traceability matrix
Outlines the contents of Test procedure scripts and its execution
Clearly details the Test Completion Criteria along with list of supporting documents.
Software Testing Essentials
QSIT Copyright
18
www.qsitglobal.com
Page:9
Testing induction Purpose of Test Plan
Provides assurance of level of test coverage through traceability matrix
Outlines the contents of Test procedure scripts and its execution
Clearly details the Test Completion Criteria along with list of supporting documents.
Software Testing Essentials
19
www.qsitglobal.com
Test Planning and Estimation This involves following activities:
Test Plan Identifier
Introduction
Test Items
Features to be tested
Features not to be tested
Approach
Item Pass/Fail criteria. Suspension /Resumption Criteria
Schedule/ Mile Stone
Entry/Exit Criteria
Testing Deliverables and Risk and Contingencies
Resources & Responsibility and Approvals
Software Testing Essentials
QSIT Copyright
IEEE testplan template
20
www.qsitglobal.com
Page:10
Testing induction Test Plan – Exit Criteria Purpose : to define when to stop testing Typical Exit Criteria might be : • All tests planned have been run • A certain level of requirements coverage has been achieved • No high –priority or severe defects are left outstanding • All high risk areas have been fully tested, with only minor residual risks left outstanding • Cost – when the budget has been spent • The schedule has been achieved Eg: the release date has been reached and the product has to go live Software Testing Essentials
21
www.qsitglobal.com
Test Metrics
Software Testing Essentials
QSIT Copyright
22
www.qsitglobal.com
Page:11
Testing induction Test Metrics Collection and Analysis
Scope of process:
Metrics
Collection
and
analysis
–
Goals and objectives are set relative to the product and software (test) management process
–
Measurements are defined and selected to ascertain the degree to which the goals and objectives are being met
–
The data collection process and recording mechanism are defined and used.
Software Testing Essentials
23
www.qsitglobal.com
Test Metrics Collection and Analysis
Scope of Metrics process (Cont.):
and
analysis
–
Measurements and reports are part of a closed-loop system that provides current (operational) and historical information to technical staff and management.
–
Data on post-software product life measurement are retained for analysis that could lead to improvements for future product and process management.
Software Testing Essentials
QSIT Copyright
Collection
24
www.qsitglobal.com
Page:12
Testing induction White Box Testing Metrics or Depth Coverage Analysis
Source Code Analysis – Helps discern code quality –
Determines whether an algorithm works
–
Function is being called too frequently
–
Identifies complex areas of the source code
–
Quantifies Fault density ,Fd = Nd/KSLOC, where Nd is the number of defects, and KSLOC is the number of non-comment lines of source code.
Software Testing Essentials
25
www.qsitglobal.com
Black Box Testing Metrics or Breadth Coverage Analysis
Metrics collection focuses on the breadth of testing, such as the amount of demonstrated functionality and the amount of testing that has been performed. Categories of testing metrics to be collected during the black-box testing : – Coverage, –
Progress
–
Quality
Software Testing Essentials
QSIT Copyright
26
www.qsitglobal.com
Page:13
Testing induction Black Box Testing Metrics - Coverage Metrics
Test Coverage
System Coverage Analysis
Functional Test Coverage
Software Testing Essentials
27
www.qsitglobal.com
Black Box Testing Metrics - Progress Metrics
Test procedure execution status
Error Discovery Rate
Defect Aging
Defect Fix Retest
Defect Trend Analysis
Software Testing Essentials
QSIT Copyright
28
www.qsitglobal.com
Page:14
Testing induction Black Box Testing Metrics - Quality Metrics
Test Success Index
Defect Density
Defect Trend Analysis
Test Effectiveness
Problem Report-Acceptance Criteria Metrics
Test Automation Metric Software Testing Essentials
29
www.qsitglobal.com
Test Reporting Test Reporting is concerned with summarizing information about the testing endeavor • What happened during a period of testing such as dates when exit criteria were met • Analyzed information and metrics to support recommendation and decision about future action •An assessment of defects remaining • economic benefit of continued testing • outstanding risks • level of confidence in the tested software • Metrics should be collected during and at the end of a test level to assess • the adequacy of the test objective for that level • adequacy of the test approach taken • Effectiveness of testing with respect to its objectives
Software Testing Essentials
QSIT Copyright
30
www.qsitglobal.com
Page:15
Testing induction Test Summary – IEEE 829 To
summarize the results of the designated testing activities and to provide evaluation based on these results A test summary report shall have the following structure • Test summary report identifier • summary • Variances • comprehensive assessment • summary of results • evaluation • summary of activities • approvals
Software Testing Essentials
31
www.qsitglobal.com
Test Control Test control describes any guidance or corrective action taken as a result of information and metrics gathered and reported
Examples of Test Control actions • Reprioritize test when an identified project risk occurs (eg. Software delivered late) • Change the test schedule due to availability of test environment • Set an entry criterion requiring fixes to be retested by developer before accepting them in to a build (this is particularly useful when defect fixes continually fail) • Review of product risks and perhaps changing the risk rating to meet the target • Adjusting the scope of testing (amount of tests to be run) to manage the testing of late change requests
Software Testing Essentials
QSIT Copyright
32
www.qsitglobal.com
Page:16
Testing induction Incident Management An Incident is any significant, unplanned event that occurs during testing that requires subsequent investigation or correction • Discrepancies between actual and expected outcomes need to be logged as incident • Incident should be tracked from discovery and classification to correction and confirmation of the solution • In order to manage all incidents to completion, an organization should establish a process and rules for classification
Software Testing Essentials
33
www.qsitglobal.com
Incident Management A tester or reviewer typically logs the following information regarding the incident
Date of issue, issuing organinsation, author, approvals and status Scope, severity and priority of the incident References, including the identity of the test case specification that reveled the problem. Date the incident was discovered Urgency or priority to fix Severity of the impact on the system Degree of impact on stakeholders interest Software or system life cycle process in which the incident was observed
Software Testing Essentials
QSIT Copyright
34
www.qsitglobal.com
Page:17
Testing induction Incident Management : Incident Report outline Sl No
Heading
Details
1
Test Incident report identifier
The unique identifier assigned to this test incident report
2
Summary
A summary of the incident, detailing where expected and actual results differ, identifying at a high level the items that are affected and the steps leading up the recognition of the incident.
3
Incident description
A detailed description of the incident which should include – Inputs – Expected results – Actual results – Anomalies – Date and time – Procedure step – Environment – Attempts to repeat – Tester’s / Observer’s comments Should also include any information regarding possible causes and solution
4
Impact Software Testing Essentials
If known, document what impact the incident has on progress 35
www.qsitglobal.com
Introduction to Software Configuration Management
Software Testing Essentials
QSIT Copyright
36
www.qsitglobal.com
Page:18
Testing induction Software Configuration Management What is SCM – Software Configuration Management? A management tool that applies an engineering discipline to manage the evolution of a system from concept through to system retirement The primary focus of this discipline is to ensure Repeatability Traceability and Integrity of the system being developed and produced Ensures that everyone know what is supposed to be built, what is being built and what has been built
Software Testing Essentials
37
www.qsitglobal.com
Software Configuration Management Purpose of SCM
Maintain product integrity Make product evolution more manageable and controlled Provide product status during evolution Ensure product meets customer requirements Actively control changes – What changes were made to the software? – Why were the changes made? – When were the changes made? – Who made the changes?
Software Testing Essentials
QSIT Copyright
38
www.qsitglobal.com
Page:19
Testing induction Software Configuration Management Need for SCM
Increased software control Integration of millions of lines of code from multiple subcontractors Complex co-ordination of product’s evolution over years by many people Avoid rework due to lack of understanding of all the components that make up a software product Many variants of main product to reach out to a diversified market Reduction in cost of rework caused by changes and defects
Software Testing Essentials
39
www.qsitglobal.com
SCM – Process Dependencies
Software Testing Essentials
QSIT Copyright
40
www.qsitglobal.com
Page:20
Testing induction Software Configuration Management Common issues in SCM Developers view this as a hindrance to product improvements due to overhead associated with the change control function of SCM Some issues that are faced due to not having a good SCM Practice The latest version of source code cannot be found A difficult defect that was fixed at great expense suddenly reappears A developed and tested feature is mysteriously missing A fully tested program suddenly does not work The wrong version of the code was tested There is no traceability between the software requirements, documentation and code Programmers are working on the wrong version of the code The wrong versions of the configuration items are being baselined No one knows which modules comprise the software system delivered to the customer
Software Testing Essentials
41
www.qsitglobal.com
Software Configuration Management What is a build? Identifying a set of source items (features, fixes, changes etc) and through some means of compilation or generation, producing a set of executables and other files which can be used to perform a runtime function
What is a Package? Includes output from the build and other items like scripts, database items, documents, etc that create a set of deliverables that run in a test or production environment
Software Testing Essentials
QSIT Copyright
42
www.qsitglobal.com
Page:21
Testing induction Software Configuration Management Version Numbering:
Build version number will have the following components
Product name Eg: CMM, BSC, FIN,..
Major Number - Identifies major milestones in the product
Minor Number – Identifies intermediate milestones on the product road map
Upgrade Number - These are typically incremental on the intermediate milestones and are built on demand
Patch Number – Identified ad-hoc fixes made for immediate deployment to customer sites. These are not planned releases
Software Testing Essentials
43
www.qsitglobal.com
Configuration Management Functions
Team
Components
Process
Versions and Configurations
Audit History
Baselines
Traceability
Project Context Information Repository
Logging
Structure
Control
System Model
Access Control
Interfaces
Change Requests
Relationships
Bug Tracking
Selection
Change Propagation
Consistency
Partitioning
Construction
Accounting
Building
Statistics
Snapshots
Status
Optimization
Reports
Impact analysis Regeneration
Software Testing Essentials
QSIT Copyright
44
www.qsitglobal.com
Page:22