Testing induction
Testing Essentials Unit Levels of Testing QSIT- Quality Solutions for Information Technology Pvt. Ltd., A 806 & 807, Mittal Tower, BANGALORE- 560 001,INDIA URL: www.qsitglobal.com PH: +91 80 41134334 Fax: +91 80 25597445
Software Testing Essentials
1
www.qsitglobal.com
Unit Testing Verifies individual hardware or software units, or small groups of related code unit. Emphasis is on removal of basic coding errors. Assumptions/Pre-conditions White Box testing is Completed Generally conducted by Developer Approach Identify and Develop Unit Specifications. Refer LLD and develop unit design. Derive Unit test cases. Identify extremes and worst case test scenarios Execute implemented test cases and scenarios Modify and re-execute until test completion.
Software Testing Essentials
QSIT Copyright
2
www.qsitglobal.com
Page:1
Testing induction Unit Testing Unit testing is done at module level. Derivation of Test cases are based on the program structure Knowledge of the program is used to identify the test cases . Objective is to exercise all parts of the program . Test coverage measurements are done to verify that all program parts are exercised Drivers are used to feed the data to the program to test it, wherever required. Test stubs are used to stub for the functions which are outside the module
Software Testing Essentials
3
www.qsitglobal.com
Unit Testing
Purpose – – – – – – – – – – – –
Typos and Syntax errors Basic logic problems Code Coverage Every path/line of code new or modified should be executed, tested Code inspection should verify functionality Possible values be tested for data entry fields Error cases to be verified and required to end gracefully Data Accuracy Decimal Places Help screens Memory leaks Unit test documentation
Software Testing Essentials
QSIT Copyright
4
www.qsitglobal.com
Page:2
Testing induction Integration Testing Verifies functional groups, inter-function interfaces, user and business workflows and scenarios. Assumptions/Pre-conditions
Unit and Function Testing completed
Approach
Top down
Bottom up
Software Testing Essentials
5
www.qsitglobal.com
Integration Testing Purpose
Test all functional groups and areas.
COTS Software components. (Commercially Off the Shelf Solution)
Response time of integrated modules.
Documentation.
Software Testing Essentials
QSIT Copyright
6
www.qsitglobal.com
Page:3
Testing induction Types of approaches- Top-Down A 1
x
2
y
z
In this approach, the modules at the top of the overall logic path are tested first, using stubs for the called modules. Then the called modules are added, using stubs for modules that they might call. This continues until the entire application is tested. This can be easier to logically understand, but it saves the bulk of the testing complexity until later. Stub: When an element under test calls other elements which are not present, STUBs are needed to take their places. Software Testing Essentials
7
www.qsitglobal.com
Type of Approaches- Bottom-Up A 1
x
z
This is the opposite. You first start off with programs at the lower level, and utilize driver programs to call them. Then you replace the drivers with the actual programs and create driver programs to call them from one level up Driver: When an element under test cannot be executed by itself, a DRIVER is needed to invoke the element. Application Os Driver Hardware driver drives the hardware, driver is the inter connectivity for application and hard ware or is nothng but interface between Hardware and software... Apllication by itself cannot make or take any Action or can make any "COMMIT" on the data, so to achive you have to use the appropritate drive of the hardware you are using... Software Testing Essentials
QSIT Copyright
y
2
8
www.qsitglobal.com
Page:4
Testing induction Interface Testing Verifies the interfaces (data and control elements) with external systems. Also referred as Interoperability Testing. Assumptions/Preconditions
Unit, functional and integration testing
All Critical Errors
Simulated environment.
Approach
Study Hardware and Software requirements Arrive at critical interface requirements Plan for preparation for testing – Internal or external Set up the simulated environment Derive test cases and scenarios – Collect if any specific procedures, formats are available externally
Software Testing Essentials
9
www.qsitglobal.com
Interface Testing
Execute the test cases – Involve people if required for verification of results,
Modify and re-execute till all the critical requirements are tested.
Purpose
Interfaces with External Systems Hardware/Software specifications Normal and Exceptional test cases to be executed to verify exchange of data. Normal and Peak volumes and traffic of data should be tested Response time is tested from both the system side Relevant Documents are distributed to all appropriate stakeholders.
Software Testing Essentials
QSIT Copyright
10
www.qsitglobal.com
Page:5
Testing induction System Testing Why is System Testing Done? Provide independent perspective in testing Bring in customer perspective in testing Provide a “fresh pair of eyes” to discover defect not found earlier Test product behavior in a holistic, complete and realistic environment Test both functional and non-functional aspects of the product Build confidence in the Product Analyse and reduce the risk of releasing the product Ensure all requirements are met and ready the product for acceptance testing Software Testing Essentials
11
www.qsitglobal.com
System Testing Approach –
–
– –
–
–
Study System Requirement Specification Identify Functional and Non-Functional Test Factors Derive appropriate Test Cases. Identify critical and Exceptional test cases and scenarios. Ensure appropriate Test environment /test bed and associated components/documents are available. Modify and rerun test scenarios and cases till all the requirements or test completion criteria is met.
Software Testing Essentials
QSIT Copyright
12
www.qsitglobal.com
Page:6
Testing induction System Testing Functional Vs Non Functional Testing Testing Aspect
Functional Testing
Non-Functional Testing
Involves
Product Feature & Functionality
Quality Factor
Tests
Product behavior
Behavior and experience
Result Conclusion
Simple Steps, expected Result
Huge Data collected & Analysed
Result Varies due to
Product implementation
Product implementation, resources & configuration
Testing Focus
Defect Detection
Qualification of Product
Knowledge Required
Product and Domain
Product, domain, design, architecture statistical skills
Software Testing Essentials
13
www.qsitglobal.com
System Testing Functional Vs Non Functional Testing (Cont.) Testing Aspect
Functional Testing
Non-Functional Testing
Failure normally due to
Code
Architecture, design and code
Testing Phase
Unit, component, integration, system
System
Test case repeatability
Repeated many times
Repeated only in case of failures and for different configurations
Configuration
One-time setup for a set of test cases
Configuration changes for each test case
Software Testing Essentials
QSIT Copyright
14
www.qsitglobal.com
Page:7
Testing induction System Testing Functional System Testing Spread across various phase Problems Duplication Gray Area Some common techniques for product functional Tesing Design/architecture verification Business vertical testing Deployment testing Certification, Standards and testing for compliance
Software Testing Essentials
15
www.qsitglobal.com
System Testing Design /Architecture Verification Test cases are designed and developed to check design and architecture – Product Level Test Cases Compliments completeness of product implementation Test cases not system functional test cases are One focusing on logic data structure and unit of the product Specified in functional specification of any component Specified in design and architecture for interconnecting components Test case focus is on product implementation but not visible to customers
Software Testing Essentials
QSIT Copyright
16
www.qsitglobal.com
Page:8
Testing induction System Testing Business Vertical Testing Implementation Testing of general workflow automation systems to different business verticals. Business verticals are: Insurance, Banking, Asset Management Business Vertical Testing Focus should be on Role base operation Customization Terminology Testing is Accomplished in two ways Simulation Replication
Software Testing Essentials
17
www.qsitglobal.com
System Testing Deployment Testing Implementation Testing of general workflow automation systems to different business verticals. Business verticals are: Insurance, Banking, Asset Management Business Vertical Testing Focus should be on Role base operation Customization Terminology Testing is Accomplished in two ways Simulation Replication
Software Testing Essentials
QSIT Copyright
18
www.qsitglobal.com
Page:9
Testing induction System Testing Certification Testing Testing a product to be certified with the popular hardware operating system database other infrastructure pieces Testing a product to ensure that the standards are properly implemented is “Testing for Standards” Example: Evolving - Ipv6 in Networking, 3G in Mobile Technology Open source Community Standards - Open LDAP standard Testing a product to ensure contractual and legal requirements for a product is met is achieved by compliance testing Example: Compliance to FDA 508 Accessibility Guidelines SOX – Sarbanse – Oxley’s Act OFAQC – and Patriot Act
Software Testing Essentials
19
www.qsitglobal.com
System Testing Non-Functional System Testing Major Challenge is about Setting up environment Simulated Environment Real Customer Environment Configuration setting is a challenge since High density of environment Varied combination Cost Non availability Predicting customer data
Software Testing Essentials
QSIT Copyright
20
www.qsitglobal.com
Page:10
Testing induction System Testing Non-Functional System Testing – Key activities are Setting Entry/Exit Criteria Balancing Key Resources CPU, Disk, Memory and Network Basic Assumption Made about Resources and Configuration Utilize CPU fully and release on priority job request Utilize Memory completely relinquish when another job requires Resources are added to get better performance – Cost benefit analysis Generate many network packets as long as bandwidth and latency is available More disk space or the complete I/O bandwidth is used Predictable variation in performance, scalability are accepted for different configuration of same product Software Testing Essentials
21
www.qsitglobal.com
Performance Testing
Load Testing – Load the system with activity that simulates the legitimate user activity. – Measure the performance in terms of response time, System crash, System hang, etc,. Example; – Web page testing with 5000 virtual users Volume testing – Subject the program to heavy volumes of data. – Check fields, records and files to see if their sizes can accommodate all expected data. Example – Large volumes of data in client/server applications – Compiler fed large source program to compile – Link editor fed program with thousands of modules. Software Testing Essentials
QSIT Copyright
22
www.qsitglobal.com
Page:11
Testing induction System Testing Reliability Testing ensures 1. No errors or very few errors from repeated transactions 2. Zero downtime 3. Optimum utilization of resources 4. Consistent performance and response time of the product for repeated transactions for a specified time 5. No side-effects after repeated transactions are executed The defects are expressed in 1. Mean time between failures 2. Failure rate 3. Meantime to discover the next K faults
Software Testing Essentials
23
www.qsitglobal.com
System Testing Stress Testing 1. Repetitive tests – Executing repeated tests to ensure that at all times the code works 2.
Concurrency Test – Code is exercised in multiple path simultaneously
3.
Magnitude Test – Amount of Load to be applied to the product to check stress point
4.
Random variation of Load Test – Increase / Decrease the load on the system and check system response. (MTTR – Mean time to recover)
Software Testing Essentials
QSIT Copyright
24
www.qsitglobal.com
Page:12
Testing induction Performance Testing Typical Entry/Exit Criteria for Non-Functional Testing Type of Test
Parameters
Sample Entry Sample Exit
Scalability
Maximum Limits
Scale up to 1 million records and 1000 users
Scale up to 10 Million records 5000 users
Performance
Response Time Throughput Latency
Query for 1000 records – response time less than 3 sec
Query for 10,000 records – response time less than 3 sec
Reliability
Failure per iteration Failure per test duration
Less than 2% when queries run on 1000 records for 24 hours
Less than 0.1% when queries run on 1000 records for 24 hours
Stress
System when stressed beyond limits
Withstand 25 clients login simultaneously for 5 hours in an environment that takes 20 client
Withstand 100 client login simultaneously for 5 hours in an environment that takes 20 client
Software Testing Essentials
25
www.qsitglobal.com
Acceptance Testing / Field Testing
A formal test conducted to determine whether a software system satisfies its user acceptance criteria.
Approach – Conducted with actual environment – Conducted with actual user or User representative. – Carried with a process called Alpha and Beta testing Alpha Testing Test is conducted at developer’s site. The software is used in natural setting with the developer and the user and then recording the errors. Tests are conducted in a controlled environment. Beta testing Test is conducted at one or more customer’s site by the enduser. Developer generally not present. Environment is live and not controlled by the developer.
Software Testing Essentials
QSIT Copyright
26
www.qsitglobal.com
Page:13
Testing induction Acceptance Testing
Assumptions/Pre-Conditions – –
Completed system testing System is free from Critical defects
Purpose – – – – –
Verify system from user perspective Meets the Acceptance criteria User documents are delivered Review with Sponsor and User Plans for the implementation
Software Testing Essentials
27
www.qsitglobal.com
Regression Testing
Why do Regression Testing? –
The Changes or additions work as designed and
–
The Changes or additions do not break something that is already working and should continue to work
There are two types of regression testing in practice –
Regular regression testing Done between test cycles
–
Final regression testing Validate the final build before release
Software Testing Essentials
QSIT Copyright
28
www.qsitglobal.com
Page:14
Testing induction Regression Testing
When to do regression Testing? –
A reasonable amount of initial testing is already carried out
–
A good number of defects have been fixed
–
Defect fixes that can produce side-effects are taken care of
–
May also be performed periodically, as a pro-active measure
Software Testing Essentials
29
www.qsitglobal.com
Regression Testing
How to do Regression Testing –
A well-defined methodology for regression testing
A well defined methodology comprises of following steps 1.
Performing an initial “Smoke” or “Sanity” Test
2.
Understanding the criteria for selecting the test cases
3.
Classifying the test cases into different priorities
4.
A methodology for selecting test cases
5.
Resetting the test cases for test execution
6.
Concluding the results of a regression cycle
Software Testing Essentials
QSIT Copyright
30
www.qsitglobal.com
Page:15
Testing induction Regression Testing 1.
Smoke/Sanity Testing consists of:
•
Identifying the basic functionality that a product must satisfy
•
Designing test cases
•
Packing them into a smoke/sanity build
•
Rerun the build every time the product is build
•
If the build fails, escalating the issue to development team
q1
Software Testing Essentials
31
www.qsitglobal.com
Regression Testing 2. Criteria for selection Regression Test Cases There are two approaches to selecting the test cases First – Select set of regression test that are run for every build or change This approach is not very optimal since –
To cover all fixes the constant set of test will cover all features and tests which are not required may be run every time
–
Fixes and Changes may introduce new defects which may not have test cases
Software Testing Essentials
QSIT Copyright
32
www.qsitglobal.com
Page:16
Slide 31 q1
CM, SRS 15% Code, Design 20% Process 10% documentation 7% People 8% Others 5% qsit, 3/28/2007
Testing induction Regression Testing 2. Criteria for selection Regression Test Cases (Cont. ) Second Select test cases dynamically for each build. This requires knowledge of 1.
Defect fixes and changes made for every build
2.
The ways to test current changes
3.
The impact that the current changes may have
4.
The ways to test other impacted parts
Software Testing Essentials
33
www.qsitglobal.com
Regression Testing 2. Criteria for selection Regression Test Cases (Cont. ) How to select test cases:
Introduce test cases which has introduced maximum defects
Introduce test cases for a functionality in which a changes has been made
Include test cases in which problems are reported
Include test cases which have end-to-end workflow
Include more positive test cases
Include the area which is highly visible to the users
Software Testing Essentials
QSIT Copyright
34
www.qsitglobal.com
Page:17
Testing induction Regression Testing 3. Classifying test cases Based on the importance of customer usage test cases need to be classified into different priorities Priority 0 – Set of sanity test cases which check basic functionality run before Every acceptance build Every Major Release Priority 1 – Uses the basic and normal setup and test cases deliver high project value to both development team and customers Priority 2 – These test cases deliver moderate project value Software Testing Essentials
35
www.qsitglobal.com
Regression Testing 6. Concluding results of regression test cycle Current Results
Previous Results
Conclusion
Remarks
FAIL
PASS
FAIL
Need to improve regression process and code reviews
PASS
FAIL
PASS
This is the expected result of a good regression to say defect fixes work properly
FAIL
FAIL
FAIL
Need to analyze why defect fixes are not working?
PASS (with a work around)
FAIL
Analyze the workaround and if satisfied mark result as PASS
Workarounds also need a good review as they can also create side effects
PASS
PASS
PASS
This pattern of results gives a comfort feeling that there are no side-effects due to defect fixes
Software Testing Essentials
QSIT Copyright
36
www.qsitglobal.com
Page:18