Limited Business Circulation
Testing Essentials Unit Test Design specification QSIT- Quality Solutions for Information Technology Pvt. Ltd., A 806 & 807, Mittal Tower, BANGALORE- 560 001,INDIA URL: www.qsitglobal.com PH: +91 80 41134334 Fax: +91 80 25597445
Software Testing Essentials
1
www.qsitglobal.com
Exercise: Which Group is Better?
Testing Group 1
Testing Group 2 From Marick, Classic Testing Mistakes Software Testing Essentials
QSIT Copyright
Function Function Function Function Function Total
A B C D E
Function Function Function Function Function Total
A B C D E
Found prerelease 100 0 0 0 0 100
Two groups test the same program. • The functions are equally important • The bugs are equally significant
50 6 6 6 6 74
2
www.qsitglobal.com
Page:1 Version 1.1/Testing Essentials/001
Limited Business Circulation
Exercise: Which Group is Better?
F u n ctio n F u n ctio n F u n ctio n F u n ctio n F u n ctio n T o tal
A B C D E
F u n ctio n F u n ctio n F u n ctio n F u n ctio n F u n ctio n T o tal
A B C D E
F o u n d p re re le a se 1 00 0 0 0 0 1 00
Found la te r 0 12 12 12 12 48
T o ta l
50 6 6 6 6 74
50 6 6 6 6 74
10 0 12 12 12 12 14 8
Software Testing Essentials
10 0 12 12 12 12 14 8
3
www.qsitglobal.com
Inputs and Outputs of Test Design Specification Business Requirements
High Level Test Design
Define Process
Design Specification Detailed Test Design Specification
Adequecy Criteria
Define Techniques
Testability
Types of Testing
Testing Requirements
Completion Criteria
Test Coverage Software Testing Essentials
QSIT Copyright
4
www.qsitglobal.com
Page:2 Version 1.1/Testing Essentials/001
Limited Business Circulation
Test Data Adequacy Criteria
Test Data Adequacy Criteria
Code Based Testing
Requirement Based Testing
•Have I
•Have I
•Tested
•Thought
•Exercised
•Applied all inputs
•Forced
•Completely explored
•Found
•Run all the Scenarios
Software Testing Essentials
5
www.qsitglobal.com
Approach
Study Business Requirements Arrive at Environmental Requirements Identify test related Risks Decide Automation Requirements Prepare Test Documents Plan for Test Completion Analyze Track changes Review Test design effectiveness
Software Testing Essentials
QSIT Copyright
6
www.qsitglobal.com
Page:3 Version 1.1/Testing Essentials/001
Limited Business Circulation
Testing Techniques
Code Based (White Box testing/Structured) – Statement Coverage – Edge Coverage – Condition Coverage – Path Coverage Code Complexity – Cyclomatic Complexity
Software Testing Essentials
Specification Based (Black Box Testing/ Functional) – Boundary Value Analysis – Equivalence Partitioning – Decision table – Cause Effect Graphing – Control Flow Based Criteria – Data Flow based criteria
7
www.qsitglobal.com
Testing Techniques (Contd…)
Domain Specific Technique – Object Oriented Testing – Embedded Testing – Client Server – Web Applications Usage Based – Statistical testing (Musa’s)SRET – Special Testing Fault Based – Error Guessing – Mutation – Fault Seeding
Software Testing Essentials
QSIT Copyright
8
www.qsitglobal.com
Page:4 Version 1.1/Testing Essentials/001
Limited Business Circulation
Test Exit/Completion Criteria
Executed at least once? Requirements been tested or verified? Test Documentation Documents updated and submitted to Configuration Manager Test Summary Report
Software Testing Essentials
9
www.qsitglobal.com
Review: Defining the Test Approach
A good test approach is: – – – – –
Diversified Risk-focused Product-specific Practical Defensible
The techniques you apply should follow your test approach
Software Testing Essentials
QSIT Copyright
10
www.qsitglobal.com
Page:5 Version 1.1/Testing Essentials/001
Limited Business Circulation
Dimensions of Test Techniques
Think of the testing you do in terms of five dimensions: –
Testers: who does the testing.
–
Coverage: what gets tested.
–
Potential problems: why you're testing (what risk you're testing for).
–
Activities: how you test.
–
Evaluation: how to tell whether the test passed or failed.
Test techniques often focus on one or two of these, leaving the rest to the skill and imagination of the tester.
Software Testing Essentials
11
www.qsitglobal.com
Test Techniques – Discussion
There are as many as 200 published testing techniques. Many of the ideas are overlapping, but there are common themes.
Similar sounding terms often mean different things, e.g.: – – –
User testing Usability testing User interface testing
What are the differences among these techniques?
Software Testing Essentials
QSIT Copyright
12
www.qsitglobal.com
Page:6 Version 1.1/Testing Essentials/001
Limited Business Circulation
Test Techniques—Dominant Test Approaches
Of the 200+ published Functional Testing techniques, there are ten basic themes.
They capture the techniques in actual practice.
In this course, we call them: – Function testing – Equivalence analysis – Specification-based testing – Risk-based testing – Stress testing – Regression testing – Exploratory testing – User testing – Scenario testing – Stochastic or Random testing
Software Testing Essentials
13
www.qsitglobal.com
“So Which Technique Is the Best?” •
•
Each has strengths and weaknesses
Technique A
Think in terms of complement
Technique H
•
There is no “one true way”
•
Mixing techniques can improve coverage
Technique B
Testers Coverage Potential problems Activities Evaluation
Technique G
Technique F
Technique C Technique D
Technique E Software Testing Essentials
QSIT Copyright
14
www.qsitglobal.com
Page:7 Version 1.1/Testing Essentials/001
Limited Business Circulation
Black Box Techniques What is black box testing? – Knows only what the software is supposed to do – Doesn’t know how it operates inside Why named as black box testing? – You cannot see inside the box
Software Testing Essentials
15
www.qsitglobal.com
Black Box Techniques Who does black box testing? – Testers – Business Analysis – Clients When you do? – Once the functionality is developed Why not by developers? – Very difficult to break one’s own code – To test the software from end-user perspective – Fresh eyes always help
Software Testing Essentials
QSIT Copyright
16
www.qsitglobal.com
Page:8 Version 1.1/Testing Essentials/001
Limited Business Circulation
Requirements Based - Black Box Techniques
Low Level Techniques – –
Boundary value analysis Equivalence partitioning
High Level techniques –
Express requirements in simple formal notations like Decision table State machine Cause-effect Graph Use case
The notation allows generation of scenarios – Different test cases for every scenario – Makes requirements verifiable
Software Testing Essentials
17
www.qsitglobal.com
Boundary Value Analysis 15
6
Less than 6
A Black Box Testing Method Complements to Equivalence partition BVA leads to a selection of test cases that exercise bounding values Design test cases test – Min values of an input – Max values of an input – Just above and below input range
Software Testing Essentials
QSIT Copyright
More than 15
18
www.qsitglobal.com
Page:9 Version 1.1/Testing Essentials/001
Limited Business Circulation
Boundary Value Analysis
Two types of BVA: – Range Above and below Range – Value Above and below min and max number
Boundary Value Analysis: – Large number of errors tend to occur at boundaries of the input domain. – BVA leads to selection of test cases that exercise boundary values. – BVA complements equivalence partitioning. – Rather than select any element in an equivalence class, select those at the ''edge' of the class.
Software Testing Essentials
19
www.qsitglobal.com
Boundary Value Analysis
Examples: –
For a range of values bounded by a and b, test (a-1), a, (a+1), (b-1), b, (b+1).
–
If input conditions specify a number of values n, test with (n1), n and (n+1) input values.
–
Apply 1 and 2 to output conditions (e.g., generate table of minimum and maximum size).
–
If internal program data structures have boundaries (e.g., buffer size, table limits), use input data to exercise structures on boundaries.
Software Testing Essentials
QSIT Copyright
20
www.qsitglobal.com
Page:10 Version 1.1/Testing Essentials/001
Limited Business Circulation
Equivalence Partitioning
Divide the input domain into classes of data for which test cases can be generated. Attempting to uncover classes of errors. Divides the input domain of a program into classes of data Derives test cases based on these partitions An equivalence class is a set of valid or invalid states of input Test case design is based on equivalence classes for an input domain.
Software Testing Essentials
Invalid Inputs
Valid Inputs
SYSTEM`
Output
21
www.qsitglobal.com
Equivalence Partitioning
Invalid
Less than 6
Between 6 and 15
Invalid
More than 15
Useful in reducing the number of Test Cases required It is very useful when the input/output domain is amenable to partitioning
Software Testing Essentials
QSIT Copyright
Valid Range
22
www.qsitglobal.com
Page:11 Version 1.1/Testing Essentials/001
Limited Business Circulation
Equivalence Partitioning
Here test cases are written to uncover classes of errors for every input condition. Equivalence classes are: Range Upper bound + 1 Lower bound – 1 Within bound Value Maximum length + 1 Minimum length – 1 Valid value and Valid length Invalid value Set In-set Out-of-set Boolean True False Software Testing Essentials
23
www.qsitglobal.com
Equivalence Partitioning
Equivalence Partitioning partitions the data to partition of a set
Partition refers to collection of mutually disjoint subsets whose union is the entire set
Chose one data element from each partitioned set
The KEY is the choice of equivalence relation!
EC based testing allows – to have a sense of complete testing – Helps avoid redundancy
Software Testing Essentials
QSIT Copyright
24
www.qsitglobal.com
Page:12 Version 1.1/Testing Essentials/001
Limited Business Circulation
Exercise: BVA, EP A Life Insurance company in USA has base premium of $0.50 for all ages. Anyone under the age of 25 has to pay an additional premium of $1.65 and anyone between the age of 25 to 65 has to pay an additional $2.87 and anyone who is over 65 has to pay an additional premium of $ 6.0 per month. Derive Test cases to test the logic using the low level black box technique. Note: The Normal Life expectancy in US is considered to be 105 Years.
Software Testing Essentials
25
www.qsitglobal.com
Exercise: BVA, EP Solution EP for the scenario are: 0-25, 26-65, 66-105
BVA for the scenario are: {0-25} {-1,0, 1, 24, 25, 26} {26-65} {25, 26,27, 64,65,66} {66-105} {65,66,67, 104,105,106}
Software Testing Essentials
QSIT Copyright
26
www.qsitglobal.com
Page:13 Version 1.1/Testing Essentials/001
Limited Business Circulation
High Level Technique – Decision Table
Identify the decision variable and conditions.
Identify resultant actions to be selected or controlled
Identify which actions should be produced in response to particular combinations of conditions.
Derive the logic function for the model to validate its completeness and consistency
Exposes errors in specification
Columns in decision table are converted to test cases.
Similar to Condition Coverage used in White Box Testing Value 1
Value 2 Value 3
Login
√
√
X
Password
X
√
X
Successful Login
X
√
X
Unsuccessful Login
√
X
√
Warning Message
√(W)
NA
√(W)
Software Testing Essentials
CONDITION
ACTION
27
www.qsitglobal.com
High Level Technique – State Transition Diagram
State Transition Diagram
State 1
Input Event Transition
Action Output
State 2
Software Testing Essentials
QSIT Copyright
28
Events are caused by input Actions are likely to cause output Limited to their ability to detect some types of faults Detect the most obvious incorrect transitions and outputs Will not detect more subtle faults that are only detectable through exercising sequence of transitions.
www.qsitglobal.com
Page:14 Version 1.1/Testing Essentials/001
Limited Business Circulation
Example Manage_display_changes specifications The software responds to input requests to change display mode for a time display device. The display mode can be set to one of four values: Two corresponding to displaying either the time or date. The other two when altering either the time or the date. Four possible input requests Change mode Reset Time set Date set Software Testing Essentials
29
www.qsitglobal.com
State Transition Diagram - Example Change Mode (CM) : Activation of this shall cause the display mode to move between “display time (T)” and “display date (D). Reset (R) : If the display mode is set to T or D, then a “reset” shall cause the display mode to be set to “alter time (AT)” or “alter date “AD” modes. Time Set (TS) : Activation of this shall cause the display mode to return to T from AT. Date Set (DS) : Activation of this shall cause the display mode to return to T from AD
Software Testing Essentials
QSIT Copyright
30
www.qsitglobal.com
Page:15 Version 1.1/Testing Essentials/001
Limited Business Circulation
State Transition Diagram - Example STD for Manage_display_changes R AT Changing Time (S3)
Displaying Time (S1) TS
D
CM
T
CM
T
R AD Displaying Date (S2)
Changing Date (S4) DS D
Software Testing Essentials
31
www.qsitglobal.com
State Transition Diagrams Start Sate
S1
S2
S1
S3
S2
S4
S1
Input
-
-
-
-
-
-
-
Event
CM
CM
R
TS
R
DS
Action
Rest Date Mode Time Mode Time
Rest time
Rest Date
Rest Date
Date Display
Time Display
New Value New for time Time
New value for New date Date
Error Error Error Error Error Error Error Error Message Message Message Message Message Message Message Message
Output
End State
S2
S1
S3
S4
S1
Software Testing Essentials
QSIT Copyright
S1
S2
S1
S2
S2
S3
S3
S4
S4
TS
DS
TS
DS
CM
R
CM
R
Error
Error
Error
Error
Error
Error
Error
Error
S1
32
S2
S2
S3
S3
S4
S4
‌
‌
www.qsitglobal.com
Page:16 Version 1.1/Testing Essentials/001
Limited Business Circulation
OAT – Orthogonal Array Technique. Consider this:
11 Check Boxes in Font dialog of MS Word for various effects such as superscript, strikethrough, emboss, etc.
Obviously these effects have impact on how the characters in a particular font are displayed
Can be used in multiple combinations such as Strikethrough + Subscript + Emboss. Software Testing Essentials
33
www.qsitglobal.com
Example Cont.
The total number of combinations of effects in this case would 211 or 2048 combinations
This doesn't include different font types, styles, etc.
Number of combinations increase rapidly as additional dependent parameters are included resulting in potential explosion in the number of tests
Software Testing Essentials
QSIT Copyright
34
www.qsitglobal.com
Page:17 Version 1.1/Testing Essentials/001
Limited Business Circulation
Testing Challenges‌
Industry has a lot of evidence to suggest that most software defects occur from simple interactions between the variables of 2 parameters
How to choose the combinations out of all the possibilities from a risk based perspective where it may not be feasible to test all possible combinations?
Software Testing Essentials
35
www.qsitglobal.com
Orthogonal Array Technique OATS is a systematic, Statistical way of testing pair-wise interactions.
Useful for integration testing of software components in OO systems.
Useful for testing of combinations of configurable options such as Web page
Orthogonal arrays are two dimensional arrays of numbers which possesses the interesting quality that by choosing any two columns in the array, we receive an array of all the pair-wise combinations of values in the array.
Software Testing Essentials
QSIT Copyright
36
www.qsitglobal.com
Page:18 Version 1.1/Testing Essentials/001
Limited Business Circulation
High Level Technique – Orthogonal Array Testing
Orthogonal arrays are named in the pattern – LRuns (Levels Factors) Runs: Number of rows in the array (No.of test Cases) Factors: Number of columns in an array (No. of variables) Levels: Maximum number of values that can be taken by any single factor.
Software Testing Essentials
37
www.qsitglobal.com
How to use this technique The OATS technique is simple and straightforward. The steps are outlined below. 1. Decide how many independent variables will be tested for interaction. This will map to the Factors of the array.
2. Decide the maximum number of values that each independent variable will take on. This will map to the Levels of the array
3. Find a suitable orthogonal array with the smallest number of Runs. A suitable array is one that has at least as many Factors as needed from Step 1 and has at least as many levels for each of those factors as decided in Step 2.
4. Map the Factors and values onto the array
5. Choose values for any "left over" Levels
6. Transcribe the Runs into test cases, adding any particularly suspicious combinations that aren't Software Testing Essentials
QSIT Copyright
38
www.qsitglobal.com
Page:19 Version 1.1/Testing Essentials/001
Limited Business Circulation
A Simple Example Consider a web page with three distinct sections Top, Middle, and Bottom that can be individually shown or hidden by the user. You wish to test the interactions of the different sections. Following the instructions laid out previously, let's create a test set for the system.
Software Testing Essentials
39
www.qsitglobal.com
A Simple Example
There are three independent variables (the sections of the page).
2. Each variable can take on two values (hidden or visible).
3. An L4(23) orthogonal array will do the trick -- two levels for the values and three factors for the variables. Note that the number of runs is not necessary to pick an appropriate array.
4. Mapping the values onto the array would look like Shown in the Figure where Hidden=0 and Visible=1:
Software Testing Essentials
QSIT Copyright
40
www.qsitglobal.com
Page:20 Version 1.1/Testing Essentials/001
Limited Business Circulation
A Simple Example OA before mapping factors
Run Run Run Run
1 2 3 4
Factor 1 0 0 1 1
Factor 2 0 1 0 1
Factor 3 0 1 1 0
OA after mapping factors
Test Test Test Test
1 2 3 4
Top Hidden Hidden Visible Visible
Software Testing Essentials
Middle Hidden Visible Hidden Visible
41
Bottom Hidden Visible Visible Hidden
www.qsitglobal.com
A Simple Example
5. There are no "left over" Levels. In other words, there is a value mapped to every level in the array.
6. Taking the test case values from each run, you end up with four test cases. That is all that is needed to test all of the pair-wise interactions amongst the three variables. The test cases might transcribe to:
A. Display the home page and hide all sections. B. Display the home page and show all but the Top section. C. Display the home page and show all but the Middle section. D. Display the home page and show all but the Bottom section.
Software Testing Essentials
QSIT Copyright
42
www.qsitglobal.com
Page:21 Version 1.1/Testing Essentials/001
Limited Business Circulation
“Note” Note that not all of the possible combinations are tested. It would take eight test cases to test all of the combinations. You can add test cases that are particularly suspicious, but don't show up in the pair-wise list. For example, You might want to add a test case to display all sections or only the middle section to the list above if you felt there was a strong possibility of error with those particular combinations.
Software Testing Essentials
43
www.qsitglobal.com
An Example That Doesn't Fit the Array Let's look at an example that doesn't fit perfectly into any available array. This example takes into account an objectoriented system that contains • A client class (C1) with two subclasses (C2 and C3). These client classes interact with a server class hierarchy consisting of class S1 with subclasses S2 and S3. The server class contains a method foo() which takes an instance of class M1 as a parameter. M1 has two subclasses, M2 and M3. Figure below depicts the classes involved.
Software Testing Essentials
QSIT Copyright
44
www.qsitglobal.com
Page:22 Version 1.1/Testing Essentials/001
Limited Business Circulation
An Example That Doesn't Fit the Array To test all combinations of the classes involved, it would take 27 test cases (three clients that can each send three messages to three servers -- 3 x 3 x 3 = 27). That doesn't seem outrageous, but this assumes that the method foo() can be tested with only a single test case. In most circumstances, it will take many test cases to test a particular method. Also, this interaction is probably a very small portion of the entire system being tested. Using the OATS technique can significantly reduce the number of test cases.
Software Testing Essentials
45
www.qsitglobal.com
An Example That Doesn't Fit the Array 1. There are three independent variables (the client, the server, and the message class) 2. Each variable can take on three values. 3. Ideally, we would us an array that contains three levels and three factors (an L?(33) OA). However, no such published array exists. Therefore, we need to look for the smallest array that will handle our problem. An L9(34) orthogonal will work. It has the three levels for the values and four factors is more than enough for the three variables. 4. Mapping the values onto the array would look like Figure 4 where: A. For Client, C1=0; C2=1; C3=2. B. For Server, S1=0; S2=1; S3=2. C. For Message, M1=0; M2=1; M3=2.
Software Testing Essentials
QSIT Copyright
46
www.qsitglobal.com
Page:23 Version 1.1/Testing Essentials/001
Limited Business Circulation
An Example That Doesn't Fit the Array
OA before mapping factors Factor 1 Run Run Run Run Run Run Run Run Run
1 2 3 4 5 6 7 8 9
0 0 0 1 1 1 2 2 2
Factor 2
Factor 3
0 1 2 0 1 2 0 1 2
Software Testing Essentials
Factor 4
0 1 2 1 2 0 2 0 1
47
0 2 1 1 0 2 2 1 0
www.qsitglobal.com
An Example That Doesn't Fit the Array
OA after mapping factors
Test Test Test Test Test Test Test Test Test
1 2 3 4 5 6 7 8 9
Client C1 C1 C1 C2 C2 C2 C3 C3 C3
Software Testing Essentials
QSIT Copyright
Server S1 S2 S3 S1 S2 S3 S1 S2 S3
48
Message M1 M2 M3 M2 M3 M1 M3 M1 M2
www.qsitglobal.com
Page:24 Version 1.1/Testing Essentials/001
Limited Business Circulation
An Example That Doesn't Fit the Array 5. There are no "left over" Levels. However, you'll notice that there was an extra Factor in the original array. This factor can simply be ignored; it does not change the properties of the test set generated from the array. You still get an even distribution of the pair-wise combinations. 6. Taking the test case values from each run, you end up with nine test cases. As mentioned before, these nine combinations might map to a larger set of test cases that must be executed against each of the nine combinations.
Software Testing Essentials
QSIT Copyright
49
www.qsitglobal.com
Page:25 Version 1.1/Testing Essentials/001