pairwise software test design - what exactly is pairwise coverage? what is 3-way coverage?
Post on 05-Dec-2014
255 Views
Preview:
DESCRIPTION
TRANSCRIPT
Understanding Combinations
(Pairs and Triples)
Each of the end points on this mind map (like “Young” and “Friendly”) are parameter values.
More
ParametersParameter Values
This is what they look like in Hexawise.
More
ParametersParameter Values
When you click on the “Create Tests” button...
More
1
Hexawise generates a set of 2-way tests that contains every single pair of those test parameter values in at least one of the tests.
2
• Medical Devices: D.R. Wallace, D.R. Kuhn, Failure Modes in Medical Device Software: an Analysis of 15 Years of Recall Data, International Journal of Reliability, Quality, and Safety Engineering, Vol. 8, No. 4, 2001. • Browser, Server: D.R. Kuhn, M.J. Reilly, An Investigation of the Applicability of Design of Experiments to Software Testing, 27th NASA/IEEE Software Engineering Workshop, NASA Goddard SFC 4-6 December, 2002 . • NASA database: D.R. Kuhn, D.R. Wallace, A.J. Gallo, Jr., Software Fault Interactions and Implications for Software Testing, IEEE Trans. on Software Engineering, vol. 30, no. 6, June, 2004. • Network Security: K.Z. Bell, Optimizing Effectiveness and Efficiency of Software Testing: a Hybrid Approach, PhD Dissertation, North Carolina State University, 2006.
1 input or 2 inputs (“pairwise)
3 inputs
4, 5 or 6 inputs
Studies show that 84% of defects in production could have been found by testing for every pair of parameter values.
2-way coverage is useful because so many defects can be quickly found by testing all pairs of parameter values.
More
But what does 2-way coverage mean
EXACTLY?
It means this...
More
Every single pair of parameter values appears together in at least one of these six test cases:
More
Stylish and
Tax Collector?
For example...
More
Stylish and
Tax Collector?
Test # 2
That’s a pair of values. So at least one test includes it.
More
Old and
Shabby?
More
Old and
Shabby?
Test # 4
That’s a pair of values. So at least one test includes it.
More
Hostile and
Skydiving Instructor?
More
Hostile and
Skydiving Instructor?
Also test # 4
That’s a pair of values. So at least one test includes it.
More
Youngand
Hostile?
More
Youngand
Hostile?
Test # 6
That’s a pair of values. So at least one test includes it.
More
We could do this for quite a while.
Choose any two parameter values.
Any pair of test inputs you choose will appear in at least one of the 2-way tests.
For every set of Parameter Values entered into Hexawise there is a finite set of pairs that can be tested.
More
There are exactly 24 possible pairs of parameter values in this simple example.
All of the pairs will be tested in at least one test.
More
All 24 of those possible pairs are tested in at least one of the following 6 tests.
• Medical Devices: D.R. Wallace, D.R. Kuhn, Failure Modes in Medical Device Software: an Analysis of 15 Years of Recall Data, International Journal of Reliability, Quality, and Safety Engineering, Vol. 8, No. 4, 2001. • Browser, Server: D.R. Kuhn, M.J. Reilly, An Investigation of the Applicability of Design of Experiments to Software Testing, 27th NASA/IEEE Software Engineering Workshop, NASA Goddard SFC 4-6 December, 2002 . • NASA database: D.R. Kuhn, D.R. Wallace, A.J. Gallo, Jr., Software Fault Interactions and Implications for Software Testing, IEEE Trans. on Software Engineering, vol. 30, no. 6, June, 2004. • Network Security: K.Z. Bell, Optimizing Effectiveness and Efficiency of Software Testing: a Hybrid Approach, PhD Dissertation, North Carolina State University, 2006.
1 input or 2 inputs (“pairwise)
3 inputs
4, 5 or 6 inputs
It is an extremely efficient way to find defects given that: • Studies show that 84% of defects in production could have been
found by testing for every pair of parameter values and • 2-way coverage can be achieved in a relatively small number of tests.
Again, the reason it is a good idea to test all possible pairs is because...
More
But what about coverage of combinations involving 3 or more parameter values?
More
It depends!
Sometimes, a specific combination of 3 values will be included in your set of 2-way tests.
Other times, combinations of 3 values will not be included in your set of tests.
Let’s look at a few examples.
3-way combinations involve 3 specific parameter values.
More
Youngand
Stylish and
Skydiving Instructor?
3-way combinations involve 3 specific parameter values.
More
Youngand
Stylish and
Skydiving Instructor?
Oldand
Stylish and
Skydiving Instructor?
Shabbyand
Hostile and
Tax Collector?
Youngand
Stylish and
Hostile?
Let’s look at a few examples of 3-way combinations andsee if they are covered in our 2-way tests.
More
Youngand
Stylish and
Skydiving Instructor?
This 3-way combination does appear in test # 3
We got lucky.
This 3-way combination does appear in the 2-way tests.
More
Oldand
Stylish and
Skydiving Instructor?
This 3-way combination does not appear in the 2-way tests.
More
Shabbyand
Hostile and
Tax Collector?
This 3-way combination does not appear in the 2-way tests.
More
Youngand
Stylish and
Hostile?
This 3-way combination does not appear in the 2-way tests.
More
If you want to make sure that all possible “triples” like these are covered by at least one of your tests, you will need to create
3-way tests.Let’s now take a look at the 3-way tests from Hexawise that includes at least one test for every possible combinations of “triples.”
These 3-way tests contain at least one test for every triple.
More
Some 3-way combinations are covered in 2-way tests.
All 3-way combinations are covered in 3-way tests.
More
Youngand
Stylish and
Skydiving Instructor?
Oldand
Stylish and
Skydiving Instructor?
Shabbyand
Hostile and
Tax Collector?
Youngand
Stylish and
Hostile?
Test # 6
Test # 3
Test # 8
Test # 4
Reviewing the same 3-way combinations as before, we see that all of them are covered in the 3-way tests. That’s because every single triple is included in the 3-way tests.
More
For additional information, please see:
training.hexawise.com or
help.hexawise.com
top related