Understanding Combinations

(Pairs and Triples)

Each of the end points on this mind map (like “Young” and “Friendly”) are parameter values.

More

ParametersParameter Values

This is what they look like in Hexawise.

More

ParametersParameter Values

When you click on the “Create Tests” button...

More

1

Hexawise generates a set of 2-way tests that contains every single pair of those test parameter values in at least one of the tests.

2

• Medical Devices:  D.R. Wallace, D.R. Kuhn, Failure Modes in Medical Device Software: an Analysis of 15 Years of Recall Data, International Journal of Reliability, Quality, and Safety Engineering, Vol. 8, No. 4, 2001.    • Browser, Server:  D.R. Kuhn, M.J. Reilly, An Investigation of the Applicability of Design of Experiments to Software Testing, 27th NASA/IEEE Software Engineering Workshop, NASA Goddard SFC 4-6 December, 2002 .   • NASA database:  D.R. Kuhn, D.R. Wallace, A.J. Gallo, Jr., Software Fault Interactions and Implications for Software Testing, IEEE Trans. on Software Engineering, vol. 30, no. 6, June, 2004.   • Network Security:  K.Z. Bell, Optimizing Effectiveness and Efficiency of Software Testing: a Hybrid Approach,  PhD Dissertation, North Carolina State University, 2006.  

1 input or 2 inputs (“pairwise)

3 inputs

4, 5 or 6 inputs

Studies show that 84% of defects in production could have been found by testing for every pair of parameter values.

2-way coverage is useful because so many defects can be quickly found by testing all pairs of parameter values.

More

But what does 2-way coverage mean

EXACTLY?

It means this...

More

Every single pair of parameter values appears together in at least one of these six test cases:

More

Stylish and

Tax Collector?

For example...

More

Stylish and

Tax Collector?

Test # 2

That’s a pair of values. So at least one test includes it.

More

Old and

Shabby?

More

Old and

Shabby?

Test # 4

That’s a pair of values. So at least one test includes it.

More

Hostile and

Skydiving Instructor?

More

Hostile and

Skydiving Instructor?

Also test # 4

That’s a pair of values. So at least one test includes it.

More

Youngand

Hostile?

More

Youngand

Hostile?

Test # 6

That’s a pair of values. So at least one test includes it.

More

We could do this for quite a while.

Choose any two parameter values.

Any pair of test inputs you choose will appear in at least one of the 2-way tests.

For every set of Parameter Values entered into Hexawise there is a finite set of pairs that can be tested.

More

There are exactly 24 possible pairs of parameter values in this simple example.

All of the pairs will be tested in at least one test.

More

All 24 of those possible pairs are tested in at least one of the following 6 tests.

• Medical Devices:  D.R. Wallace, D.R. Kuhn, Failure Modes in Medical Device Software: an Analysis of 15 Years of Recall Data, International Journal of Reliability, Quality, and Safety Engineering, Vol. 8, No. 4, 2001.    • Browser, Server:  D.R. Kuhn, M.J. Reilly, An Investigation of the Applicability of Design of Experiments to Software Testing, 27th NASA/IEEE Software Engineering Workshop, NASA Goddard SFC 4-6 December, 2002 .   • NASA database:  D.R. Kuhn, D.R. Wallace, A.J. Gallo, Jr., Software Fault Interactions and Implications for Software Testing, IEEE Trans. on Software Engineering, vol. 30, no. 6, June, 2004.   • Network Security:  K.Z. Bell, Optimizing Effectiveness and Efficiency of Software Testing: a Hybrid Approach,  PhD Dissertation, North Carolina State University, 2006.  

1 input or 2 inputs (“pairwise)

3 inputs

4, 5 or 6 inputs

It is an extremely efficient way to find defects given that: • Studies show that 84% of defects in production could have been

found by testing for every pair of parameter values and • 2-way coverage can be achieved in a relatively small number of tests.

Again, the reason it is a good idea to test all possible pairs is because...

More

But what about coverage of combinations involving 3 or more parameter values?

More

It depends!

Sometimes, a specific combination of 3 values will be included in your set of 2-way tests.

Other times, combinations of 3 values will not be included in your set of tests.

Let’s look at a few examples.

3-way combinations involve 3 specific parameter values.

More

Youngand

Stylish and

Skydiving Instructor?

3-way combinations involve 3 specific parameter values.

More

Youngand

Stylish and

Skydiving Instructor?

Oldand

Stylish and

Skydiving Instructor?

Shabbyand

Hostile and

Tax Collector?

Youngand

Stylish and

Hostile?

Let’s look at a few examples of 3-way combinations andsee if they are covered in our 2-way tests.

More

Youngand

Stylish and

Skydiving Instructor?

This 3-way combination does appear in test # 3

We got lucky.

This 3-way combination does appear in the 2-way tests.

More

Oldand

Stylish and

Skydiving Instructor?

This 3-way combination does not appear in the 2-way tests.

More

Shabbyand

Hostile and

Tax Collector?

This 3-way combination does not appear in the 2-way tests.

More

Youngand

Stylish and

Hostile?

This 3-way combination does not appear in the 2-way tests.

More

If you want to make sure that all possible “triples” like these are covered by at least one of your tests, you will need to create

3-way tests.Let’s now take a look at the 3-way tests from Hexawise that includes at least one test for every possible combinations of “triples.”

These 3-way tests contain at least one test for every triple.

More

Some 3-way combinations are covered in 2-way tests.

All 3-way combinations are covered in 3-way tests.

More

Youngand

Stylish and

Skydiving Instructor?

Oldand

Stylish and

Skydiving Instructor?

Shabbyand

Hostile and

Tax Collector?

Youngand

Stylish and

Hostile?

Test # 6

Test # 3

Test # 8

Test # 4

Reviewing the same 3-way combinations as before, we see that all of them are covered in the 3-way tests. That’s because every single triple is included in the 3-way tests.

More

For additional information, please see:

training.hexawise.com or

help.hexawise.com