a test design poster for smarter testing · a test design poster for smarter testing slide 4...
TRANSCRIPT
1
A Test Design Poster
For Smarter Testing
Peter Zimmerer
Siemens AG
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing © Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Slide 2
A Test Design Poster
for Smarter Testing
EuroSTAR 2005Copenhagen, Denmark
Peter Zimmerer
Senior Principal Engineer
Siemens AG
Corporate Technology Software & Engineering 1
D-81730 Munich, Germany
http://www.siemens.com/research-and-development/
http://w4.siemens.com/ct/
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 3
Contents
• Introduction
• Test design methods
Here: methods, paradigms, techniques, styles, and ideas to create a test.
• Examples and references
• Problem statement
• Poster Test Design Methods on One Page
• Guidelines and experiences
• Summary
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 4
Introduction – Test design methods
• Good test design, i.e. high-quality tests,
is very important.
• There are many many different test
design methods and techniques.
• Static, dynamic
• Black-box, white-box, grey-box
• Based on fault model, experience, exploratory
• Statistical (user profiles), random (monkey)
• The tester„s challenge is to adequately
combine these methods dependent
on the given problem, domain, and
requirements.
• This is art as well!
• Black-box test design methods are
often based on models.
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 5
Some systematic methods for test design
• Black-box (interfaces, data, models)
• Traceability matrix
• Use case-based testing, scenario testing
• Equivalence class partitioning
• Boundary value analysis
• State-based testing
• Cause-effect graphing
• Decision tables
• Combinatorial testing
• White-box (internal structure, paths)
• Control flow testing
• Data flow testing
• Selection, usage and
applicability depends on
the
• specific domain
• used software technology
• test requirements:
required test intensity,
quality criteria, risks
• existing test basis:
specifications, documents,
models
• project factors: constraints
and opportunities
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 6
Example – There are always too many test cases ...
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 7
Examples – Demo
• Microsoft PowerPoint
• Microsoft Word
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 8
Some references
• Classical testing books cover test design to some extent.
• Boris Beizer: Software Testing Techniques
• Lee Copeland: A Practitioner's Guide to Software Test Design
• James Whittaker: How to Break Software
• James Whittaker, Herbert Thompson: How to Break Software
Security
• Standard for Software Component Testing by the British
Computer Society Specialist Interest Group in Software Testing
(BCS SIGIST) (see http://www.testingstandards.co.uk/)
• There are many different training opportunities by different
providers.
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 9
Problem statement
• Starting from a risk-based testing strategy an adequate test design
is the key for effective and efficient testing.
• Automation of bad test cases is a waste of time and money!
• There are many different test design methods around for a long
time (perhaps too many?) and a lot of books explain them in detail.
• There are different
• categorizations, classifications, and dimensions
• interpretations and understandings
of test design methods which does not simplify their usage …
• So, when we look into practice we can see that often there is quite
limited usage of these test design methods at all.
• What are the reasons behind that?
• How can we overcome this and improve our testing approaches?
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 10
Possible reasons and consequences (1)
• Developers do not have/spend time to read a book on testing.
• Missing the big picture.
• Information is not available at a glance.
• Which test design methods are there at all?
• Which test design method should I use in which context?
• A specific test design method is not “available” when needed.
• A specific test design method is too detailed or too complicated
to be used in practice.
• Often the focus is on depth and on perfectionism.
Remark: But that can be especially required, e.g. in a safety-
critical environment!
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 11
Possible reasons and consequences (2)
• Test design tools are typically focused and implement only a few
test design methods, e.g.:
• ATD – Automated Test Designer (AtYourSide Consulting,
http://www.atyoursideconsulting.com/):
Cause-effect graphing
• BenderRBT (Richard Bender, http://www.benderrbt.com/):
Cause-effect graphing, quick design (orthogonal pairs based test engine)
• CaseMaker (Diaz & Hilterscheid, http://www.casemaker.de/):
Equivalence classes, cause-effects and combinations, boundary value
analysis, error guessing, pairwise combinations
• CTE (Razorcat, http://www.ats-software.de/):
Classification tree editor
• Reactis (Reactive Systems, http://www.reactive-systems.com/):
Generation of test data from, and validation of, Simulink and Stateflow
models
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 12
Poster Test Design Methods on One Page (1)
• Idea:
Systematic, structured and categorized overview about different
test design methods on one page.
• Focus more on using an adequate set of test design methods than
on using only one single test design method in depth/perfection.
• Focus more on concrete usage of test design methods
than on defining a few perfect test design methods in detail
which are not used then in the project.
• Focus more on breadth instead on depth.
• Do not miss breadth because of too much depth.
• Do not miss the exploratory, investigative art of testing.
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 13
Poster Test Design Methods on One Page (2)
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 14
Poster Test Design Methods on One Page (3)
• Categories of test design methods are orthogonal and independent
in some way but should be combined appropriately.
• The selection of the used test design methods depends on many
factors, e.g.:
• Requirements of the system under test and the required quality
• Requirements for the tests – quality of the tests,
i.e. the required intensity and depth of the tests
• Testing strategy: effort/quality of the tests, distribution of the testing in the
development process
• Existing test basis: specifications, documents, models
• Problem to be tested (domain) or rather the underlying question (use case)
• System under test or component under test
• Test level/Test step
• Used technologies (software, hardware)
• Suitable tool support: for some methods absolutely required
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 15
Poster Test Design Methods on One Page (4)
• The effort/difficulty for the test design methods or rather the
resulting test intensity is subdivided into 5 levels:
• 1 very low, simple
• 2 low
• 3 means
• 4 high
• 5 very high, complex
• This division into levels is dependent on the factors given above
for the selection of the test design methods and therefore can
only be used as a hint and guideline.
• A test design method also may be used continuously from
“intuitive use” up to “100% complete use” as required.
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 16
Example:
Requirement-based with traceability matrix
• In addition describe every test design method on one page to
explain their basic message and intention.
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 17
Example: Equivalence class partitioning and
boundary value analysis with 2 parameters
b
a
b
b
a a
max
min
maxmin
b
a
b
b
a a
max
min
maxminb
a
b
b
a a
max
min
maxmin
# test cases for n parameters and
one valid equivalence class: 4n + 1
# test cases for n parameters and
one valid and invalid equivalence
class: 6n + 1
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 18
Guidelines and experiences
• Pick up this poster and
• give it to every developer and tester in your team or
• put it on the wall in your tester office or
• make it the standard screensaver or desktop background for all team
members or …
• Use the poster as a checklist for existing test design methods.
• Selection of test design methods is dependent on the context!
So, you should adapt the poster to your specific needs.
• The poster increases visibility and importance of test design
methods, especially also for developers to improve unit testing.
• The poster results in a closer cooperation of testers and
developers: you have something to talk about ...
C O
R P
O R
A T
E
T E
C H
N O
L O
G Y
© Siemens AG, CT SE 1, Peter ZimmererNovember 29, 2005
Software &EngineeringDevelopmentTechniques
A Test Design Poster for Smarter Testing Slide 19
Summary
• There exist many different methods for adequate test design.
• When looking into practice often these test design methods are
used only sporadically and in a non-systematical way.
• A test design poster containing a systematic, structured and
categorized overview about test design methods will help you to
really get them used in practice in your projects.
• Do not miss breadth because of too much depth.
• This will result in a better and smarter testing overall which will
pay off very soon.
• Use this poster after this conference as one mean to improve the
usage of test design methods in your projects!!!