model based testing
DESCRIPTION
Model Based Testing. Texas A&M University Department of Computer Science Sameh S. Sharkawi. Outline. Introduction What is MBT Motivation behind MBT How MBT Works Benefits of MBT Problems of MBT Current State of MBT in Industry Software Tools for MBT Conclusion References. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/1.jpg)
Model Based Testing
Texas A&M University
Department of Computer Science
Sameh S. Sharkawi
![Page 2: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/2.jpg)
Outline Introduction
What is MBT Motivation behind MBT
How MBT Works Benefits of MBT Problems of MBT Current State of MBT in Industry Software Tools for MBT Conclusion References
![Page 3: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/3.jpg)
Introduction (What is MBT) Definition
“Model-based testing refers to software testing where test cases are derived in whole or in part from a model that describes some (usually functional) aspects of the system under test (SUT) .” *
* Wikipedia Definition of Model Based Testing.
![Page 4: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/4.jpg)
Introduction (What is MBT Contd.) MBT is an approach in which you define the
behavior of a system in terms of actions that change the state of the system.
Such a model of the system results in a well-defined Finite State Machine (FSM) which helps us to understand and predict the system’s behavior
![Page 5: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/5.jpg)
Introduction (Motivation behind MBT) Testing is expensive
30-50% of development costs Poor testing is VERY expensive
Down time Maintenance costs Rework Law suits
Model Based Testing promises Increased effectiveness of testing Similar or decreased costs Reuse of design artifacts
![Page 6: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/6.jpg)
Introduction (Motivation behind MBT Downtime Costs) Brokerage operations $6,450,000 Credit card authorization $2,600,000 Ebay (1 outage 22 hours) $225,000 Amazon.com $180,000 Package shipping services $150,000 Home shopping channel $113,000 Catalog sales center $90,000 Airline reservation center $89,000 Cellular service activation $41,000 On-line network fees $25,000 ATM service fees $14,000
![Page 7: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/7.jpg)
How MBT Works
![Page 8: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/8.jpg)
How MBT Works? The model is usually an
abstract, partial presentation of the system under test's desired behavior.
The test cases derived from this model are functional tests on the same level of abstraction as the model.
These test cases are collectively known as the abstract test suite
![Page 9: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/9.jpg)
How MBT Works? Contd. The abstract test suite cannot
be directly executed against the system under test because it is on the wrong level of abstraction.
Therefore an executable test suite must be derived from the abstract test suite that can communicate with the system under test.
This is done by mapping the abstract test cases to concrete test cases suitable for execution.
![Page 10: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/10.jpg)
How MBT Works? Contd. Deriving Test Cases Algorithmically
Test Case Generation by Theorem Proving Test Case Generation by Constraint Logic
Programming Test Case Generation by Model Checking Test Case Generation by Symbolic Execution
![Page 11: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/11.jpg)
How MBT Works? Contd.
Spec Explorer (UW-MSR Summer Institute '04)
Model
Implementation
Test OracleTest Cases
Run Providesactual results for
Provides expected results forGenerates
PassFail
![Page 12: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/12.jpg)
Benefits of MBT
![Page 13: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/13.jpg)
Benefits of MBT Starting from specification
Involves testers early in the development process Teams testers with developers Forces testability into product design
Building the test interface Finds design and specification bugs - before code exists The model is the test plan - and is easily maintained Automated test suite generation Coverage is guaranteed - increases testing thoroughness Zero test suite maintenance costs
Automated test suite execution Finds code and interface bugs Includes a framework for the testing of distributed
applications Reduces test execution costs
![Page 14: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/14.jpg)
Benefits of MBT Contd. These benefits can be summarized as:
Machine Time
![Page 15: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/15.jpg)
Benefits of MBT Contd. Machines
A typical test engineer Earns €41 500 ($50 000 USD) per year Works 40 to 50 hours per week
A typical test machine Costs €250 ($300 USD) per year to buy and operate Can work 100 hours per week
![Page 16: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/16.jpg)
Benefits of MBT Contd.
react.cs.uni-sb.de/mbt2006/talks/mbt4masses.pdf
![Page 17: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/17.jpg)
Problems of MBT
![Page 18: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/18.jpg)
Problems of MBT
![Page 19: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/19.jpg)
Problems of MBT Contd.
![Page 20: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/20.jpg)
Problems of MBT Contd.
![Page 21: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/21.jpg)
Problems of MBT Contd.
![Page 22: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/22.jpg)
Problems of MBT Contd.
Process shift Up front investment in test
Personnel shift Higher education and sophistication
Spec Explorer (UW-MSR Summer Institute '04)
![Page 23: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/23.jpg)
Current State of MBT in Industry
![Page 24: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/24.jpg)
Current State of MBT in Industry IBM Research
Most projects were done by PhD Holders Very Little was successful (Just in the Lab, never
actually went into market) Too Complicated to use and analyze
Microsoft Research The Spec# programming system is a new attempt
at a more cost effective way to develop and maintain high-quality software
![Page 25: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/25.jpg)
Current State of MBT in Industry Spec# is a computer language which
includes and extends C# Adds pre/post conditions, contracts Adds high-level data types with convenient
notations Adds logical quantifiers like FORALL and EXISTS
A Spec# model is just a program! You can run it like a C# program It can call framework code NEW You can explore it (run all of its possible
behavior) which is how we do MBT with Spec#
Spec Explorer (UW-MSR Summer Institute '04)
![Page 26: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/26.jpg)
Current State of MBT in Industry (Spec Explorer) A model exploration and testing tool
for .NET: Authoring of models in Word Model exploration (generating FSM) FSM visualization FSM traversal and test suite generation Automatic implementation binding Online test-suite execution and offline test-suite
code generation
Spec Explorer (UW-MSR Summer Institute '04)
![Page 27: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/27.jpg)
Current State of MBT in Industry (Spec #)
Model: Calculator (viewpoint: only starting and stopping, and switching scientific mode)
bool Running = false;bool Scientific = false;
[Action] void SetRunning(bool newRunning) requires Running != newRunning; { Running = newRunning;}
[Action] void SetScientific(bool newScientific) requires Running; requires Scientific != newScientific; { Scientific = newScientific;}
System State
Spec#: Pre-Condition
describes when action is enabled
State Update
Tells Spec Explorer
this is an action. Action
invocations do appear in tests.Spec Explorer (UW-MSR Summer Institute '04)
![Page 28: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/28.jpg)
Software Tools for MBT
![Page 29: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/29.jpg)
Software Tools for MBT Conformiq Test Generator is a model-based testing tool
using UML state charts that represent testing strategies Leirios Test Generator is a model-based testing tool that
generates tests automatically from deterministic system specifications
Reactis Tester is another model-based testing tool that focuses on control systems
TGV is a tool for the generation of conformance test suites for protocols
TorX is also a prototype testing tool for conformance testing of reactive software
Lurette is an automated testing tool of reactive programs written in Lustre
AsmL Test Tool can generate tests directly from an AsmL model
AutoFocus (in german) is a graphical tool for developing and modeling distributed systems with integrated testing facilities
![Page 30: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/30.jpg)
Software Tools for MBT (AsmL) AsmL is the Abstract State Machine
Language. It is an executable specification language based on the theory of Abstract State Machines. The current version, AsmL 2 (AsmL for Microsoft .NET), is embedded into Microsoft Word and Microsoft Visual Studio.NET. It uses XML and Word for literate specifications. It is fully interoperable with other .NET languages. AsmL generates .NET assemblies which can either be executed from the command line, linked with other .NET assemblies, or packaged as COM components.
![Page 31: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/31.jpg)
Conclusion
![Page 32: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/32.jpg)
Conclusion Modeling is not easy
What details do you leave out? How do you check the model? There are no recipes for success
What needs to happen Easier notation Incremental approach Pilots and examples
![Page 33: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/33.jpg)
Questions How do we guarantee that the generated
expected outputs and inputs are enough and cover all the cases?
How to make for the errors and exceptions that may occur due to running the program on different environments and different OSs?
How is automated generation of inputs and outputs work when an actual human interaction is required as part of the system testing?
Does MBT cover stress testing? Does MBT cover how user friendly the
system is or it only covers functionality?
![Page 34: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/34.jpg)
References
![Page 35: Model Based Testing](https://reader036.vdocuments.net/reader036/viewer/2022062323/568158f6550346895dc63292/html5/thumbnails/35.jpg)
References http://en.wikipedia.org/wiki/Model-based_testing http://blogs.msdn.com/nihitk/articles/144664.aspx http://research.microsoft.com/specsharp/ http://react.cs.uni-sb.de/mbt2006/talks/
mbt4masses.pdf http://react.cs.uni-sb.de/mbt2006/talks/
ModelBasedTestingSoberEvaluation.pdf http://aetgweb.argreenhouse.com/papers/1999-
icse.pdf Colin Campbell, Wolfgang Grieskamp,
Lev Nachmanson, Wolfram Schulte, Nikolai Tillmann, Margus Veanes “Spec Explorer:An Integrated Environment for Model-Based Testing.” Foundations of Software Engineering Microsoft Research, Redmond