model-based testing · 2017. 6. 6. · model-based testing (dit848 / dat261) spring 2017 lecture 4...
TRANSCRIPT
![Page 1: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/1.jpg)
Model-Based Testing (DIT848 / DAT261)
Spring 2017 Lecture 4
Testing: The Bigger Picture
Gerardo Schneider Department of Computer Science and Engineering
Chalmers | University of Gothenburg
Some slides based on material by Magnus Björk and Thomas Arts 1
![Page 2: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/2.jpg)
The Bigger Picture
Requirements
Specification
Architectural Design
Detailed Design
Coding
Unit Test
Integration Test
System Test
Acceptance Test
The techniques we have seen so far are mainly at this
level
2
![Page 3: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/3.jpg)
Unit tests � Testthesmallestcomponentsindividually
� O4endonebytheprogrammerwhowrotethecode
� Lessstrictrequirementsofdocumenta<on� Largepartofdocumenta<onreplacedbyexecutabletestsuites(Cunit,Junit
orsimilar),whichmustthereforebeclearlywriCen
� Nolessimportantthananyothertest!
� Infact,maybethemostimportanttest:� Unittestseasiertodothanothertests–wellinvested<me� Bugsdiscoveredearlyeasiertofix� Sospendingeffortonunittestsreducesworklater
� Recommendedeffort:equalamountof<mespentwri<ngcodeandunittests
3
![Page 4: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/4.jpg)
Unit tests – typical flow Programmer:
� writescode� runssta<cverifica<ontoolsuchassplint(forC)� writesandrunsunittestsuitetotestthecode
� UsingframeworksuchasJUnit,CUnit
� complementsblackboxtestsuitewithwhiteboxtechniques� Coveragechecking(iden<fymissingtestcases)–Gcov,Emma� Valgrind:MonitormemorybehaviourofC/C++programs
Thecolleaguesoftheprogrammerdo:
� Codereview4
![Page 5: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/5.jpg)
Unit tests Test Driven Development (TDD) Programmer:
� writesunittestcases� Runstestsuite,makessureitfails
� writescodeun<ltestsuitedoesnotfail� Addsmoretestcasesifneeded
� runssta<cverifica<ontoolsuchassplint(forC)
� complementsblackboxtestsuitewithwhiteboxtechniques� Coveragechecking(iden<fymissingtestcases)–Gcov,Emma� Valgrind:MonitormemorybehaviourofC/C++programs
Thecolleaguesoftheprogrammerdo:
� Codereview 5
![Page 6: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/6.jpg)
Benefits of TDD � Focusonwhatthecodedoesbeforeimplementa<on
� Helpfulwhenwri<ngthecode
� Programmergetsveryquickfeedback
� Easiertomaintain� resultsinbeCercoverageofunittestsuite
� Note:TDD”mandatory”inXP6
![Page 7: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/7.jpg)
Integration tests � Testdifferentcombina<onsofcomponents
� Differentstrategiesforintegra<on� Bigbang� BoCom-up� Top-Down� Sandwich
User interface
Function 1 Function 2
Lib 1 Lib 2 Lib 3
Lib 4
Thanks to Bruegge & Dutoit for material (through Magnus) 7
![Page 8: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/8.jpg)
Integration tests: Big Bang approach
� A4erunittests,integrateallcomponentsatonce
� Essen<allyasystemtest
� Badidea!Don’tuseit� Hardtolocatebugs(havetosearchthewholesystem)� Cri<calandperipheralcomponentsgetthesame
aCen<on� Onlypossibleverylateindevelopmentcycle
8
![Page 9: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/9.jpg)
� Startwiththesubsystemsinthelowestlayerofcallhierarchy
� Integratesuchcomponentswithcomponentsthatusethem
� Donerepeatedlyun<lwholesystemisintegrated
� Specialcodeneeded:Testdriver� Arou<nethatcallssubsystemsandpassestestcasestoit
� CanbedoneinframeworkssuchasCUnit
Integration tests: Bottom-up approach
9
![Page 10: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/10.jpg)
Integration tests: Bottom-up approach
User interface
Function 1 Function 2
Lib 1 Lib 2 Lib 3
Lib 4
Lib 4
Driver
Lib 1
Driver
Lib 4 Lib 1
Driver
Lib 4
Lib 2
Driver
Lib 4
Lib 2
Functions 1
Driver
Driver: A routine that calls subsystems and passes test cases to it
10
![Page 11: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/11.jpg)
Integration tests: Top-down approach
� Testtoplevelcomponentsfirst,itera<velyintegratecomponentsthatarecalledbythecomponentsthatarealreadyincluded.Repeatun<lthewholesystemisintegrated
� Specialcodeneeded:Stub� Hasthesameinterfaceasthecomponentitreplaces� Returnsfakedata(probablydescribedinthetestcase)� Passesinforma<onofthecalltothetestframework
� XUnitmaybeuseful
� Pros:� Testcasesdefinedintermsofprogramspec.� Easytoseebehaviourateachstage(userinterface)
� Cons:� Wri<ngstubsdifficultandtedious� Makingautomatedtestsuitemaybeharder(e.g.ifGUI)
11
![Page 12: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/12.jpg)
User interface
Function 1 Function 2
Lib 1 Lib 2 Lib 3
Lib 4
User interface
Stub Stub
User interface
Fun 1 Stub
Stub Stub
User interface
Fun 1 Stub
Stub Lib 2
Stub Stub: a piece of (dummy) code used to stand in for some other programming functionality
Integration tests: Top-down approach
12
![Page 13: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/13.jpg)
Sandwich approach
� DobothboComupandtopdown,meetinthemiddle
� Muchparalleliza<on:� Firstphase:
� Toplayerwithstubs� Middlelayerwithdriversandstubs� BoComlayerwithdrivers
� Secondphase:� Topandmiddlelayer(toplayerreplacesdrivers)� MiddleandboComlayer(boComlayerreplacesstubs)
13
![Page 14: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/14.jpg)
Integration tests: Sandwich approach
Driver
Function 1 Function 2
Lib 1 Lib 2 Lib 3
Lib 4
User interface
Stub Stub
Lib 1
Lib 4
Driver
Lib 2
Lib 4
Driver
Lib 3
Driver
Driver
Function 1 Function 2
Stub Stub Stub
User interface
Function 1 Function 2
Stub Stub Stub 14
![Page 15: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/15.jpg)
Integration Tests: what to consider when choosing approach
� Whichpartsofthesystemaremostcri<cal?� Choosestrategythatrevealserrorincri<calpartsearly,and
includescri<calpartsinmanytests
� Whichapproachmeanslesswork?� Topleveltestmaybehardertoautomate(e.g.GUIs)� Howtominimizeworkspentwri<ngdriversandstubs
� Availabilityofcomponents� IfcodingdoneboCom-up,thenboCom-upintegra<ontestscan
bestartedearlier15
![Page 16: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/16.jpg)
To make integration tests easier
� Dothoroughunittests
� Makewelldefinedinterfacesbetweenmodules
16
![Page 17: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/17.jpg)
System tests � Testthefullsystem
� Coverfullspecifica<on� Testautoma<onmaybehardtoachive
� Systemtestsmaybeexpensiveandtedious
� Combineblackandwhiteboxtes<ngasbefore
� Testbothnormalandabnormalusesofthesystem� Performancetes<ng
� Pushsystemtoitslimits� Thegoalistotrytobreakthesystem� Maybeusedtoiden<fyboClenecks,tobedealtwithinnextitera<on
ofdevelopment17
![Page 18: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/18.jpg)
System tests � Stresstes'ng:exceedparameters:numberofrequests,…
� Volumetes'ng:largeamountsofdata
� Configura'ontes'ng:differentcombina<onsofHW&SW
� Compa'bilitytes'ng:usewitholdersystems
� Securitytes'ng:trytobreakin� Timingtes'ng:<meresponses&func<ons
� Environmentaltes'ng:effectsoftemperature,movement,…
� Qualitytes'ng:reliability,maintainability,availability
� Recoverytes'ng:erroneousormissinginput
� Humanfactorstes'ng:testuserinterfaceonusers 18
![Page 19: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/19.jpg)
Acceptance tests
� Customermostlyresponsibleforacceptancetest
� Alphates<ng� Donebycustomerundersupervisionofdeveloper� Usuallydoneincontrolledenvironment(developer’ssystems)� Developercanquicklyfixbugs
� Betates<ng� Productusedbycustomersinrealenvironment� Developerstypicallynotpresent� Differencefromrestofproductlife<me:
� O4enonlyselectedcustomers� Customercannotrelyonso4ware
19
![Page 20: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/20.jpg)
Fixing bugs � Ac<ondependsonseverityofbug
� Low-priorityfailuresmaybeputon”knownbugs”list,includedinreleasenotes
� Alwaysdoregressiontesta4erfixingbugs!� Bugfixesarelikelytobreaksomethingelse
� Bugtrackingtoolso4enuseful(Example:Bugzilla)� Maintainslistofbugs� Assignspriori<esandresponsiblepeopleforeachbug� Keepsremindingpeopleabouttheirhighprioritybugs� Searchablebugindex(withhistory)
20
![Page 21: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/21.jpg)
Regression tests � Mustbedonea4ereverychangetosourcecode
� Regressiontestssignificantlycheaperiftestsuiteisautomated
� Some<mesnotfeasibletoredoalltests.Ifso,iden<fyasubsetofcasesthatcoverasmuchaspossible
� Tool:Tinderbox� Automa<callychecksoutcommiCedcode,compilesitandruns
testsuite(needsothertoolforthat,suchasDejaGnu)� Iden<fiescompila<onerrorsandfailingtestcases� Pointsoutwho’sresponsible� Maintainshistory� O4enruns100%ofthe<meonabunchofdedicatedmachines
21
![Page 22: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/22.jpg)
Test in General: a test… Determinewhetherthestatementsaretrueorfalse.Ifastatementisfalse,jus'fyyouranswer
1. Therearetwokindsoftes<ng:dynamicandsta<c.
2. Ifyouget100%codecoveragethenyoucanguaranteethatyourso4warehasbeenthoroughlytestedandcanstoptes<ng.
3. XUnitisbeCerthanJUnitsinceyoucangetbeCertests.4. TheVmodelteachesusthatwecandoacceptancetestsassoonaswehavethe
requirements,evenbeforewestartdeveloping.
5. Differenttes<ngmethodsandtechniquesapplytoeachtestlevel(aspresentedintheVmodel).
6. Tes<nganddebuggingarethesame.
7. Onegoodthingaboutthesandwichapproachforintegra<ontes<ngisthatenhancesparalleliza<on(thatis,developersandtesterscanworkinparallel).
8. Performancetes<ngisonekindoftest,partoftheso-calledsystemtes<ng.
9. Thebestwaytodointegra<ontestisthesandwichapproach.Groups 2-5 persons: 15 min 22
![Page 23: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/23.jpg)
Test in General: solution… 1. F–Tes<ngisbydefini<ondynamic
2. F–Codecoverageisonlyoneaspect;thereisnoguaranteeingeneraltoget100%confidence
3. F–XUnitisafamilyoftestunits,includingCUnit
4. FAcceptancetestisonlydonea4erthereissomethingtotestagainsttherequirements.Donebycustomer
5. T6. F–Tes<ng:establishtheexistenceofdefects;debugging:
loca<ng&repairingthoseerrorsfoundduringtes<ng
7. T8. T9. F–dependsonhowthesystemisbuilt 23
![Page 24: Model-Based Testing · 2017. 6. 6. · Model-Based Testing (DIT848 / DAT261) Spring 2017 Lecture 4 Testing: The Bigger Picture Gerardo Schneider Department of Computer Science and](https://reader036.vdocuments.net/reader036/viewer/2022071015/5fce6917a7d20b11f83ee56f/html5/thumbnails/24.jpg)
Next lecture
24
� EFSM: Extended Finite State Machines