the test case as executable example
DESCRIPTION
CWI Lectures on Understanding Software, in honor of Paul Klint. June 14, 2012.TRANSCRIPT
![Page 1: The Test Case as Executable Example](https://reader034.vdocuments.net/reader034/viewer/2022050920/549cf2fab47959b0318b4835/html5/thumbnails/1.jpg)
The Test Case as Executable Example
Arie van Deursen
@avandeursen
1
CWI Lectures – Understanding Software
June 14, 2012
![Page 2: The Test Case as Executable Example](https://reader034.vdocuments.net/reader034/viewer/2022050920/549cf2fab47959b0318b4835/html5/thumbnails/2.jpg)
20th International Conference on Program Comprehension
Study of
the human activity of comprehending software;
the processes and technologies for supporting it
2
![Page 3: The Test Case as Executable Example](https://reader034.vdocuments.net/reader034/viewer/2022050920/549cf2fab47959b0318b4835/html5/thumbnails/3.jpg)
Programming by Example
Programming by Example, Daniel Halbert, 1984, Xerox
Spreadsheet Data Manipulation Using Examples, CACM, 2012 3
![Page 4: The Test Case as Executable Example](https://reader034.vdocuments.net/reader034/viewer/2022050920/549cf2fab47959b0318b4835/html5/thumbnails/4.jpg)
The Testing Perspective
• A test case is an executable example of system behavior
• Stakeholder communication needs examples
• Throughout the full development cycle
4
![Page 5: The Test Case as Executable Example](https://reader034.vdocuments.net/reader034/viewer/2022050920/549cf2fab47959b0318b4835/html5/thumbnails/5.jpg)
Cucumber
5
Feature: Log in and out
As an administrator
I want to restrict access to certain portions of my site
In order to prevent users from changing the content
Scenario: Logging in
Given I am not logged in as an administrator
When I go to the administrative page
And I fill in the fields
| Username | admin |
| Password | secret |
And I press "Log in"
Then I should be on the administrative page
And I should see "Log out"
Scenario: Logging out ...
webrat_steps.rb:
Then /^I should see "([^\"]*)"$/ do |text|
response.should contain(text)
end
![Page 6: The Test Case as Executable Example](https://reader034.vdocuments.net/reader034/viewer/2022050920/549cf2fab47959b0318b4835/html5/thumbnails/6.jpg)
6
Plugin / Junit Testing in Eclipse. 1000s of tests
![Page 7: The Test Case as Executable Example](https://reader034.vdocuments.net/reader034/viewer/2022050920/549cf2fab47959b0318b4835/html5/thumbnails/7.jpg)
Test Suite Information Needs
7
Test Confessions: A Study of Testing Practices for Plug-In Systems Michaela Greiler, Arie van Deursen, Margaret-Anne Storey, ICSE 2012
![Page 8: The Test Case as Executable Example](https://reader034.vdocuments.net/reader034/viewer/2022050920/549cf2fab47959b0318b4835/html5/thumbnails/8.jpg)
Challenges
• Set of examples is incomplete
• Set of examples is too large
– Some grouping needed
– Redundancy
• Set of examples is disconnected
– Requirements traceability
– Backward coverage
8
Test Similarity
![Page 9: The Test Case as Executable Example](https://reader034.vdocuments.net/reader034/viewer/2022050920/549cf2fab47959b0318b4835/html5/thumbnails/9.jpg)
End-to-End Test
Integration Test
Requirements Analysis
High Level Design
...
...
Unit Test Code
9
Measuring Test Case Similarity to Support Test Suite Understanding Michaela Greiler, Arie van Deursen, Andy Zaidman, TOOLS 2012
![Page 10: The Test Case as Executable Example](https://reader034.vdocuments.net/reader034/viewer/2022050920/549cf2fab47959b0318b4835/html5/thumbnails/10.jpg)
Measuring Test Case Similarity
10
Test Traces
Tests ranked by similarity
Shared word count
Test-aware instrumentation
![Page 11: The Test Case as Executable Example](https://reader034.vdocuments.net/reader034/viewer/2022050920/549cf2fab47959b0318b4835/html5/thumbnails/11.jpg)
Preliminary Evaluation
• 75 unit tests
• 14 acceptance tests
• Ground truth by hand
• 37 unit
• 6 end to end
• Ground truth from book
11
• Most similar match: #1 or #2 in ground truth
• Top 10: all (but one) explainable & useful
• Surprises: mostly point to issue in test suite.
• Work in progress: Analyzing Eclipse test suites.
![Page 12: The Test Case as Executable Example](https://reader034.vdocuments.net/reader034/viewer/2022050920/549cf2fab47959b0318b4835/html5/thumbnails/12.jpg)
Rethinking “Test Cases as Examples”
• Languages for writing examples – Different levels of abstraction / stakeholders
– Executable
– Analyzable
– Connected to execution traces
12
• Modularization and grouping of examples – Different levels of abstraction / stakeholders
– Requirements / design / code coverage
– Multiple views
– Connected via execution traces
![Page 13: The Test Case as Executable Example](https://reader034.vdocuments.net/reader034/viewer/2022050920/549cf2fab47959b0318b4835/html5/thumbnails/13.jpg)
In Conclusion
• A test case is an executable example
• Never enough, yet also too many
• Test case connection via trace similarities
• Language engineering opportunities ahead
13
![Page 14: The Test Case as Executable Example](https://reader034.vdocuments.net/reader034/viewer/2022050920/549cf2fab47959b0318b4835/html5/thumbnails/14.jpg)
Photo credits
• Passau: Andreas Brandl http://icpc12.sosy-lab.org/
• Programming by Example, picture JanHeering (CWI). Photo Eelco Visser. http://eelcovisser.org/post/91/example-driven-research
• The Testing Perspective: Photo “Reaching for Understanding”, Flickr http://www.flickr.com/photos/elizabethsalib/2741807789/
• Andromeda Galaxy http://www.nasa.gov/mission_pages/spitzer/multimedia/spitzer-20070604.html
• Colliding galaxies http://hubblesite.org/newscenter/archive/releases/2007/08/
• Paul Klint, lecturing. http://www.frankwatching.com/archive/2009/05/18/het-nationaal-ict-event/
14