using execution feedback in test case generation
DESCRIPTION
Using Execution Feedback in Test Case Generation. CMSC 737 – Presentation Bao Nguyen [email protected]. Two Strategies. Static plan Category Partition Data flow analysis (path, branch, def-use, etc) Predicate based: BOR – BRO => Try to guess ahead!!! Dynamic plan - PowerPoint PPT PresentationTRANSCRIPT
![Page 2: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/2.jpg)
Two StrategiesStatic plan
Category Partition Data flow analysis (path, branch, def-use, etc)Predicate based: BOR – BRO=> Try to guess ahead!!!
Dynamic planExecution information as feedbackGenerating test cases on the fly
![Page 3: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/3.jpg)
What’s next…Test case generation based on execution feedbackCase study: two recent papers in ICSE’07What I’m looking atConclusion
![Page 4: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/4.jpg)
What’s next…Test case generation based on execution feedbackCase study: two recent papers in ICSE’07What I’m looking atConclusion
![Page 5: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/5.jpg)
Why Static Plans Are Not Enough?Software is so complex:
Empirical products => Difficult to formalizeSubjective solutions => Difficult to useHuman actuators => Difficult to manageIntelligent products => Difficult to predictDynamic environments => Difficult to address
Hard to predict ahead !!!Hard to exhaustedly test !!!
![Page 6: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/6.jpg)
General FrameworkSpecifications
ProgramExecution
f(e)Feedback
-
RequiredQuality rQ
ObservedQuality
oQ
oQrQe
(Adapted from “Software Cybernetics: Progress and Challenges” - Aditya P. Mathur )
Test case Engine
![Page 7: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/7.jpg)
General FrameworkSpecifications
ProgramExecution
f(e)Feedback
-
RequiredQuality rQ
ObservedQuality
oQ
oQrQe
(Adapted from “Software Cybernetics: Progress and Challenges” - Aditya P. Mathur )
Test case Engine
![Page 8: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/8.jpg)
General FrameworkSpecifications
ProgramExecution
f(e)Feedback
-
RequiredQuality rQ
ObservedQuality
oQ
oQrQe
(Adapted from “Software Cybernetics: Progress and Challenges” - Aditya P. Mathur )
Test case Engine
![Page 9: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/9.jpg)
What’s next…Execution feedback based test case generationCase study: two recent papers in ICSE’07
OOP Testing: Pacheco, et al. “Feedback-Directed Random Test Generation”
GUI Testing: Yuan and Memon. “Using GUI Run-Time State as Feedback to Generate Test Cases”
What I’m looking atConclusion
![Page 10: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/10.jpg)
What’s next…Execution feedback based test case generationCase study: two recent papers in ICSE’07
OOP Testing: Pacheco, et al. “Feedback-Directed Random Test Generation”
GUI Testing: Yuan and Memon. “Using GUI Run-Time State as Feedback to Generate Test Cases”
What I’m looking atConclusion
![Page 11: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/11.jpg)
Xun Yuan and Atif MemonICSE’07
Using GUI Run-Time State as Feedback to Generate Test Cases
![Page 12: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/12.jpg)
Motivations of the paperPrevious work
1-way: Crash Test2-way: Smoke Test
Longer test cases detected additional faults
Unable to run multi-way test coverage 2-way run for months
[TSE’05]Try to prune edge
![Page 13: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/13.jpg)
Key ideaUse GUI states as feedback to identify
“important” edges Called Event Semantic Interaction Edges
Generate new longer test cases covering those edges
![Page 14: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/14.jpg)
Key ideaUse GUI states as feedback to identify
“important” edges Called Event Semantic Interaction Edges
Generate new longer test cases covering those edges
ESI
ESI
ESI
ESI
ESI
![Page 15: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/15.jpg)
Event Semantic InteractionHeuristic: Two events executed together results differently
than executed in isolation => semantic interaction
<e1>Select Eclipse
<e2>Drag
<e1; e2>
![Page 16: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/16.jpg)
Six predicates for modeless windowsPredicate 1:
w W; p P; v V; v’ V; s.t: ((v v’) ((w, p, v) {S0 e1(S0) e2(S0)}) ((w, p, v’) e2(e1(S0))));
Predicate 2: w W; p P; v V; v’ V; v’’ V; s.t: ((v v’) (v’ v’’) ((w, p, v) {S0 e2(S0)}) ((w, p, v’) e1(S0)) ((w, p, v’’) e2(e1(S0))));
Predicate 3: w W; p P; v V; v’ V; v’’ V; s.t: ((v v’) (v’ v’’) ((w, p, v) {S0 e1(S0)}) ((w, p, v’) e2(S0)) ((w, p, v’’) e2(e1(S0))));
Predicate 4: w W; p P; v V; v’ V; v’’ V; v V; s.t: ((v v’) (v v’’) (v’’ v ) ((w, p, v) S0) ((w, p, v’) e1(S0)}) ((w, p, v’’) e2(S0)}) ((w, p, v) e2(e1(S0))));
Predicate 5: w W; p P; v V; v’ V; v’’ V; s.t: ((v’ v’’) (((w, *, *) S0) (((w, p, v) e1(S0)) ((w, p, v’) e2(S0))) ((w, p, v’’) e2(e1(S0))));
Predicate 6: w W; ENABLED P; TRUE V; FALSE V; s.t: (((w, ENABLED, FALSE) S0 ) ((w, ENABLED, TRUE) e1(S0)) EXEC(e2, w));
(More details refer to “A comprehensive framework for testing graphical user interfaces”Atif M. Memon.Ph.D. dissertation, 2001)
![Page 17: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/17.jpg)
Three contexts for events wrt windowsContext 1 : events in modeless windowContext 2 : events in same modal window
ex(S): the GUI state after executing <ex; TERM>, x= 1, 2e2(e1(S)): the GUI state after executing sequence <e1; e2; TERM>
Context 3 : events in parent and child modal windowe1(S): the GUI state after executing <e1; TERM>e2(e1(S)): the GUI state after executing sequence
<e1; TERM; e2>
![Page 18: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/18.jpg)
ExperimentsSubject applications: three OSS
CrosswordSage 0.3.5FreeMind 0.8.0GanttProject 2.0.1JMSN 0.9.9b2
Test oracleProgram crashes
![Page 19: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/19.jpg)
Result - Test case reduction
2-way 3-way 4-way 5-way99.78% 99.97% 99.99%
Table 1: Percentages of test case reduction
![Page 20: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/20.jpg)
Result - Faults detection
0123456789
CrosswordSage FreeMind GanttProject JMSN
Num
ber
of F
aults
All-edges Covering Test Cases 3,4,5-way Covering Test Cases
![Page 21: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/21.jpg)
Lessons learnedEvent handlers are implemented in multiple classesA large input space is neededCrash because the permutations of events=> Need longer test cases???
![Page 22: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/22.jpg)
Conclusion of this paperContributions
A new GUI model for test-case generationA new relationships among GUI events (i.e. context)A utilization of GUI state as feedbackA fully automatic end-to-end GUI testing processA demonstration
Future workSimplify 6 predicates and 3 contextsIdentify and classify events dominating ESIMinimize number of test casesApply feedback technique to objects outside GUI
![Page 23: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/23.jpg)
What’s next…Test case generation based on execution feedbackCase study: two recent papers in ICSE’07What I’m looking atConclusion
![Page 24: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/24.jpg)
What I’m looking atPush test case generation and test case execution closer
Generate new test cases during the execution Utilize the feedback immediately
![Page 25: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/25.jpg)
A case studyAdaptive test oracles: the QoS idea
e1 e2 … FIND_NEXT en-1 enTest Case =
S0 S1 S2 … Si-1 Si …
…
Sn-1 SnOracle
Information =
“Using Transient/Persistent Errors to develop Automated Test Oracles for Event-Driven Software”Atif M. Memon and Qing Xie. – ASE’04
![Page 26: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/26.jpg)
ConclusionSoftware is dynamic so we need a dynamic approachUsing feedback in software testing is feasibleSomewhat related to control theories
(i.e. software cybernetic)Drawback: Like hill climbing
=> local optimizationCan mutants (like in GA) overcome this?Systematically vs. Randomly
![Page 27: Using Execution Feedback in Test Case Generation](https://reader036.vdocuments.net/reader036/viewer/2022062814/56816802550346895ddd8203/html5/thumbnails/27.jpg)
QuestionsWhat does “Event Semantic Interaction” in section 4
mean?What are the threats to validity and what are the
weaknesses in Xun’s experiments?