automatically repairing broken workflows for evolving gui...
TRANSCRIPT
Automatically Repairing Broken Workflows for Evolving GUI Applications
2013.11.27
Gigon Bae
Sai Zhang, Hao Lü, and Michael D. Ernst
ISSTA 2013
Introduction
Motivation
Overall Approach
Method
Empirical Evaluation
Related Work
Conclusion
Discussion
Contents
2 | 27
Broken Workflow Problem
3 | 27
The software developers evolve the GUI over time to improve the user experience. This can create usability problems for end-users, breaking an end-
user’s workflow --- a sequence of UI actions to complete a specific task.
Recovering from such broken workflows can be tedious, laborious, and frustrating. E.g., Microsoft Office products.
Crossword version 0.3.0
Crossword version 0.3.5
Test engineers often write test scripts to mimic end-user workflows to automate testing of a GUI application.
Such test scripts are fragile to UI changes. 74% of test scripts become unusable between successive
releases of a typical GUI application [1]. Even simple modifications to GUIs resulted in 30% to 70%
changes to test scripts [2].
GUI Evolution Problems to SW Developers
4 | 27
[1] A. M. Memon and M. L. Soffa. Regression testing of GUIs. In FSE, 2003.[2] M. Grechanik, Q. Xie, and C. Fu. Maintaining and evolving GUI-directed test scripts. In ICSE, 2009.
Manually repairing every broken workflows Too many GUI screens and UI actions
: tedious and frustrating
Programmatically comparing the GUIs of two versions, identifying changed GUI elements, and then locating UI actions that reference these modified GUI elements For some UI actions, their effects cannot be observed
without actually executing actions. The GUI can change substantially
: without knowing the precise “action semantics”, it is generally impractical to find a replacement GUI elements and a suitable action on it from the new application version.
Prior Approaches
5 | 27
During the evolution of a GUI application, the underlying system that implements a given functionality often stays relatively the same between versions, even when its GUI evolves rapidly.
Key Observations
6 | 27
A program-analysis-based solution: uses program analysis to repair broken workflows that are affected by GUI evolution.
The suggested technique, called FlowFixer, recommends replacement UI actions in the updated GUI application to complete the same workflow.
1. Dynamic Workflow Profiling
2. Static Method Matching
3. Random Action Execution
4. Replacement Action Recommendation
Approach
7 | 27
Input: two versions of the application
Output: replacement UI actions in the updated GUI application
Overview
8 | 27
Old version New version
FlowFixer first instruments the old version of the tested application offline by inserting code to monitor each method’s execution at run time.
FlowFixer ask user to demonstrate a workflow on its GUI.
Event handlers and other methods invoked at run time are recorded in the execution trace.
1. Dynamic Workflow Profiling
9 | 27
Previous work on matching program elements across program versions is unavailable.
does not support the latest Java version.
focuses on identifying changes by a small set of refactorings.
Created a tool to perform method matching Uses the first one that succeeds
1. Identical Method heuristic
2. Similar Name heuristic: using Levenshtein string similarity (default threshold: 0.9)
3. Co-evolving heuristic: observing the deleted method and the replacement methods in the same revision
2. Static Method Matching
10 | 27
FlowFixer employs random testing to generate and execute random UI actions on the new version, and observes the invoked event handlers and methods in the background.
For the sake of efficiency, it approximates the exploration by executing each UI action (at most) once.
Several kinds of UI events such as key pressing, mouse moving, and window disposing are ignored.
3. Random Action Execution
11 | 27
Example of Random Execution Algorithm
12 | 27
Window 1
Modal Dialog 1
Initial screen
Invokes:method A (event handler)method Bmethod C …
Recommends a list of replacement UI actions that may complete the same workflow in the new version.
Check each matched method in the new version and then infer which UI action is most likely to invoke it.
Focuses on the “uniqueness” of a method being invoked by UI actions.
4. Replacement Action Recommendation
13 | 27
4. Replacement Action Recommendation
14 | 27
1/3 1/3 1/3
1/1
Event handler
FlowFixer is implemented by using WALA framework and the UISpec4J library.
WALA: to perform offline instrumentation of Java bytecode.
UISpec4J: to execute UI actions and gather method invocation information.
The FlowFixer implementation is publicly available at
http://workflow-repairer.googlecode.com
Implementation
15 | 27
1. How accurate is FlowFixer in repairing broken workflows for real-world GUI applications? (what is the rank of the actual replacement UI actions in FlowFixer’s output?)
2. How long does it take for FlowFixer to repair a broken workflow?
3. How does FlowFixer’s effectiveness compare to an existing approach based on GUI comparison?
4. How does FlowFixer’s effectiveness compare to an alternative approach based on static analysis?
Research Question
16 | 27
Subject programs
Five popular GUI-based applications available at SourceForge.net• Chose two versions that contain significant GUI changes.
› Analyzed all documented workflows from user manual.
Subject Programs and Broken Workflows
17 | 27
FlowFixer repairs realistic broken workflows with high accuracy for evolving GUI applications with non-trivial GUI changes
Result: RQ1 - Accuracy
18 | 27
RQ1 : how accurate is FlowFixer in repairing broken workflows for real-world GUI applications? (what is the rank of the actual replacement UI actions in FlowFixer’s output?)
Example of Repairing Workflow
19 | 27
Invoked UndoableEditImpl.createTemporaryFile in common.
- Uses 27 minutes to execute random UI actions on each subject program.
- Spends an average of 3.2 minutes to recommend replacement UI actions for one workflow.
Result: RQ2 – Time Cost
20 | 27
FlowFixer repairs realistic broken workflows with acceptable time cost.
RQ2: how long does it take for FlowFixer to repair a broken workflow?
Comparing GUIs of two app. Versions without analyzing the program is insufficient in repairing realistic broken workflows.
Result: RQ3 – Comparison with a GUI-Comparison Approach
21 | 27
RQ3: how does FlowFixer’s effectiveness compare to an existing approach based on GUI comparison?
※ M. Grechanik, Q. Xie, and C. Fu. Maintaining and evolving GUI-directed test scripts. In ICSE, 2009.
※
Random testing, though neither sound nor complete, can provide more accurate and useful results than static analysis.
Result: RQ4 – Comparison with Static Analysis
22 | 27
RQ4: how does FlowFixer’s effectiveness compare to an alternative approach based on static analysis?
Analyzing and Testing GUI Applications Automated GUI Testing Techniques
• Maximize coverage.
Avoiding bugs in GUI applications• Prevent an existing bug from happening again.
Supporting Software Evolution Debugging techniques
• Help software developers understand divergent program behaviors between versions
Automatically repairing unit tests and GUI test scripts
Patching Software Errors Automated software error patching
Related Work
23 | 27
Presented a practical technique for repairing broken workflows for evolving GUI applications.
Showed that FlowFixer is accurate and efficient. The source code of FlowFixer is publicly available.
Future work would be performing a user study to evaluate FlowFixer’s usefulness to end-users and investigating some possible applications of the FlowFixer technique. E.g., automatic update of the documented workflows in
the user manual as a GUI app. evolves.
Conclusion
24 | 27
The technique finds only one broken action.
A broken action that is replaced to a sequence of actions might not be found.
Considering methods invoked at runtime would be useful for other GUI testing techniques.
Discussion
25 | 27
Thank you :)