fusion: a tool for facilitating and augmenting android bug...

4
FUSION: A Tool for Facilitating and Augmenting Android Bug Reporting Kevin Moran, Mario Linares-Vásquez, Carlos Bernal-Cárdenas, Denys Poshyvanyk College of William & Mary Department of Computer Science Williamsburg, VA 23187-8795, USA {kpmoran, mlinarev, cebernal, denys}@cs.wm.edu ABSTRACT As the popularity of mobile smart devices continues to climb the complexity of “apps” continues to increase, making the development and maintenance process challenging. Current bug tracking systems lack key features to effectively support construction of reports with actionable information that di- rectly lead to a bug’s resolution. In this demo we present the implementation of a novel bug reporting system, called Fu- sion, that facilitates users including reproduction steps in bug reports for mobile apps. Fusion links user-provided in- formation to program artifacts extracted through static and dynamic analysis performed before testing or release. Re- sults of preliminary studies demonstrate that Fusion both effectively facilitates reporting and allows for more reliable reproduction of bugs from reports compared to traditional issue tracking systems by presenting more detailed contex- tual app information. Tool website: www.fusion-android. com Video url: https://youtu.be/AND9h0ElxRg Categories and Subject Descriptors D.2.7 [Software Engineering]: Distribution, Maintenance, and Enhancement General Terms Experimentation, Design Keywords Bug reports, android, reproduction steps, auto-completion 1. INTRODUCTION It is clear that as smart device usage reaches ubiquitous levels (e.g., 2.7 billion active smartphone users in 2014[20]), developers need tools to support them in maintaining high- quality apps. Software maintenance activities are known to be generally expensive and challenging [24] and one of the most important maintenance tasks is bug report resolution. However, current bug tracking systems such as Bugzilla [3], Mantis [10], the Google Code Issue Tracker [7], the GitHub Issue Tracker [6], and commercial solutions such as JIRA Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ICSE’16 May 14-22, 2015, Austin, Texas USA Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00. [9] rely mostly on unstructured natural language bug de- scriptions. While these descriptions can be supplemented with structured information such as reproduction steps or stack traces, and files such as screenshots, the inclusion of such information typically depends on the reporter’s experi- ence and attitude towards providing these details. Previous studies have also shown that the information most useful to developers is often the most difficult for reporters to provide and that the lack of this information is a major reason be- hind non-reproducible bug reports [19, 15]. Therefore, the reporting process can be cumbersome, and the additional effort means that many users are unlikely to enhance their reports with extra information [16, 18, 17, 13]. The above issues point to a more prominent problem for bug tracking systems in general: the lexical gap that nor- mally exists between bug reporters (e.g., testers, beta users) and developers. Reporters typically only have functional knowledge of an app, even if they have development experi- ence themselves, whereas the developers working on an app tend to have intimate code level knowledge. When a de- veloper reads and attempts to comprehend (or reproduce) a bug report, she has to bridge this gap, reasoning about the code level problems from the high-level functional de- scription in the bug report. If the lexical gap is too wide the developer may not be able to reproduce and/or subsequently resolve the bug report. To address this fundamental problem of making bug re- ports more useful (and reproducible) for developers, this pa- per presents the implementation of a novel tool, called Fu- sion, that facilitates reporters creating detailed bug reports in order to provide more actionable information to develop- ers. Fusion implements the novel approach that was pre- sented and evaluated in our previous work [23]. Fusion first employs fully automated static and dynamic analysis tech- niques to gather screen-shots and other relevant information about an app before it is released for testing. Reporters then interact with the web-based report generator using the auto- completion features in order to provide the bug reproduction steps. By linking the information provided by the user with features extracted through static and dynamic analyses, Fu- sion presents an augmented bug report to the developer that contains actionable information with well-defined steps to reproduce a bug. 2. THE FUSION REPORTING TOOL Fusion’s current target user base consists of two ma- jor groups: mobile application developers and beta users or testers . As such there are two user-facing scenarios for the 1

Upload: phungdung

Post on 22-Jul-2018

213 views

Category:

Documents


0 download

TRANSCRIPT

FUSION: A Tool for Facilitating and Augmenting AndroidBug Reporting

Kevin Moran, Mario Linares-Vásquez, Carlos Bernal-Cárdenas, Denys PoshyvanykCollege of William & Mary

Department of Computer ScienceWilliamsburg, VA 23187-8795, USA

{kpmoran, mlinarev, cebernal, denys}@cs.wm.edu

ABSTRACTAs the popularity of mobile smart devices continues to climbthe complexity of “apps” continues to increase, making thedevelopment and maintenance process challenging. Currentbug tracking systems lack key features to effectively supportconstruction of reports with actionable information that di-rectly lead to a bug’s resolution. In this demo we present theimplementation of a novel bug reporting system, called Fu-sion, that facilitates users including reproduction steps inbug reports for mobile apps. Fusion links user-provided in-formation to program artifacts extracted through static anddynamic analysis performed before testing or release. Re-sults of preliminary studies demonstrate that Fusion botheffectively facilitates reporting and allows for more reliablereproduction of bugs from reports compared to traditionalissue tracking systems by presenting more detailed contex-tual app information. Tool website: www.fusion-android.

com Video url: https://youtu.be/AND9h0ElxRg

Categories and Subject DescriptorsD.2.7 [Software Engineering]: Distribution, Maintenance,and Enhancement

General TermsExperimentation, Design

KeywordsBug reports, android, reproduction steps, auto-completion

1. INTRODUCTIONIt is clear that as smart device usage reaches ubiquitous

levels (e.g., 2.7 billion active smartphone users in 2014[20]),developers need tools to support them in maintaining high-quality apps. Software maintenance activities are known tobe generally expensive and challenging [24] and one of themost important maintenance tasks is bug report resolution.However, current bug tracking systems such as Bugzilla [3],Mantis [10], the Google Code Issue Tracker [7], the GitHubIssue Tracker [6], and commercial solutions such as JIRA

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ICSE’16 May 14-22, 2015, Austin, Texas USACopyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00.

[9] rely mostly on unstructured natural language bug de-scriptions. While these descriptions can be supplementedwith structured information such as reproduction steps orstack traces, and files such as screenshots, the inclusion ofsuch information typically depends on the reporter’s experi-ence and attitude towards providing these details. Previousstudies have also shown that the information most useful todevelopers is often the most difficult for reporters to provideand that the lack of this information is a major reason be-hind non-reproducible bug reports [19, 15]. Therefore, thereporting process can be cumbersome, and the additionaleffort means that many users are unlikely to enhance theirreports with extra information [16, 18, 17, 13].

The above issues point to a more prominent problem forbug tracking systems in general: the lexical gap that nor-mally exists between bug reporters (e.g., testers, beta users)and developers. Reporters typically only have functionalknowledge of an app, even if they have development experi-ence themselves, whereas the developers working on an apptend to have intimate code level knowledge. When a de-veloper reads and attempts to comprehend (or reproduce)a bug report, she has to bridge this gap, reasoning aboutthe code level problems from the high-level functional de-scription in the bug report. If the lexical gap is too wide thedeveloper may not be able to reproduce and/or subsequentlyresolve the bug report.

To address this fundamental problem of making bug re-ports more useful (and reproducible) for developers, this pa-per presents the implementation of a novel tool, called Fu-sion, that facilitates reporters creating detailed bug reportsin order to provide more actionable information to develop-ers. Fusion implements the novel approach that was pre-sented and evaluated in our previous work [23]. Fusion firstemploys fully automated static and dynamic analysis tech-niques to gather screen-shots and other relevant informationabout an app before it is released for testing. Reporters theninteract with the web-based report generator using the auto-completion features in order to provide the bug reproductionsteps. By linking the information provided by the user withfeatures extracted through static and dynamic analyses, Fu-sion presents an augmented bug report to the developerthat contains actionable information with well-defined stepsto reproduce a bug.

2. THE FUSION REPORTING TOOLFusion’s current target user base consists of two ma-

jor groups: mobile application developers and beta users ortesters . As such there are two user-facing scenarios for the

1

tool’s operation. From a developer’s perspective the work-flow is purposefully simple; the only action required is thesubmission of an .apk of the latest build of the applicationfor which they want to enable bug reporting. Once this fileis submitted, Fusion’s automated program analysis tech-niques extract the information necessary to facilitate processof creating a bug from the reporters end. Once reports arefilled out, developers can access them through a web por-tal. From a reporter’s perspective, they construct a reportusing a web interface. After selecting the application forwhich they would like to report a bug, the reporter enters abrief description and contextual information about the bug(e.g. device used, screen orientation). They then use a seriesof auto-filled combo boxes to construct reproduction steps,including screenshots.

2.1 FUSION ArchitectureFusion’s architecture can be seen in Figure 1. First, Fu-

sion collects information related to the GUI componentsand event flow of an app using the Static Application An-alyzer and the Dynamic Analysis Engine. Then the toolleverages the information collected during the analysis tofacilitate a reporter constructing a detailed bug report withreproduction steps, and screenshots. The static and dy-namic analyses must be performed before each version ofan app is released for testing or before it is published toend users. Both program analysis components store theirextracted data in the Fusion database (Fig. 1 - 3 ).

2.1.1 Static AnalysisThe goal of the Static App Analyzer (Fig. 1 - 1 ) is to

extract all of the GUI components and associated informa-tion from the app source code. For each GUI component,the Primer extracts: (i) possible actions on that component,(ii) type of the component (e.g., Button, Spinner), (iii) ac-tivities the component is contained within, and (iv) classfiles where the component is instantiated. Thus, this resultsin a universe of possible components within the domain ofthe application, and establishes traceability links connect-ing GUI-components to code specific information such asthe class or activity where they are located.

The Static App Analyzer utilizes several tools to extractthe information outlined above. First it uses the dex2jar[4]and jd-cmd [8] tools for decompilation; then, it convertsthe source files to an XML-based representation using sr-

cML [11]. We also use apktool [2] to extract the resourcefiles from the app’s APK. The ids and types of GUI compo-nents were extracted from the xml files located in the app’sresource folders (i.e., /res/layout and /res/menu of thedecompiled application or src). Using the srcML represen-tation of the source code, we are able to parse and link theGUI-component information to extracted app source files.

2.1.2 Dynamic AnalysisThe Dynamic Analysis Engine (Fig. 1 - 2 ) is used to

glean dynamic contextual information and enhance the data-base with both run-time GUI and application event-flow in-formation. The goal of the Engine is to explore an app ina systematic manner, ripping and extracting run-time in-formation related to the GUI components during executionincluding: (i) the text associated with different GUI com-ponents (e.g., the “Send” text on a button to send an emailmessage), (ii) whether the GUI component triggers a tran-

Static Application Analyzer

Dynamic Analysis Engine

Physical Device or Emulator

Reporting Interface

Report Viewing Interface

FUSION Database

1 26

3

Auto-Completion

Engine

45

Figure 1: Fusion Architecture

sition to a different activity, (iii) the action performed onthe GUI component during systematic execution, (iv) fullscreen-shots before and after each action is performed, (v)the location of the GUI component object on the test de-vice’s screen, (vi) the current activity and window of eachstep, (vii) screen-shots of the specific GUI component, and(viii) the object index of the GUI component (to allow fordifferentiation between different instantiations of the sameGUI component on one screen).

The Engine performs this systematic exploration of theapp using the UIAutomator [1] framework included in theAndroid SDK. This systematic execution of the app is sim-ilar to existing approaches in GUI ripping [12, 14, 22, 21].Using the UIAutomator framework allows us to capture casesthat are not captured in previous tools such as pop-up menusthat exist within menus, internal windows, and the onscreenkeyboard. To effectively explore the application we imple-mented our own version of a systematic depth-first search(DFS) algorithm for application traversal that performs clickevents on all the clickable components in the GUI hierarchyreachable using the DFS heuristic.

2.2 Reporting Bugs with FUSIONDuring the Report Generation Phase, Fusion aids the re-

porter in constructing the steps needed to recreate a bug bymaking suggestions based upon the “potential” GUI statereached by the declared steps. This means for each step s,Fusion infers — online — the GUI state GUIs in whichthe target app should be by taking into account the historyof steps. For each step, Fusion verifies that the suggestionmade to the reporter is correct by presenting the reporterwith contextually relevant screen-shots, where the reporterselects the screen-shot corresponding to the current actionshe wants to describe.

2.2.1 Report Generator User InterfaceAfter first selecting the app to report an issue for, a re-

porter interacts with Fusion by filling in some identifyinginformation (i.e., name, device, title) and a brief textual de-scription of the bug in the top half of the UI. Next, thereporter inputs the steps to reproduce the bug using theauto-completion boxes in a step-wise manner, starting fromthe initial screen of a cold app launch1, and proceeds untilthe list of steps to reproduce the bug is exhausted. Accord-

1Cold-start means the first step is executed on the first win-dow and screen displayed directly after the app is launched.

2

Figure 2: Fusion Reporting Interface

ing to the various fields in Figure 2, the reporter would firstfill in their (i) name (Field 1), (ii) device (Field 2), (iii)screen orientation (Field 3), (iv) a bug report title (Field 4),and (v) a brief description of the bug (Field 5).

The first drop down list (see Figure 2 - Field 6) corre-sponds to the possible actions a user can perform at a givenpoint in app execution. For example, let’s say the reporterselects click as the first action in the sequence of steps asshown in Figure 2. The possible actions considered in Fu-sion are click(tap), long-click(long-touch), type, and swipe.The type action corresponds to a user entering informationfrom the device keyboard. When the reporter selects thetype option, we also present them with a text box to collectthe information she typed in the Android app.

The second dropdown list (see Figure 2 - outlined in blue)corresponds to the component associated with the action inthe step. Fusion presents the following information, whichcan also be seen in Figure 2: (i) Component Type: this is thetype of component that is being operated upon, (e.g. but-ton, spinner, checkbox), (ii) Component Text : the text asso-ciated with or located on the component, (iii) Relative Lo-cation: the relative location of the component on the screenaccording to the parameters in Figure 3, and (iv) Compo-nent Image: an in-situ (i.e., embedded in the dropdown list)image of the instance of the component. The relative lo-cation is displayed here to make it easier for reporters toreason about the on-screen location, rather than reasoningabout pixel values. In the example from Figure 2 above,Fusion will populate the component dropdown list with allof the clickable components in the main Activity since thisis the first step and the selected action was click.

Fusion uses two techniques to handle instances of seem-ingly identical GUI-components appearing on the same screen.

Figure 3: Relative Location Enumeration and Ex-ample Augmented Screenshot

First, it differentiates each duplicate component in the listthrough specifying text “Option #”. Second Fusion at-tempts to confirm the component entered by the reporter ateach step by fetching screen-shots from the Fusion databaserepresenting the entire device screen (e.g., Figure 3).

After the reporter makes selections from the drop-downlists, they have an opportunity to enter additional informa-tion for each step (e.g., a button had an unexpected behav-ior) in a natural language text entry field. For instance, thereporter might indicate that after pressing the “OK” buttonthe pop-up window took longer than expected to disappear.

2.2.2 Report Generator Auto-Completion EngineThe Auto-Completion Engine of the web-based report gen-

erator (Figure 1- 4 ) uses the information collected up-frontduring the Analysis Phase. When Fusion suggests comple-tions for the drop-down menus, it queries the database forthe corresponding state of the app event flow and suggestsinformation based on the past steps that the reporter has en-tered. Since we always assume a“cold”application start, theAuto-Completion Engine starts the reproduction steps entryprocess from the app’s main Activity. We then track the re-porter’s progress through the app using predictive measuresbased on past steps.

Fusion presents components to the reporter at the gran-ularity of activities, or application screens. During thesuggestion process, Fusion looks back through the historyof reported user interactions and looks for possible transi-tions from the previous steps to future steps depending onthe history of the components interacted with. If Fusion isunable to capture the last few steps from the reporter due tothe incomplete application execution model mentioned ear-lier, then Fusion presents the possibilities from all knownscreens of the application. Due to the limited nature of theDFS heuristic used by the Dynamic Analysis Engine, theremay be action-Gui-Component pairs that are not availablein the auto-filled combo boxes. To handle these cases grace-fully, we allow the reporter to select a special option whenthey cannot find a component in the auto-completed combobox. When picking this option, the reporter would manuallyfill in (i) the type of the component, (ii) any text associatedwith the GUI-component and (iii) the relative location ofthe GUI-component as denoted in Figure 3.

2.2.3 Viewing FUSION ReportsThe Auto Completion Engine saves each step to the database

as reporters complete bug reports. Once a reporter finishesfilling out the steps and completes the data entry process,a screen containing the final report, with an automaticallyassigned unique ID, is presented to the reporter and saved

3

Figure 4: Example Fusion Bug Report

to the database for a developer to view later (see Figure 4for an example report).

The Report presents information to developers in threemajor sections: First, preliminary information including thereport title, device, and short description (shown in Figure4 in blue). Second, a list of the Steps with the followinginformation regarding each step is displayed (highlighted inblue in Figure 4): (i) The action for each step, (ii) the typeof a component, (iii) the relative location of the component,(iv) the activity Java class where the component is instan-tiated in the source code, and (v) the component specificscreenshot. Third, a list of full screen-shots correspondingto each step is presented at the bottom of the page so the de-veloper can trace the steps through each application screen(this section is highlighted in green in Figure 4).

3. EVALUATIONTo evaluate Fusion (see full details in [23]) we investi-

gated its ease of use, as well as the reproducibility of theFusion reports compared to reports created using GoogleCode Issue Tracker (GCIT). First, in a bug-creation study werecruited eight students (four undergraduate or non-expertsand four graduate or experts) to construct bug reports us-ing Fusion and GCIT — as a representative of traditionalbug tracking systems— for 15 real-world world bugs for 14open-source apps from F-Droid [5]. We collected survey re-sponses from these participants regarding the ease of use anduser preferences of each tool. Next, in a bug-reproductionstudy we evaluated the reproducibility of the Fusion andGCIT reports generated by the first group of participants.These reports (120 for each type) and the original bug re-ports extracted from the respective app issue trackers wereevaluated by a new set of 20 graduate student participantsthrough attempted bug reproduction on physical devices.The results of this study indicate that developersusing Fusion reports are able to reproduce more re-ports compared to traditional bug tracking systemssuch as the GCIT.

4. DEMO REMARKS AND FUTURE WORKIn this demo, we presented Fusion, a novel implementa-

tion of an enhanced bug reporting system for Android ap-plications. Fusion facilitates reporters crafting detailed re-ports containing reproduction steps, screenshots, and trace-ability links to code artifacts, by informing the reportingprocess with data gleaned from static and dynamic programanalyses. As future work, we plan to (i) investigate more so-phisticated methods of modeling program behavior, such asusing statistical language models, (ii) to improve our DFSengine through supporting more gestures, (iii) and to useFUSION as a tool for reporting feature requests.

5. REFERENCES[1] Android uiautomator http://developer.android.com/

tools/help/uiautomator/index.html.[2] apktool https://code.google.com/p/android-apktool/.

[3] Bugzilla issue tracker https://bugzilla.mozilla.org.

[4] dex2jar https://code.google.com/p/dex2jar/.[5] F-droid. https://f-droid.org/.

[6] Github issue tracker https://github.com/features.

[7] Google code issue trackerhttps://code.google.com/p/support/wiki/IssueTracker.

[8] jd-cmd decompiler https://github.com/kwart/jd-cmd.[9] Jira bug reporting system

https://www.atlassian.com/software/jira.[10] Mantis bug reporting system https://www.mantisbt.org.

[11] srcml http://www.srcml.org.

[12] D. Amalfitano, A. R. Fasolino, P. Tramontana,S. De Carmine, and A. M. Memon. Using gui ripping forautomated testing of android applications. ASE’12, pages258–261.

[13] J. Aranda and G. Venolia. The secret life of bugs: Goingpast the errors and omissions in software repositories. pages298–308, May 2009.

[14] T. Azim and I. Neamtiu. Targeted and depth-firstexploration for systematic testing of android apps.OOPSLA ’13, pages 641–660. ACM, 2013.

[15] N. Bettenburg, S. Just, A. Schroter, C. Weiss, R. Premraj,and T. Zimmermann. What makes a good bug report?SIGSOFT ’08/FSE-16, pages 308–318.

[16] N. Bettenburg, R. Premraj, T. Zimmermann, and S. Kim.Extracting structural information from bug reports.MSR’08.

[17] N. Bettenburg, R. Premraj, T. Zimmermann, and S. Kim.Duplicate bug reports considered harmful... really? pages337–345, Sept 2008.

[18] S. Davies and M. Roper. What’s in a bug report? ESEM’14, pages 26:1–26:10.

[19] M. Erfani Joorabchi, M. Mirzaaghaei, and A. Mesbah.Works for me! characterizing non-reproducible bug reports.MSR 2014, pages 62–71.

[20] Ericsson. Ericsson mobility report.http://www.ericsson.com/res/docs/2014/ericsson-mobility-report-november-2014.pdf.

[21] M. Linares-Vasquez, M. White, C. Bernal-Cardenas,K. Moran, and D. Poshyvanyk. Mining android app usagesfor generating actionable gui-based execution scenarios. In12th Working Conference on Mining Software Repositories(MSR’15), to appear, 2015.

[22] A. Machiry, R. Tahiliani, and M. Naik. Dynodroid: Aninput generation system for android apps. ESEC/FSE’13,pages 224–234.

[23] K. Moran, M. Linares-Vasquez, C. Bernal-Cardenas, andD. Poshyvanyk. Auto-completing bug reports for androidapplications. ESEC/FSE’15, pages 673–686. ACM.

[24] G. Tassey. The economic impacts of inadequateinfrastructure for software testing. Technical report,National Institute of Standards and Technology, 2002.

4