ACaMIA: Automating Configuration across MultipleInteractive Applications

Paper # 22 — 14 pages

ABSTRACT

Desktop setup and configuration is a common task in anycomputing environment, and its automation would savetime and money. Existing automation tools are generallyapplication-specific (i.e., macros). Most non-trivial config-uration tasks, however, have dependencies across applica-tions. To allow automation in an application-independentmanner, past work circumvents the GUI by modifying thekernel to track causal dependencies across applications andthen taking diffs of the affected files. Such kernel modifica-tions, however, are unsuitable for the commercial operatingsystems typical in the desktop environment, and file diffsare fragile in the face of complex, application-specific andbinary file formats.

This paper introduces ACaMIA, a tool that uses a newapproach to automating configuration. Instead of circum-venting the GUI, ACaMIA recognizes that the GUI is theright layer for this task. Modern GUIs are rich enoughto tackle most configuration tasks directly, without fragilestate introspection techniques. And unlike application fileformats, they present a uniform interface that is relativelyconsistent across applications. By watching a few users per-form a configuration task, ACaMIA generates a canonicalautomation script that enables future users to perform thesame task. We implemented a prototype of ACaMIA andshowed that it works effectively in automating the configu-ration tasks in our lab that most commonly required admin-istrator intervention.

1 INTRODUCTION

Desktop configuration updates are basic ongoing tasksin enterprise and campus networks. New software instal-lations, upgrades, security patches, and problem fixes alltrigger configuration updates. Such updates typically in-volve user interaction via the graphical user interfaces ofmultiple applications. Installing a new software package,for example, can easily involve a Web browser interactingwith a software download site, an email client (to acquirea license for the package), an installer, and the package it-self. Automating this currently manual interaction promisesto save organizations both user time and administrativecost [12]. But despite these substantial incentives, existingautomation systems have significant limitations — they ei-ther work only within a single application (a typical ex-ample is Javascript macros, which work only within a Web

browser [3]) or require changes to the operating system ker-nel to track dependencies across applications [15], whichmakes them unacceptable to most organizations, which arecommitted to using a stock, proprietary, off the shelf oper-ating system.

This paper introduces ACaMIA, a new approach forautomating configuration updates. The basic idea behindACaMIA is to 1) record the graphical user interface actionsthat an initial, small set of users take when they performthe configuration update, 2) combine the resulting traces toextract a parameterized sequence of GUI actions that to-gether accomplish the update, and 3) apply this sequenceon demand to automatically perform the update to othermachines. A key insight behind this approach is the iden-tification of the graphical user interface as the appropriateinterposition layer for recording and replaying configura-tion actions. While working at this layer imposes significanttechnical challenges (which may be why previous systemswere designed to avoid dealing with the graphical user in-terface altogether [15]), the rewards for a viable solutionare substantial — the system has access to a universal in-terface that enables it to work seamlessly across multipleapplications, including applications it has never previouslyencountered.

ACaMIA generates a sequence of parameterized GUIactions that automate and canonicalize a particular config-uration task. Specifically, given a set of recorded configu-ration traces that together include the full range of GUI ac-tions required to perform the update on various machines,ACaMIA analyzes the set to classify the user actions as fol-lows:

• Auto Enter: Actions that are the same in all traces.Such actions are placed in the parameterized action se-quence intact.

• User Enter: Actions that only the human user knows.Examples include passwords and personal preferences.In the parameterized action sequence these actionsprompt the user for the information required to com-plete the action. Note that since ACaMIA recordspasswords at the level of the graphical user interface(which almost invariably obscures passwords), it doesnot record the actual passwords.

• Possible Auto Enter:Actions with a few common val-ues that occur repeatedly across the traces. A commonexample is printer names — organizations usually havea standard set of printers. In the parameterized action

1

sequence these actions let the user select one of the pre-viously observed values or enter a new value.

• Irrelevant: Actions that are irrelevant to the config-uration update at hand. A common example is inter-actions with entertainment software such as music orvideo players that take place while the user is perform-ing the configuration task. These actions to not appearin the parameterized action sequence.

The key technical challenge for ACaMIA is combiningmultiple recorded GUI traces to obtain a single parameter-ized sequence of actions that will perform the update onother machines. This process is difficult for three reasons.

• First, UI objects have no global names and their statedepends on the UI context. For example a text box hasno global name and does not take input unless it is in anopen window and the window has no open child win-dow. Thus, ACaMIA creates its own naming system thatincludes the UI context, and hence identifies GUI ob-jects and their state across different machines.

• Second, a trace of a successful configuration activitymay contain erroneous segments. For example, a usermay enter a wrong value, move forward to the next setof windows, realize her mistake, go back to the corre-sponding window, correct the error, then continue herconfiguration task. ACaMIA parses the traces to learnwhen a value is committed and removes erroneous as-signments.

• Third, ACaMIA must map sequences of actions thatoccur in different orders in different traces. For exam-ple, one trace may have the user download a softwarepackage, then obtain a certificate, and a second tracemay show the two tasks in reverse order. ACaMIA can-not address this problem by ignoring action order al-together, because in most configurations order matters,e.g., one cannot access a child window unless one opensits parent window. ACaMIA uses the UI context to mapthe same actions across traces. It learns valid action or-der by exploiting that actions are possible only whenthey become in-focus, e.g., one has to click the filemenu before the “save” action is available to the userinterface.

We implemented ACaMIA using the Microsoft ActiveAccessibility interface [4]. Originally developed to enableaccessibility aids for users with impaired vision, the ac-cessibility interface has been built into all versions of theWindows platform since Windows 98 and is now widelysupported on this platform [5]. Moreover, there is a strongpush to incorporate similar functionality in Linux and otheropen software projects [11].

We evaluated ACaMIA by using it to automate thefive most common configuration updates in our departmentthat typically require administrator intervention, such asIMAP configuration, printer configuration, etc. Our eval-uation employs a population of 12 users, who perform the

configuration updates both manually and using ACaMIA.The users perform the configurations on their personal ma-chines. Our results show that ACaMIA is effective at au-tomating configuration updates. Specifically:

• ACaMIA reduced the average time a user spent on theseconfiguration problems by almost 3x.

• ACaMIA automated over 90% of the required ac-tions, thus reducing the average number of user ac-tions per configuration task from 25 to less than 2. Ofthese remaining non-automated actions, more than 90%of them are passwords or other personal informationwhich cannot be generically automated.

2 RELATED WORK

Configuration management is an important, unsolvedarea in systems research. The closest to our work is Au-toBash [15], which helps recover from configuration errorsby recording user actions to fix a problem on one computerand replaying them on another computer that is experienc-ing the same problem. ACaMIA is inspired by AutoBashbut differs from it in two important ways. First, AutoBashrequires kernel modifications to track causal dependenciesacross processes and system states. Kernel modificationshowever are unsuitable for the commercial operating sys-tems typical in the desktop environment. Second, AutoBashautomates configuration updates by taking diffs of the af-fected files and applying them to a different machine. Filediffs however are fragile in the face of complex application-specific file formats. ACaMIA operates at the user interfacewhich, unlike file formats, is relatively-consistent acrossapplications. Furthermore, ACaMIA does not require ker-nel modifications.

Much prior work focuses on diagnosing configura-tion problems. PeerPressure [16] and its predecessor,Strider [17] diagnose configuration problems caused by er-roneous values in the Windows registry. They trace registryentries affected by an application and use statistical meth-ods to identify the registry entry that is likely to have causedthe problem. Strider compares registry entries on a sick ma-chine against their values on a healthy machine, whereasPeerPressure eliminates the need to explicitly identify ahealthy machine by resorting to a large population of ma-chines, under the assumption that most machines have cor-rect states. Yuan et al. [19] diagnose runtime problems bymapping low-level system description (sequences of sys-tem calls) to known problems using a Support Vector Ma-chines (SVM) classifier. Chronus [18] tries to find the pointin time when a system’s configuration became erroneous.It does so by testing a predicate against past snapshots ofsystem state, and solves the problem by rolling back to themost recent operational state. These systems identify theroot cause of a configuration problem but do not automatecomplex or new configurations. In contrast, ACaMIA auto-mates new and complex configurations that the user wants

2

to perform but does not know how to perform, or does notwant to spend time or effort to figure them out.

NetPrints [6] diagnoses configuration problems in homenetworks. It collects examples of good and bad networkconfigurations, builds a decision tree, and then determinesthe set of configuration changes that must happen to changea configuration from bad to good. This approach is success-ful for fixing configuration in the home network, where theset of possible configurations is relatively small. However,it becomes intractable with complex configuration taskslike those addressed by ACaMIA. For example, PeerPres-sure [16] applies a similar approach to NetPrints to diag-nose Windows configuration problems, but given the com-plexity of the environment, it focuses on finding a singleproblematic registry entry.

Mirage [9] addresses configuration upgrade. It clustersmachines according to their environment (i.e., OS, runtimelibraries, executables, environment variables, and configu-ration files) and argues that upgrades should be performedin a staged approach by picking few machines in each clus-ter. Mirage is complementary to ACaMIA, and its cluster-ing approach can be incorporated in the process of creatinga generalized trace.

ACaMIA in some ways falls in the area of Program-ming by Example (PBE). PBE refers to a programmingtechnique where the system records actions performed bya user and produces a generalized program that can be usedlater in similar scenarios [10]. Past works on PBE how-ever require the intervention of a human programmer torecord user actions in a scripting language and parameter-ize the script to work in various environments [8, 13, 14,7]. These projects focus on developing high-level script-ing languages that are easy to use by an average user [?,13]. They are application-dependent, (e.g., Chickenfoot en-ables a user to customize her Web browser), or operate inspecial research environments (e.g., DocWizard provideswalk through documentation for the Eclipse system [7]).ACaMIA departs from prior work on PBE in three mainways. First, it addresses configuration updates across ap-plications and in typical working environments (e.g., Win-dows). Second, instead of requiring a human to write au-tomation scripts, it exploits Windows accessibility interfaceto automate recording and replay. Third, it develops new al-gorithms that combine traces of user actions to automate aconfiguration update and eliminate irrelevant actions andpersonal information.

Finally, there are a few commercial tools that allow ahuman expert to write a script that automates a specificconfiguration task (i.e., a macro). These include AutoIt [2],AutoHotKey [1], iMacros [3]. In contrast to these tools,ACaMIA does not require a human to write a macro foreach configuration task; By watching a few users perform aconfiguration task, ACaMIA generates a canonical automa-tion script that enables future users to perform the same

• Download and install the latest version of Thunderbird

• At the Account Wizard - New Account Setup, choose 'Email Account' click next.

• Enter your full name as well as your full email address

• Next, for Server Information, choose IMAP

• Incoming Server: imap.FooLab.FooUniversity.edu

• Check 'Use Global Inbox'

• Outgoing Server: outgoing.FooLab.FooUniversity.edu

• Enter your IMAP email name for Incoming and Outgoing User Name

• For Account Name, enter your full email address, and Click finish

• Next, you need to configure IMAP

• From the Thunderbird Menu, Click Tools, then Click Account Settings

• From the Account Setting menu on the left, choose Server Settings.

• Under the Security Settings - Use Secure Connection option, choose "SSL" and

click "Use Secure Authentication

• While still in the Account Settings window, Choose the Outgoing Server from

the menu on the left and click edit.

• Change the Security Authentication to use "SSL"

• Click OK. Click OK again to save your Account Settings information.

• Double click the inbox

• When prompted for a certificate, choose " Accept This Certificate

Permanently."

• You will now be prompted to enter your password.

• You should now be able to securely send and receive email

Figure 1—Documentation for configuring IMAP

task.

3 ACAMIA’ S USAGE SCENARIO

Consider, for example, configuring a new email account– a task that occurs for every new student, professor, andstaff member in our department. In an attempt to reducethe amount of time that administrators spend dealing withthis task, the infrastructure group provides online documen-tation that walks the users through the configuration. Asshown in Fig. 1, the task involves a relatively complex pro-cedure and requires the user to interact with multiple pro-grams (the Web browser, the installer, and the mail pro-gram). Experience has shown that many users are unable tocomplete the task on their own despite the detailed docu-mentation, and eventually seek the administrator’s help.

ACaMIA can change this scenario. ACaMIA watches afew users as they set up their email account and combinestheir actions to generate a canonical form that enables fu-ture users to perform the same task. Specifically, a user whois willing to share her configuration solution with otherusers in the department, starts by activating ACaMIA’srecording button. She then goes about her configurationprocedure following some online documentation like that inFig. 1 or her own experience. Once she has successfully fin-ished her configuration, she stops the recording and uploadsthe resulting trace to the configuration server by clickingthe upload button in the ACaMIA window. ACaMIA willask the user whether the configuration was successful, andwill upload only configuration traces deemed successful bythe user.

When an ACaMIA client uploads a trace, the configu-ration server asks the user for her name and password. Theserver accepts solutions only from a user with a valid pass-word and labels each solution with the user’s name. Theserver also asks the user to label her configuration by pick-ing from a list of known configuration tasks that are of in-

3

terest to the user’s organization, e.g., “Setting your IMAPAccount”. If none of the existing labels applies, the usertypes a new label. The server analyzes the traces to producea canonicalized automated configuration, which it continu-ously refines as more traces are uploaded.

Next, the server processes the traces associated with aconfiguration task to produce a canonicalized automatedconfiguration. This process involves a few important steps.

• Filter irrelevant information. A common example isinteractions with entertainment software such as musicor video while the user is performing the configurationtask. These events are irrelevant to the original configu-ration task and should not be included in the automatedsolution.

• Filter user mistakes. A user may provide a wrongvalue while performing a particular assignment, whichshe corrects later. For example, a user who is follow-ing the instructions in Fig. 1 may make a typograph-ical error while entering the name of the IMAP server.ACaMIA needs to identify such mistakes and filter themout, otherwise many user inputs may look random andunamenable to automation – e.g., without filtering thewrong server name, ACaMIA may confuse the name ofthe server as private information though it is the samefor the whole department, and hence can be easily auto-mated.

• Parameterize user inputs.The server compares thetraces to identify which user inputs are private and needto be provided by the user and which are public and canbe gleaned from the traces. For the email configurationexample, a user’s name and password cannot be enteredautomatically, while the IMAP server is likely the samefor everyone in an enterprise network and hence can beautomated or proposed as the default option.

• Generate if-then automated configurations. Thestates of the different machines may vary in ways thataffect the set of actions that must be performed to up-date the configuration. Consider again the example inFig. 1. Some users may already have Thunderbird in-stalled while others do not. Thus, the ACaMIA serverparses the traces for different execution paths, and gen-erates if-then automated solutions.

Once an automated configuration is generated, it isadded to a database that users can browse online. Nexttime a user needs to perform the configuration task, shecan download the automated configuration and ask herACaMIA client to play it. ACaMIA will run the automatedconfiguration and stop only when it needs a private inputfrom the user. For example, an ACaMIA automated solu-tion of the configuration in Fig. 1 would stop only at thetwo highlighted bullets to allow the user to enter her name,email account, and password.

Figure 2—Windows, Views, and Objects.A window may have differenttabs, each tab is a view. This figure for example shows a particular viewof the “Options” window in Thunderbird. The checkbox labeled “Displayemotions and graphics” is an object in this view.

4 DESIGN AND I MPLEMENTATION

This section describes how ACaMIA performs its job.We start with a formal model that defines the UI objects thatACaMIA interacts with, how they are navigated, and whatinteraction capabilities are feasible. We then tie the modelto the system implementation based on the UI accessibil-ity interface. Next, we proceed to describe how ACaMIAparses the traces, filters irrelevant and erroneous events,classify user inputs into private and public, and finally au-tomates a configuration task.

5 SYSTEM M ODEL

Modern user interfaces are fairly complex. Having aformal model enables us to capture ACaMIA’s behavior inan accurate implementation-independentmanner. This sim-plifies reasoning about the design and porting it to differentimplementations.

5.1 UI and System State

The system state is captured in a set of typed state vari-ablessv∈ SV. Each state variable has a valuevalue(sv) ∈String ∪ Integer ∪ Boolean ∪ list(String ∪ Integer ∪Boolean).

Both human users and ACaMIA interact with the sys-tem state via the user interface (UI). The UI defines threetypes of entities: objects, views, and windows. Objects en-able a user to view and edit the system state (e.g., a textboxor a checkbox) or to navigate between views (e.g., theNext button). Each viewv has a fixed set of UI objectsobjects(v) = {o1, ...,ok}. Views maintain the context ofan object and also serve as navigation units, i.e., a usermoves between views and cannot access an object unlessshe first accesses the view containing that object. Windowsdefine a tree hierarchy whose root is the desktop. Each win-dow w has a fixed set of viewsviews(w) = {v1, ...,vk}.We can also identify the window of a particular viewv as

4

w = window(v). The views in a window may refer to dif-ferent tabs, or a sequence of dialog boxes that a user nav-igates using theNext andBack buttons. Fig. 2 shows anexample of these three entities. The figure shows the “Op-tions” window in Thunderbird. In particular, it shows the“Display/Formating” view in the “Options” window. Thecheckbox labeled “Display emotions and graphics” is anobject in this view.

We define a function that takes an object in a view andreturns the associated state variablesv = variable(v, o).Note that since objects are only defined within a con-text (e.g., a textbox is meaningless without a contextview), variable cannot be defined on an object alonewithout identifying its view. We require that each statevariable is associated with a single UI object, i.e., thatvariable(v1, o1) = variable(v2, o2) only if (v1, o1) =(v2, o2) or variable(v1, o1) is empty, in which case the ob-ject is not associated with any state variable. There is a nav-igation relation between views. Sov2 = goto(v1, o) meansthat one can move from viewv1 to v2 by clicking on buttono in view v1. Also, there is hierarchy⊑ on windows corre-sponding to nesting within the user interface. Sow1 ⊑ w2

if w1 is a child ofw2.

5.2 User Actions, Events, and Traces

Users interact with the user interface by performing ac-tions a ∈ A. Examples of actions include setting the textin a textbox to a certain value and checking or uncheck-ing a checkbox. More examples include pressing a but-ton that causes another view such as a dialog box to ap-pear and pressing theOK button in a view that causes thestate changes in the view to be written back into the systemstate. Each action operates on an object and takes place in aview. The combination of an action, an object, a view is anevente∈ E. Each user interaction session produces a tracet ∈ T, where each tracet consists of a sequence of eventse1.e2...ek. There are several kinds of events:

• Assign State Events:Events that correspond to poten-tial assignments of system state (we say potential stateassignment because in some cases the assignment maynot actually take effect until a later event, such as theuser pressing anOK button, causes the state change tobe written back into the system state). Each assign stateevent has the form〈assign, v, o:=x〉, which assigns thevaluex to the state variablevariable(v, o) of objecto inview v.

• Augment State Events:Events that correspond to po-tential operations that add new entries to system state.An example is a list box operation that adds a new itemto a list of items in the system state. Each augment stateevent has the form〈aug, v, o, x〉, which corresponds toan action that uses the valuex to augment the state vari-ablevariable(v, o) of objecto in view v.A key difference between assign state events and aug-

ment state events is that an assign state event is idem-potent and eliminates the effect of all previous assignevents. In contrast, the value of a system state variableupdated with an augment state event depends on not justthe latest event, but also on previous events that alsoaugment the same system state variable.

• Remove State Events:This is the opposite of an aug-ment event. An example is a list box operation that re-moves an item from the list of items in the system state.Each remove state event has the form〈rem, v, o, x〉,which corresponds to an action that removes the itemwhose value isx from the state variablevariable(v, o)of objecto in view v.

• Navigate Events:During the operation of the user in-terface only a (typically very small) subset of the setof views is available, i.e., open and enabled for userinteraction. At any point during a user interaction ses-sion there is a set of available viewsAv ⊆ V. Events(such as opening the Control Panel or selecting one ofa set of tabbed windows) that change this set of avail-able views are callednavigation events. Each naviga-tion event takes place in the context of an already avail-able view.Each navigate event has the form〈nav, v, b, a, i〉, wherev is the available view in which the navigation event oc-curs,b is the name of the object whose activation causesthe navigate event to occur (for example, a button thatcauses another view to appear when the user selects it),a ⊆ V is a set of views that the navigation event makesavailable, andi ⊆ V is a set of views that the navigationevent makes inavailable.

• Commit Events: Events that cause pending statechanges (from state events) to be written back into thesystem state. Commit events can commit a single ob-ject, all objects in a view, or all objects in all view ina window. Commit events of the form〈com, v, b, o〉means that buttonb in view v causes the state changeof objecto in v to be written in the system state. Com-mit events of the form〈com, v, b〉 cause all of the statechanges for all of the objects in viewv (these objects areobjects(v)) to be written back to the system state. Com-mit events of the form〈w-com, v, b〉 mean that buttonbin view v commits the pending states of all objects in allviews in the window containing viewv (i.e., it commitsall objects in all views inw = window(v)).

• Commit and Navigate Events:These events commitpending state changes then navigate the system to a newview. They have the form〈com&nav, v1, b, v2〉 wherebuttonb in v1 commits all states in viewv1, closes viewv1 making it inavailable and navigates tov2. An alterna-tive form is 〈w-com&nav, v1, b, v2〉, where buttonb inview v1 commits all pending states in all views of thewindow w = window(v1), closes windoww and all ofits views and navigates to viewv2.

5

• Abort Events: These events abort pending statechanges. Similarly to commit events, we can defineabort events over an object (i.e.,〈abort, v, b, o〉), awhole view (i.e.,〈abort, v, b〉), or a whole window, (i.e.,〈abort, v, b〉).

• Abort and Navigate Events:These events abort pend-ing state changes then navigate the system to a newview. They have the form〈abort&nav, v1, b, v2〉 wherebuttonb in v1 aborts all states in viewv1, closes viewv1 making it inavailable and navigates tov2. An alter-native form is〈w-abort&nav, v1, b, v2〉, where buttonbin view v1 aborts all pending states in all views of thewindow w = window(v1), closes windoww and all ofits views and navigates to viewv2.

5.3 Mapping Model to Accessibility Interface

We implement ACaMIA using the Microsoft ActiveAccessibility interface. ACaMIA obtains all informationabout the structure of the user interface from the accessi-bility interface and the OS’s window manager. In particu-lar, ACaMIA obtains from the OS information about low-level inputs such as mouse-clicks and keyboard presses, aswell as the window handle for where the event occurred.ACaMIA uses this information to query the accessibilityinterface for the current view of the window and its objectsand their states (i.e., it queries the accessibility interfacefor the widget that has keyboard focus or where the mouseclick has occurred).

Naming Objects and Views:The first challenge forACaMIA is that UI objects, views, and windows have noglobal names that work across machines or even acrossreboots of the same machine. To address this challenge,ACaMIA defines its own naming system that carries the vi-sual context. Specifically, ACaMIA refers to a UI object byconcatenating its role and name as returned by the acces-sibility interface. For example, the checkbox in Fig. 2, hasa role=checkbox and name=”Display emotions as graph-ics”. Clearly, this object name may not be unique and ismeaningful only in the context of the view in Fig. 2. Thus,ACaMIA carries the view of an object as a context for anyaction that involves the object. ACaMIA gives each viewa primary name and a secondary name. The primary namehas to be the same for two views to be the same. It is cre-ated by taking a hash of the names of the permanent objectsin the view. Said differently, we name a view using the col-lection of its objects. We exclude a few objects which maynot always appear in a view, such as scroll bars (which maydisappear if the user enlarges the window), tool bars, andlist items. The secondary name is used only when the pri-mary name matches multiple views. The secondary name isa concatenation of the default values in the view (like the ti-tle of the window) and the non-permanent objects. Finally,we refer to a window by the set of views that it contains.Note that do not need to have window names that workacross machines. We use the window concept to define the

relation between views. Thus, we only maintain the relationbetween views (e.g., a set of views is a child of another setof views).

Recording Events:As stated earlier, ACaMIA tracksmouse clicks and keyboard strocks, and queries the accessi-bility interface to retrieve the view and object that receivedthe mouse click or the keyboard focus. ACaMIA uses thisinformation to create an event trace. The accessibility inter-face does not classify an event into assign event, navigateevent, commit event, etc. ACaMIA has to label each eventby tracking how the associated view and object state havechanged as a result. It uses the following rules:

• Object state (but not number of items) changes and viewdoes not change→Assign state event.This event occurswhen a user types in a textbox or checks/unchecks acheckbox, etc.

• List object has an extra item in it and view does notchange→ Augment event.Examples include adding anemail account to an account list, or adding a new font toa font list, etc.

• List object has one fewer item and view does not change→ Remove event.This is the opposite of the previousevent.

• Neither object nor view changes, and object type isnot inert→ commit event.We have encountered caseswhere a button click does not make any visually observ-able change. These buttons are typically associated witha hidden state, like rebooting a process, or flushing acache, etc. In this case, ACaMIA cannot read the valueof the state variable associated with the button if any.In the absence of further information, ACaMIA tries topreserve the event in case it is necessary for the config-uration task. Hence, it labels it as a commit event asso-ciated with a hidden state variable that ACaMIA cannotread.

• View changes and object is theCancel or Abort button→ Abort and navigate event.While navigation and as-signment events are easy to identify from the visual con-text, commit and abort events are more subtle. Our cur-rent implementation uses a heuristic that is based on thename of a button (e.g., cancel) and common navigationmethods to guess whether a particular button commitspending states. In particular,Cancel andAbort buttonsare mapped to〈w-abort&nav, v1, o, v2〉, i.e., they abortall pending state for the whole window that contains thecurrent view.

• View changes to a view in the parent window, and objectis neitherCancel nor Abort → Commit and navigateevent.Whenever a user closes a child view and goesback to its parent window, she either aborts her actionsor commit them. If she went back to the parent windowby pressing aCancel or anAbort button, then the pre-vious rule applies and all pending states in all views inthe current window are aborted. Otherwise all pending

6

state in all views in the current window are committed.• View changes to a view that is not in the parent window

and object is not theCancel button→ Navigate event.This event happens when a user moves between tabbedviews in the same window, follows theNext andBackbuttons in wizards, or moves between different appli-cation windows. These cases however differ in whetherthe previous view is still available or not. For example,moving between tabs in the same window closes thecurrent view and opens a new one, while moving be-tween completely different applications keeps both thecurrent and new views available.

Replaying Events:ACaMIA merges the traces to cre-ate a canonical automated trace that applies the configu-ration on new machines (see below). Once such a trace isavailable, replaying it on a new machine is relatively sim-ple. At each step, ACaMIA queries the accessibility inter-face for all visible views. It uses the view primary name– and when needed the secondary name – to find the rele-vant view for the current event. It parses that view to findthe relevant object, and applies the event to the object. Af-ter it applies an event, ACaMIA waits for a short periodthen queries the accessibility interface to check the visualoutcome of the applied event. If the currently accessibleviews enable the next event (e.g., the checkbox needed tobe checked is in an accessible view), ACaMIA proceeds toapply the next event. Otherwise, ACaMIA waits for a shortperiod and checks again whether the view of the next eventis available. It retries for a few times and aborts if unsuc-cessful at finding the next view. Depending on CPU load,there might be a short delay before a UI event takes place.To deal with this issue, ACaMIA exploits that the UI guidesthe user to which events are available. Specifically, onlythose events that involve available views (i.e., a view thatis open and available for user interaction) can take placeat any point in time. Thus, ACaMIA needs to wait until aview becomes available to perform an action on one of itsobjects. Information about which views are available canbe obtained via the accessibility interface.

5.4 Filtering Irrelevant Activities

During the collection of traces that perform a given con-figuration update, users may perform irrelevant activitiessuch as listening to music or randomly exploring the UI.These irrelevant activities cause spurious events to occurinthe collected trace. ACaMIA aims to detect these irrelevantactivities and eliminate any events associated with them.

At a high-level, ACaMIA takes a standard approach toaddress this problem, namely it assumes that any given ir-relevant activity is uncommon and does not repeat acrossmany traces. For each view, ACaMIA computes the per-centage of traces that include that view at least once (i.e.,they have events occurring in that view). ACaMIA attemptsto exclude views that appear sufficiently rarely, as long as

later views deemed essential do not have some direct de-pendency on these views (e.g. a user followed an unusualpath to navigate to an essential dialog). ACaMIA also at-tempts to excludes views that can be shown not to includeany commit events and those associated with OS processesotherwise identified as irrelevant to the task at hand. Thenavigational events that invoked these views are removed,along with any update and commit events that might’ve oc-cured within it.

5.5 Filtering Redundant or Erroneous Assignments

Users may make mistakes during a configuration activ-ity, then go back and correct the mistake. For example, auser may enter a wrong value, move forward to the nextset of windows, realize her mistake, go back to the corre-sponding window, correct the error, then continue her con-figuration task. ACaMIA parses the recorded traces to learnwhen an assignment, augment, or delete event, is commit-ted and filters erroneous or redundant such events. Belowwe describe the detailed algorithm.

The algorithm processes the trace twice: once back-wards to remove all assignment/augment/delete events ex-cept the last committed event, and once forward to re-move all corresponding commit events except the com-mit event immediately following the last committed assign-ment/augment/delete event.Backward Pass:The backward pass maintains two sets ofview, object tuples: a pending setP and a committed setC. Both sets are initialized to the empty set. The algorithmthen performs the following actions for each event as theevents are encountered during the backwards pass over thetrace.

• Commit Events: For a commit evente = 〈com, v, o〉,the algorithm checks to see if〈v, o〉 ∈ C. If not, it adds〈v, o〉 to the setP of pending objects. Otherwise, it doesnothing.For a commit evente = 〈com, v, b〉, the algorithmchecks allo ∈ objects(v). If 〈v, o〉 6∈ C, the algorithmadds〈v, o〉 to P. Otherwise, it does nothing. Similarly,for all events of the form〈w-com, v, b〉, the algorithmchecks allo ∈ objects(vi) for anyvi that is a view inw = window(v).

• Commit and Navigate Events:are treated similarly tocommit events.

• Assign/Augment/Delete Events:Assign, delete, andaugment events are all treated similarly. Thus, we limitour description to assign events. For an assign evente = 〈assign, v, o:=x〉, the algorithm checks to see if〈v, o〉 ∈ P. If so, it adds〈v, o〉 to C and removes〈v, o〉from P. Otherwise, it removese from the enclosingtrace. This is because either a subseqent event will over-write the effect ofe or there is no commit event forefollowing e in the trace.

• Navigate Events:Navigation events have no effect on

7

effect on pending and committed assignments. Note thatwhile navigation may cause a view o become unavail-able it does not mean that the assignments have beenaborted. An example occurs when a user open a childwindow, the resulting view will sit on top of the parentwindow making all parent views unavailable for userinteraction. However pending assignments in the parentwindow are not lost.

• Abort Events: Abort events act in the opposite way ofcommit events, i.e., they make the algorithm remove thecorresponding tuple〈v, o〉 from the pending setP, caus-ing these uncommitted assignments to be lost. For ex-ample, the evente = 〈abort, v, b〉, causes the algorithmto remove all tuples that involve objects in viewv, i.e.,〈v, o〉, from P. The same applies for aborting a singleobject assignment or a whole window.

• Abort and Navigate Events: are treated similarly toabort events.

At the end of the backward pass, there is at most oneassign/augment/delate event〈v, o〉 in the trace for eachv, opair. Moreover, there is also a corresponding commit eventeafter〈v, o〉 in the trace. There may also be other redundantcommit events for〈v, o〉. The forward pass removes theseredundant commit events.Forward Pass: The forward pass maintains two sets ofview, object tuples: a pending setP and a found setF. Bothare initialized to the empty set. The algorithm performs thefollowing action for each commit event that it encountersduring the forward pass over the trace:

• Object Commit Events: For each object commit evente = 〈com, v, o〉, the algorithm checks if〈v, o〉 ∈ P, i.e.,there is an assigment to the corresponding tuple that ispending to be committed. If so, it removes〈v, o〉 fromP and adds〈v, o〉 to F. Otherwise, it removese fromthe trace —e is redundant because there is no pend-ing assign state event for it to commit. Either a previ-ous commit event committed the last committed assignevent for〈v, o〉 or the last committed assign event for〈v, o〉 has yet to appear in the trace.

• View and Window Commit Events: Commit eventsthat commit a whole view or a whole window are re-duced to object commit events for all objects in the cor-responding view or window. They are then processed asdescribed above.

• Commit and Navigagte Events:Commit and navigateevents are decomposed into commit events, which aretreated as described above, and navigate events whichare ignored during the forward pass.

• Assign/Augment/Delete Events: For each as-sign/augment/delete event, the algorithm adds theaffected tuple〈v, o〉 to P.

Finally, the forward pass can be augmented to removeredundant abort events in a similar manner to how we re-moved redundant commit events.

5.6 Parameterize User Inputs

Now that ACaMIA has removed erroneous and unnec-essary assign/augment/delete events, it can compare thetraces to identify which user inputs are private and needto be provided by the human, and which are public and canbe gleaned from recorded traces.

For each object in each view, ACaMIA parses alltraces to find all unique values that were given to thatobject via assign/augment/delete events. It also finds thenumber of traces that have each value. It creates a tablethat maps〈v, o〉 to list(tupleaction, value, occurrences).For example, say that〈v, o〉 refers to a view andtextbox object where the user types the name ofher printer. The corresponding table entry may looklike 〈assign, printer floor1, 8〉, 〈assign, printer floor2, 2〉,which indicates that, in 8 traces, the user typed“printer floor1” when asked for the name of her printer,and in 2 traces, the user typed “printerfloor2”. Another en-try could refer to a specific checkbox and may have thevalue list(〈assign, checked, 10〉), which means that in all10 traces the user checked this checkbox.

Next, ACaMIA parses the recorded trace to parametrizeassign/augment/delete events, i.e., to decide whether it canautomate them or needs to take user input. Without lossof generality, let us focus on assign events. For each suchevent, ACaMIA looks up the table entry for the associatedtuple tuplev, o and labels the event with one the followinglabels.

• Auto Enter: The is the case if the tupletuplev, o is as-signed the same value in all recorded traces. Any assignevent to such tuple will be performed automatically byassigning the value from the traces.

• User Enter: The is the case if the tupletuplev, o takesunique values across traces. Hence, ACaMIA has to askfor user input when performing the assign event.

• Possible Auto Enter: The is the case if the tupletuplev, o shows a few common values that occur repeat-edly across traces (e.g., a printer name). When ACaMIAneeds to assign a value to such tuple, it lets the user se-lect one of the previously observed values or enter a newvalue.

Note that events that involve passwords are alwaystreated as user-enter. Specifically, since ACaMIA recordspasswords at the level of the graphical user interface (whichalmost invariably obscures passwords), it does not recordthe actual passwords. Furthermore, whenever it sees suchobscured entry in a trace it always marks the correspondingassign event as user-enter.

5.7 Filtering Unnecessary Navigation Events

Next, ACaMIA removes unnecessary navigation events.These events may result from users making navigation mis-takes, or they may be left from the previous section where

8

we removed the unnecessary assignment events but not theevents that navigated to/from the corresponding views. Re-moving unnecessary navigation events is not a prerequisitefor correct configuration. However, it simplifies automationand may save significant execution time.

Our approach to removing unnecessary navigationevents relies on two ideas. First, we can remove any se-quence of only navigation events as long as we can nav-igate immediately from the available views before the se-quence to the next view after the sequence. Second, in-stead of navigating to a particular view multiple times, eachtime changing the state of a subset of its objects, we wouldlike to navigate to a view only once, make all object statechanges in that view, and leave the view. Thus, our secondoptimization tries to delay all state changes to the last timewe enter a view, and removes all unnecessary navigation tothat view. Below we describe an algorithm that implementsthese ideas.

The algorithm operates with the aid of four two-dimensional tables. Each column of each table correspondsto a time step in the trace. In these tables, only navigationevents cause transitions between time steps (i.e., transitionsbetween columns). Each row of each table contains infor-mation for a single view. Each entry in each table thereforecontains information about a given view at a given pointduring the trace.

• Available Views: The tableavailablei,j : Booleanrecords whether viewvi is available (for user interac-tion) at time stepj.

• Event Lists: The tableeventsi,j : list(event) containsthe list of events other than nagivation events performedin view vi at time stepj.

• Necessary Views:The tablenecessaryi,j : Booleanrecords whether a given viewvi is necessary at stepj.There are two ways for a viewvi to be necessary atstepj: either some events occur invi during time stepj, or vi presents acommon viewacross traces and timestepj is the first time step whenvi is available. Com-mon views are identified per application. In§5.4, wehave described how we associate a view with its appli-cation (e.g., a FireFox view). A view is a common viewif it appears at least once in all traces that run the cor-responding application. We deem such common viewsas necessary and ensure that they stay in the automatedversion.

• Navigation Events: The tablenavigationsj : eventcontains a single row that refers to the navigation eventthat caused the transition from columnj to columnj +1in the other tables.

5.7.1 Initialization step

Given a trace, the algorithm builds the above four ta-bles by simulating the execution of the trace forwards intime. The table construction algorithm maintains a current

set of available viewsA and the current time stept. When itprocesses a nagivation evente it sets columnt of availablebased onA, setsnavigationst to e, updatesA based on howe affects the views’ availability, then incrementst. When itprocesses an evente that is not a navigation event, it addseto the list of events stored ineventi,t, whereeoccurs in viewvi . If e is the first event in this list, it setsnecessaryi,t totrue.

5.7.2 Event re-location step

The algorithm next moves all non-navigational eventsthat occur in viewvi at any time step to the last availableoccurrence ofvi , and updates theevent table. It also up-dates thenecessary table to reflect the move – i.e., someearlier occurences of a view may become unnecessary be-cause their events were moved to later occurences.

The goal of this step is to bring all of the events thatoccur at view together with the hope of rendering earlieroccurences of the view unnecessary and hence potentiallyremovable from the final trace.

5.7.3 Removing Redundant Navigation Events

Consider any contiguous collection of time stepst1, ...,tn. It is possible to remove the navigation events fortime stepst2, ...,tn if the following conditions are met:

• Necessary View Condition: None of the view in-stances in columnst2, ...,tn−1 is necessary, i.e.,necessaryi,j is false for alli andt1 < j < tn. The ratio-nale is that all of these view instances will be removedwhen the navigation events are removed, so they mustnot be necessary.

• Available View Condition: The set of available viewsat time stept1 is a superset of the set of available viewsat time steptn, i.e.,availablei,tN impliesavailablei,t1 forall i. The rationale is that the navigation events after steptn will operate starting from the available views fromtime steptn. After removing the intermediate navigationevents, all of these available views will still be availablebecause they were open at time stept1, the new timestep from which the successive navigation events willstart.

The algorithm that removes redundant navigationevents then simply processes the time steps in reverse or-der to find maximal sequences of navigation events that itcan remove. The algorithm itself processes each time stepin turn, searching backwards from that time step to find amaximal sequence of time steps that satisfy the two con-ditions above. When it finds a non-empty maximal subse-quencet1, ...,tn, it removes columnst2, ...,tn from all fourtables. This removal corresponds to removing the naviga-tion events fromt1, ...,tn−1 — in effect, columnt1 replacescolumnst1, ...,tn. As part of this removal, the algorithm ap-pends any events from time sequencetn to the end of the

9

corresponding list of events in the time sequencet1. Specif-ically, it setseventsi,t1 to eventsi,t1.eventsi,tn for all i, andupdate thenecessary table to reflect the move. Note thatthere are no events in time stepst2, ...,tn−1 (other than theremoved navigation events).

When the algorithm terminates, it traverses the tablesabove to build a new trace. The removed navigation eventsdo not appear in this trace and the corresponding naviga-tion steps will therefore not take place when the trace isreplayed.

5.8 Generating a Canonical Automated Trace

By now, ACaMIA has filtered irrelevant activities andunnecessary assignments. It has also parameterized eventsthat involve state change (i.e., assign, augment, and deleteevents) and hence it knows which of them can be performedautomatically and which requires user input. Furthermore,it has removed as many as it can of the unnecessary navi-gation events. Thus, at this stage, each trace has been trans-formed to a sequence of parameterized actions and, in prin-ciple, can be used to automate the required configurationtask on a new machine.

The states of the different machines, however, may varyin ways that affect the set of events required to performthe configuration. Consider for example an update that re-quires downloading a particular software package using aWeb browser. If our automated configuration uses FireFoxbut the machine has only IE, the configuration will fail un-necessarily. More generally, different machines may havedifferent relevant states which require them to follow dif-ferent paths to perform the configuration.

To address this issue, ACaMIA parses the traces for dif-ferent execution paths, and generates an if-then automatedsolution. First, ACaMIA marks each parameterized trace bythe set of used programs (i.e., the executables). When au-tomating a configuration on a particular machine, ACaMIAstarts by taking a System Restore snapshot in case it needsto restore the machine to its previous state. ACaMIA thentries to execute the shortest parameterized trace that in-volves only programs that the machine already has (unlessthe program is installed as part of the configuration). Sec-ond, if the automated execution fails to complete, i.e., atsome point ACaMIA fails to find the view and object re-quired for the next event, ACaMIA tries a different auto-mated trace. Again, ACaMIA picks a trace that does notrequire any program that the machine does not have butalso a trace that does not have the view where the previousautomation failed. ACaMIA tries a few traces. If the au-tomation fails again, ACaMIA probes the user for whethershe wants to restore her old system state or continue theincomplete configuration manually.

6 L IMITATIONS

The current design of ACaMIA has rather limited sup-port for tasks that require the user to navigate the file sys-tem. Specifically, ACaMIA uses the recorded traces to dis-cover how to navigate among views. The underlying as-sumption is that there is some commonality in the viewsinvolved in a particular task across different traces, andhence we can learn a valid navigation path by observingonly a few traces. This assumption does not apply to nav-igation through a user’s file system, however, because dif-ferent users may have completely different directory struc-tures and file names. For example, consider a task that in-volves installing a new software package. Different usersmay download the package, navigate their file browser to arandom directory, and store the package there. A machinethat later runs the automated configuration produced fromthis trace may have none of the directories used in the trace.Thus, there is no way to navigate on the file system of anew machine in a manner similar to any of the recordedtraces. Currently, ACaMIA deals with this problem by re-quiring users to download all software packages requiredfor configuration updates into a specificDownload direc-tory. Note that this limitation is only for file navigation donedirectly by the user, and does not include file storage per-formed directly by a program, which is done in fixed lo-cations. None-the-less, it would be beneficial to extend themodel and implementation to have more flexible supportfor the file system.

ACaMIA’s design is also limited to applications thathave a reasonable support of the accessibility interface. Ourexperience indicates that all major Microsoft applications(Windows, MS Office, IE, Visual Studio , etc.) as well asmost other popular applications (e.g., FireFox, Thunder-bird, Matlab, Open Office, Adobe Acrobat) have sufficientaccessibility support to enable a tool like ACaMIA.1 TheU.S. government has taken a strong stance on software ac-cessibility, ensuring that all products which are sold to themprovide at least a minimum level of accessibility support.For this reason and others, both commercial vendors andthe open software community have made a commitment tosupport accessibility [11]. Lastly, note that smaller vendorswho do not directly support Accessibility are often buildingtheir applications, or at least their configuration interfaces,directly using widgets from Windows’s standard toolkit.Since the standard widget toolkit comes with basic Acces-sibility support built-in, this provides sufficient accesstoconfigure most of these types of applications as well.

A second limitation involves out-of-band channels. Forexample, consider the scenario where a user wants to in-stall a software package but cannot complete the installa-

1Some of these applications would benefit from more robust implementa-tions of certain features, but they all provide the main accessibility func-tionalities. For example, Acrobat does not name its UI objects, so we fallback on identifying a UI object by its role and a hash of its neighbors inthe view.

10

tion without some information that she receives via email.The likelihood of automation success depends on the com-plexity of extracting the needed information from the email.For example, if all users receive the exact same email (e.g.,there is one license for the whole site), ACaMIA will rec-ognize this relevant state as auto-enter and copy it from thetraces. If the email always has the same structure exceptfor a user name and password, then, in principle, ACaMIAmay be extended to take file diffs and handle this scenariobut our current implementation does not handle it. As theemail structure becomes more customized it will be harderto automate the interaction between the installer and theemail program.

We also assume that a user can identify a success-ful configuration and uploads only successful configurationrecordings. If a user uploads an unsuccessful configuration,which the server uses in generating an automated config-uration, the resulting automation may fail. Note that thisrisk is similar to the risk a user takes whenever they seeka solution to their configuration problems either on-line ordirectly from others. As an automated system, however, wecan take advantage of Microsoft’s built in System Restorestate restoration mechanism and create a system snapshotbefore applying a ACaMIA solution, then restoring the sys-tem to that state if the user declares the configuration unsuc-cessful.

Finally, sharing configurations with other users raisessecurity and privacy issues. ACaMIA’s current design tar-gets a single organization or an enterprise network, whereusers know and trust each other. ACaMIA accepts config-uration traces only from users with valid passwords andprovides automation services to a user after validating herpassword. Also, ACaMIA can be easily augmented to al-low the user to choose a subset of the recorded traces toproduce the automated configuration. This enables a con-sumer of an automated configuration to choose only tracesfrom people she trusts. Finally, since the UI obscures allpasswords, ACaMIA naturally inherits this property.

7 EVALUATION

We evaluate ACaMIA on actual configuration tasks thathave been documented by the system administrators in thecomputer science lab in our university. We compare itsperformance to a human user population that executes thesame tasks using the online documentation.

(a) Tested Configurations.Fig. 3 describes the tested con-figurations. The description in the figure is an anonymizedversion of the online documentation provided by our labinfrastructure group to help CS students, professors andstaff members in performing these configurations. As canbe seen from the figure these are real-world configurationsproblems. Each of them involves multiple steps and re-quires interaction with multiple applications.

(b) User Population.Our evaluation employs 12 human

users, all of which are graduate students in computer sci-ence. The users have relatively-diverse backgrounds whichspan: systems, artificial intelligence, programming lan-guages, and computer networks. They are the natural au-dience for the online documentation in Fig. 3.

(c) Setup.We performed the experiments in two sessions.Before each session, we took a system restore point on eachmachine to ensure that we can restore the machine to itsoriginal state. Once the session ends we collect the resultingtraces as well as some statistics, then restore the machine toits original state. In the first session, the users were askedto perform the configurations manually. They were directedto the online documentation for each task and given unlim-ited time to finish the configurations (but less than one day).The users performed these configurations on their own lap-tops. All used laptops run Windows XP but are otherwisecustomized by their owners. While the users performed theconfiguration manually, ACaMIA recorded their UI interac-tions. ACaMIA processed the recorded traces offline as de-scribed in§4. It generated a canonicalized automated con-figuration for each of the tested tasks. In the second session,the same users ran the automated configurations providinginputs only when ACaMIA probes them.

One limitation of our experimental setup is that themanual configuration session happens before the ACaMIAsession. First, we need a few manual traces to generate anautomated configuration, and hence it would not be feasi-ble to reverse the session order. Second, we believe thatthe order has a very limited impact on the performance ofACaMIA since once a configuration is automated the hu-man users have little to no interaction with the execution.

7.1 Configuration Successes vs. Failures

Our first result reports the number of users who failed –or succeeded – in completing each of the five configurationtasks. On the one hand, this allows the reader to gain insightto the relative difficulty of the tested configurations tasks.On the other hand, it shows how a tool like ACaMIA canreduce the need for administrator intervention.

Table 4 reports which user completed which task. Thetable shows that the first three tasks (IMAP configuration,double-sided printing, and joining the active directory) areharder from a user perspective than the last two. The IMAPconfiguration in particular, included a large number of stepsand as a result users tended to make mistakes and go backto correct them. Some users lost track and could not fin-ish the configuration. Enabling double-sided printing wasfairly bimodal; about half the users finished the task quicklywith no unnecessary actions, while others took a long timeand could not complete the configuration successfully. Acommon problem has been that some users, though giventhe documentation, did not follow it because they felt thatthey know how to perform certain tasks. ACaMIA on theother hand, has successfully automated all five configura-

11

Enabling Double-sided Printing for Windows

• Depending on the printer model , go to Start> Settings> Printer and Faxes> select the printer > Properties> configuration . Check “Duplex”

and “MBF”.

• OR go to Device settings, and under “Duplex Unit” pick Installed.

• Once you enable the duplex option , the usual "flip on long edge" or "flip on short edge" option shows up in the printer properties.

(a) Double-sided printing

Join Your Computer to the Active Directory Domain

This page describes the process of connecting an existing Windows XP system to the FooLab Active Directory domain.

Prerequisites

• Install Kerberos as described in Installing Kerberos For Windows

• Install OpenAFS as described in Installing OpenAFS

• Download and run the FooLab Kerberos REG file. This will create entries for the FooLab Kerberos realm in the Windows Registry.

Joining the domain

• Open the My Computer properties window (Right click My Computer and select Properties from the menu)

• Select the Computer Name tab and click Change.

• click Domain and type "ad.FooLab.FooUniversity.edu".

• Click more. Type "csail.mit.edu" as primary DNS suffix. Check box "Change Primary DNS suffix when Domain membership changes“

• Press OK. OK again. You will be asked for a Username and Password for Active directory. Type the username "adjoin" and the password

"adjoin". Once you enter the password , you will get a dialog box saying "Welcome to ad.FooLab.FooUniversity.edu domain." Restart the

machine.

• You should now be able to login using your FooLab Kerberos account and password and using FooLab.FooUniveristy.EDU as domain name.

• Download and Install Windows XP support tools (Optional) for command line tools such as KLIST, KSETUP etc.

(b) Joining the active directory

FireFox Certificate Problem

Issue

When navigating personal-certificate-protected FooLab websites, Firefox stops repeatedly, asking you to manually "choose a certificate to use

as identification."

Cause

The Firefox developers changed the default certificate setting from "select automatically" to "ask every time" because some countries issue

certificates to all of their citizens, and this would allow people in those countries to be tracked, by name and ID number, without their consent.

Both the impact (only your FooLab email address and full name are stored in your certificate) and probability (an attacker would have to

specifically target and request FooLab certificates, a rather small population) of this information disclosure vulnerability are quite limited for

FooLab users.

Resolution

To restore the previous Firefox behavior, open the Firefox preferences dialog (depending on your OS, this is Tools/Options, Edit/Preferences, or

Firefox/Preferences) and select the "Advanced" pane, then the "Encryption" tab then change the setting back to "Select automatically".

(c) Fix FireFox Certificate Problem

Install VirusScan Detector

• Install VirusScan Enterprise from: http://web.FooUniveristy.edu/software/

• When asked if you should update now, do update now.

(d) Install VirusScan

(e) IMAP configuration; documentation is in Figure 1

Figure 3—Online Documentation for Tested Configuration Tasks.The figure shows the online documentation provided by the infrastructure groupin the CS lab in our department. The text is anonymized and figures are removed for anonymization and space reasons. Note that the documentation forthe fifth task (IMAP configuration) is already given in Figure1.

ACaMIA User

1

User

2

User

3

User

4

User

5

User

6

User

7

User

8

User

9

User

10

User

11

User

12

IMAP Configuration yes yes yes yes yes no no yes yes yes no yes yes

Double-Sided Printing yes yes no yes no no yes yes no no no yes yes

Active Directory yes yes yes yes no yes no yes yes yes yes yes yes

Virus Scan yes yes yes yes yes yes yes yes yes yes yes yes yes

Fix FireFox Certificate

Problem

yes yes yes yes yes yes yes yes yes yes yes yes yes

Figure 4—Configuration Successes and Failures.Tables shows that multiple users failed to perform some configuration tasks despite using onlinedocumentation. All users succeeded with ACaMIA.

tions on all user machines, as shown in the table. This re-veals that ACaMIA can significantly improve users’ abilityto complete configuration tasks without administrator inter-vention.

7.2 ACaMIA’s Impact on Reducing User Inputs

The success of an automation tool is perhaps best mea-sured in terms of how much it reduces the need for inputsfrom a human user. Thus, for each configuration task andfor each user, we compute the number of human actionsperformed during the configuration. An action is any input

12

Figure 5—ACaMIA’s impact on reducing user actions. The figureshows the average number of user inputs per task, both for manual con-figuration and a ACaMIA automated configuration. ACaMIA automatedconfiguration probes the user only when it needs private information suchas a password, a username, or an email account. ACaMIA.

the user provides to a UI object. Examples include typing ina textbox, checking a checkbox or a radio button, clickingon a menu item, etc.

Fig. 5 shows a bar graph of the average number of hu-man actions per task, both during manual configuration andwhile running ACaMIA’s automated configuration. The av-erage is taken over all 12 users, and the thin lines showthe minimum and maximum across users. The figure showsthat ACaMIA significantly reduces the need for human in-volvement in the tested configuration. Specifically, in themanual mode, on average, the human users had to performbetween 8–40 actions per task. ACaMIA reduced this num-ber to 0–7 actions.

ACaMIA gets this gain by requiring human interven-tion only when the task needs user-specific inputs, such asname, password, or account information. Note that IMAPconfiguration requires the user to enter her name, email ac-count, and password. ACaMIA requires 7 user inputs be-cause the email program asks the user to enter her name andher account three times. The current version of ACaMIAcannot tell that these are all the same, and hence the userneed not be probed multiple times. However, even withoutthis additional optimization, ACaMIA has significantly de-creased the number of user actions by as much as 4×. Theonly task where ACaMIA requires human intervention fora global input is active directory, which requires a passwordthat is shared among all users. However, since most GUIsidentify private information such as passwords, ACaMIAexplicitly does not automate the entry of this private input.The other three tasks do not require any user private in-formation, and ACaMIA has performed them automaticallywithout any user input.

7.3 ACaMIA’s Impact on Reducing ConfigurationTime

Another benefit of ACaMIA is its ability to reduce theoverall time required to complete a configuration, whichadds to better user experience. Fig. 6 shows the averagetime consumed by each configuration task, both for the caseof manual configuration and ACaMIA’s automated config-

Figure 6—Configuration time with and without ACaMIA. The figureshows the average amount of time per configuration task as well as themin and max values. It reveals that ACaMIA can improve user experienceby speeding up configuration updates.

Figure 7—ACaMIA is effective at filtering unnecessary actions.Thefigure plots the sum of automated and user actions average across users,both for ACaMIA and manual configuration. It shows that despite be-ing automated, ACaMIA performs a configuration task using significantlyfewer actions than the average human user. ACaMIA.

uration. The average is taken over all users. The thin barsshow the minimum and maximum values. The figure re-veals that ACaMIA is effective at reducing the overall con-figuration time, particularly for complex tasks that tend totake longer time. We note that ACaMIA could not reducethe time needed for installing the VirusScan because mostof this time is spent on the download itself.

7.4 How Effective is ACaMIA at Filtering IrrelevantActions?

ACaMIA’s design tries to filter out any unnecessary ac-tions/events that do not help in performing the desired con-figuration task. Here we check that our attempt to removesuch actions is successful. Fig. 7 shows the average num-ber of actions involved in performing each of the configu-ration tasks. The figure also shows the maximum and min-imum number of actions across all users. These are all ac-tions, i.e., both automated and user performed. The figureshows that despite being completely automated, ACaMIAperforms significantly fewer actions than the average hu-man user. In fact, in most tasks, ACaMIA performs feweractions than the minimum across all human users. Thisshows that our filtering algorithms are effective at removingunnecessary events/actions without affecting correct behav-ior.

13

Machine Type Used by ACaMIA’s Automation

IE only Used IEhline FireFoxonly

Used FireFox

Broken IE, func-tional FireFox

Tried IE, Failed to complete, movedto FireFox

Figure 8—ACaMIA customizes an automated configuration depend-ing on the state of the machine on which it executes the configuration.

7.5 ACaMIA’s Conditional Execution

As described earlier, the state of the different machinesmay vary in ways that affect the set of events required toperform a particular configuration. ACaMIA deals with thisissue with an if-then execution style. In this section, wewant to check that ACaMIA can customize the automatedconfiguration according to the state of the machine. To ex-periment with this issue, we run ACaMIA’s automated in-stallation of VirusScan on three types of machines: 1) a ma-chine that has IE uninstalled and FireFox installed,2 2) amachine that has only IE, and 3) a machine that has a bro-ken version of IE and a functional version of FireFox. Notethat installing VirusScan requires a Web browser. Thus, wewould like ACaMIA to use the working browser on eachmachine.

Table 8 shows the results of running ACaMIA’s auto-mated configuration on these three machine types. As ex-pected, ACaMIA checked for the required programs anddiscovered that it needs to run an automation trace thatinvolves IE on the IE-only machine, and FireFox on theFireFox-only machine. ACaMIA did not know that theIE version on the third machine is broken. Thus, it triedto automate the installation using IE. Since IE is broken,ACaMIA noticed that it could not find the next view in itstrace events. It waited a bit, then switched to an automationoption that does not have the failed IE-view. Since the ma-chine has FireFox, it picked an automation option that usesFireFox to download VirusScan. This caused the configu-ration to succeed.

8 CONCLUSION

ACaMIA is a tool for automating desktop setup andconfiguration. It requires no kernel modifications and worksacross various applications. Our experimental results showthat ACaMIA is effective in automating complex tasks thatmany users fail to complete even in the presence of on-line documentation. Further, ACaMIA achieves this perfor-mance without requiring the administrator to write any au-tomation scripts. It simply watches a few users as they in-teract with the GUI to perform the desired task. ACaMIAcan then generalize the users’ actions and play them backto the GUI to automate the desired task on a new machine.Thus, ACaMIA advances the-state-of-the-art in how orga-nizations deal with desktop configuration updates. It can2We made Windows uninstall IE so that it is not in the program list.

save an organization from spending an excessive amount ofresources on administrative support and reduce user frus-tration.

REFERENCES

[1] AutoHotkey, a Free Mouse and Keyboard Macro Program.http://www.autohotkey.com/.

[2] AutoIt, a freeware Windows automation language.http://www.autoitscript.com/.

[3] iMacros for FireFox.https://addons.mozilla.org/.[4] Microsoft Active Accessibility.

http://msdn.microsoft.com/.[5] Microsoft Active Accessibility.

http://en.wikipedia.org/.[6] B. Aggarwal, R. Bhagwan, T. Das, S. Eswaran, V. N. Padmanabhan,

and G. M. Voelker. Netprints: Diagnosing home networkmisconfigurations using shared knowledge. InUsenix NSDI, 2009.

[7] L. D. Bergman, V. Castelli, T. A. Lau, and D. Oblinger.Docwizards: a system for authoring follow-me documentationwizards. InUIST, 2005.

[8] M. Bolin, M. Webber, P. Rha, T. Wilson, and R. Miller. Automationand customization of rendered web pages. InUIST ’05:Proceedings of the 18th annual ACM symposium on User interfacesoftware and technology, 2005.

[9] O. Crameri, N. Knezevic, D. Kostic, R. Bianchini, andW. Zwaenepoel. Staged deployment in mirage, an integratedsoftware upgrade testing and distribution system. InSOSP, 2007.

[10] A. Cypher.Watch What I Do: Programming by Demonstration.MIT Press, 1993.

[11] T. L. Foundation. IAcessibility2.http://www.linuxfoundation.org/.

[12] D. A. Holland, W. Josephson, K. Magoutis, M. I. Seltzer,C. A.Stein, and A. Lim. Research issues in no-futz computing.Workshopon Hot Topics in Operating Systems, 2001.

[13] G. Little, T. Lau, A. Cypher, J. Lin, E. Haber, and E. Kandogan.Koala: Capture, share, automate, personalize business processes onthe web. InCHI’07, 2007.

[14] N. McFarlane. Fixing web sites with greasemonkey. InLinux J.2005.

[15] Y. Su, M. Attariyan, and J. Flinn. Autobash: improvingconfiguration management with operating system causalityanalysis.ACM SOSP, 41(6), 2007.

[16] H. J. Wang, J. C. Platt, Y. Chen, R. Zhang, and Y.-M. Wang.Automatic misconfiguration troubleshooting with peerpressure. InOSDI: Proceedings of the 6th symposium on operating systemsdesign and implementation, Berkeley, CA, USA, 2004. USENIXAssocation.

[17] Y.-M. Wang, C. Verbowski, J. Dunagan, Y. Chen, H. J. Wang,C. Yuan, and Z. Zhang. Strider: A black-box, state-based approachto change and configuration management and support. InLISA:Proceedings of the 17th Large Installation System AdministrationConference, Berkeley, CA, USA, 2003. USENIX Assocation.

[18] A. Whitaker, R. S. Cox, and S. Gribble. Configuration debugging assearch: Finding the needle in the haystack. InOSDI: Proceedings ofthe 6th symposium on operating systems design andimplementation, Berkeley, CA, USA, 2004. USENIX Assocation.

[19] C. Yuan, N. Lao, J. rong Wen, J. Li, Z. Zhang, Y. min Wang, andW. ying Ma. Automated known problem diagnosis with eventtraces. InIn EuroSys, 2006.

14