Ontology-based Application Integration || An Ontology of User Interfaces and Interactions

Download Ontology-based Application Integration || An Ontology of User Interfaces and Interactions

Post on 09-Dec-2016




2 download

Embed Size (px)


<ul><li><p>Chapter 6An Ontology of User Interfaces and Interactions</p><p>Abstract This chapter describes the development of a ontology of the user interfacesand interactions domain used for UI integration. As foreseen by most ontologyengineering approaches (e.g., Uschold and Gruninger 1996, p. 15, and Fernandez,Gomez-Perez, and Juristo 1997, p. 37), building the ontology was started by collectingthe relevant concepts from the domain. To this end, various UI description languages(UIDLs) have been analyzed. Section 6.1 shows a survey of the languages inspected.Based on the concepts identied in this survey, the domain ontology has been built.Section 6.2 describes this ontology with its basic concepts and design patterns.</p><p>6.1 Existing User Interface Description Languages</p><p>Various languages for describing user interfaces exist. They have been developedwith different purposes: for creating more modular and device-independent, portableUIs, for communicating requirements between developers and other stakeholders(Guerrero-Garcia et al, 2009, p. 36), to facilitate rapid development, or to providea compressed form for transferring the interface to the client in case of limitedbandwidth (Abrams et al, 1999, p. 1700).</p><p>This section reviews a selection of those languages, identies their key concepts,and depicts modeling techniques. The results of this section are then used to build aconcise and complete domain ontology of user interfaces and interactions.</p><p>Surveys on UI description languages haven been given by Souchon and Vander-donckt (2003, pp. 377), (Paterno` et al, 2008b, pp. 23) and Guerrero-Garcia et al (2009,pp. 36). Together, they list 20 of such languages. The selection made for this sectionhas been made based on popularity in the literature, relevance with respect to themodeling goals of the ontology, availability of detailed documentation (as the exactset of tags or elements is needed for building the vocabulary), and expressiveness.</p><p>To structure the languages analyzed, we follow the Cameleon reference framework,which foresees four levels of models (Calvary et al, 2002, pp. 22):</p><p>, DOI 10.1007/978-1-4 - - _ , Springer Science+Business Media, LLC 2011 </p><p>, H. Paulheim Ontology-based Application Integration614 1430 8</p><p>1196</p></li><li><p>120 6 An Ontology of User Interfaces and Interactions</p><p> The concepts and tasks model denes the domain objects that can be manipulatedwith a user interface (e.g. enter a persons weight), and the tasks which can beperformed by a user of that interface.</p><p> The abstract user interface (AUI) model denes the UI components that areforeseen on an abstract level, without targeting at specic widget types (e.g., acomponent to enter a numerical value). The AUI is independent of a specicmodality (Paterno` et al, 2008a, p. 238), such as speech or mouse-and-keyboardinteraction.</p><p> The concrete user interface (CUI) model denes the concrete widgets that fullthe task, e.g. a slider widget or a validated text input eld. The CUI relies ona specic modality (Paterno` et al, 2008a, p. 239).</p><p> The nal user interface is the runnable user interface, which can be directlyexecuted on a computer, either as binary code or in form of an interpreted language,such as HTML.</p><p>Figure 6.1 shows the languages examined in this section, classied in theCameleon reference framework. Since the focus is on the expressivity of the re-spective languages, the nal user interface has been merged with the concrete userinterface, since the fact whether a specication is directly executable or not does notnecessarily interfere with the expressivity of the underlying language.</p><p>Concrete/final UIAbstract UIConcepts and Tasks</p><p>high abstraction,platform independence</p><p>low abstraction,platform dependence</p><p>UsiXML</p><p>XIML Xforms and HTML</p><p>XUL</p><p>LZX</p><p>WAI ARIA</p><p>UIML VocabulariesUIML</p><p>TeresaXML (AUI)TeresaXML (CTT)</p><p>MARIA XML</p><p>Fig. 6.1: Overview of examined UI description languages</p><p>6.1.1 HTML and XForms</p><p>HTML is in the rst place not a language for dening user interfaces, but for textmarkup, i.e., for dening structural blocks in a text and hyperlinks between different</p></li><li><p>6.1 Existing User Interface Description Languages 121</p><p>texts. In its original form, which dates back to the early 1990s, HTML could be usedto dene paragraphs, headlines, lists, images in text, and hyperlinks (Berners-Leeand Connolly, 1993).</p><p>The original HTML specication has changed in two directions: rst, more ne-grained capabilities for controlling the layout of web pages, have emerged, whichculminated in the specication of Cascading Style Sheets (CSS) (W3C, 2010a), alayout denition language. Second, with the advent of more complex and interactiveweb-based applications, reasonable extensions towards describing interactions withweb-based applications gained focus in the HTML specication. The currentlyproposed version of HTML, the working draft of HTML5 (W3C, 2011b), foresees aset of language means for describing interactions.</p><p>There are different components for dening forms in HTML, which are one ofthe main mechanisms for creating interactive applications. The central elements ofHTML forms are:</p><p>input for dening text input elds. There are different content types that can bedened for the input eld, such as date or telephone number, as well as rangesfor numerical values. Behavioral characteristics such as autocomplete supportmay also be specied. Input elds for texts with line breaks are dened withtextarea.</p><p>select and other elements are used to dene input elds for selecting from apredened list of values.</p><p>button is used to dene buttons. There are special buttons for submitting andresetting forms, as well as buttons with user dened behavior. For reasons ofbackward compatibility, buttons may also be dened with input.</p><p>Other elements are used to output data, either in textual or in graphical form.There are further elements for specifying interaction, such as the provision of asummary/detail view of the same data, or menus. Besides form elements, HTMLdenes different types of events that can occur when interacting with a web-basedapplication. Those events encompass</p><p>User actions such as clicking, dragging and dropping, scrolling, hitting the key-board, etc.</p><p>Browser-related events such as going on line or off line, changing the displayedpage, printing the page, etc.</p><p>Media events occurring when showing embedded media, such as playing, pausing,or resuming the media.</p><p>Generic events such as undo and redo.</p><p>In total, HTML5 denes 14 different types of (interactive) UI components, and71 different events that can be raised by those components. It provides only littlemeans to specify the actual behavior, i.e., what is supposed to happen when the userpresses a button or submits a form. Furthermore, there is only limited support fordescribing the state of components, they can only be dened as hidden or visible,and input elements may additionally be dened as enabled or disabled.</p></li><li><p>122 6 An Ontology of User Interfaces and Interactions</p><p>With the advent of web-based formats other than HTML, such as WML for mobileapplications (Wireless Application Protocol Forum, Ltd., 2001) or VoiceXML fordialogue-based applications (W3C, 2010c), there has been an effort to factor outsome of the form denition part into an abstract, device-independent format of itsown, called XForms (W3C, 2009e). It allows for dening input elds with valuerestrictions, as well as processing logic of those forms. XForms has eight differenttypes of (interactive) UI components, and 43 different event types. For developingthe ontology, we have concentrated on XForms, which provides a more conceptualview on form based applications than the hypertext-centric HTML does.</p><p>6.1.2 XIML</p><p>XIML (Puerta and Eisenstein, 2002, pp. 214) provides very high-level means fordening user interfaces. Predened abstract components in XIML are (Puerta andEisenstein, 2001, pp. 3):</p><p>Task allows for the denition of user tasks, composed of partial tasks, and theirexecution order and conditions.</p><p>Domain allows for dening the objects that the user can view and/or manipulatein the user interface.</p><p>User allows for dening different user roles.Presentation allows for dening the layout of a user interface.Dialog denes the behaviour of the user interface.</p><p>Various links can be made between those different models, stating which usersperform which tasks, which domain objects are displayed by which presentationelements, and so on (Vanderdonckt, 2000, p. 10).</p><p>As XIML is dened at a very abstract level, there are (intentionally) no concrete UIelements, events, user roles, etc. dened. The developer has to dene those elementsall by himself, which one the hand grants a high exibility, but on the other handlimits the use of the language in terms of development efforts and compatibility ofdifferent XIML models (Vanderdonckt, 2000, p. 14). Furthermore, XIML providesnumerous extension points which allow the developer to create their own custommodels, classes, relations, etc.</p><p>As the last update to XIML dates back to 2004, it is not clear whether XIML willbe developed any further in the future.</p><p>6.1.3 XUL</p><p>XUL (Mozilla, 2011b) is an XML-based user interface description language. It isinterpreted by Mozillas Gecko Engine (Mozilla, 2011a). The user interfaces of</p></li><li><p>6.1 Existing User Interface Description Languages 123</p><p>popular applications like Thunderbird and Firefox are dened in XUL (Feldt, 2007,p. 77).</p><p>XUL denes different basic elements, e.g., window types and interactive widgets,general ones such as text input elds as well as specic ones such as date or colorpickers, summing up to 83 element types. Furthermore, there are elements forcontrolling the layout, templates for showing data from data sources, and means toadd behavior with JavaScript. In addition, XUL denes 44 event types.</p><p>Other features of XUL include the XML Binding Language XBL for deningcustom widget types, a mechanism called overlays which is used for customizationand manipulation of existing XUL UIs (similar to the restructure technique in UIML),a mechanism for invoking components written in other programming languages, andthe possibility to create automatic installers for XUL based applications.</p><p>6.1.4 TeresaXML and MARIA XML</p><p>TeresaXML is a set of two XML schemas: one for describing tasks, and one fordescribing abstract user interfaces. Together with an authoring platform and a setof transformation mechanisms, they form the Teresa platform (Paterno` et al, 2008a,pp. 235). For dening tasks, the ConcurTaskTrees (CTT) method is used (Paterno`et al, 1997, pp. 362).</p><p>The Teresa toolkit provides editors for both the concepts and tasks models and theAUI model (see gure 6.2). Generators can transform from concepts and tasks to abasic AUI denition, as well as from an AUI to various CUI models, such as desktopapplications and speech based UIs.</p><p>The AUI language of Teresa facilitates the denition of user interfaces as aset of presentation elements, their visual arrangement (called composition) andtheir interactive arrangement (called connection). A small set of basic abstractpresentation elements is given, such as single and multiple choice inputs, or textoutputs.</p><p>The successor of Teresa XML is MARIA (Model-based lAnguage foR InteractiveApplications), which provides a set of enhancements and improvements over TeresaXML, which encompass the possibility to dene a data and an event model, dynamicmanipulations of UI elements, and continuously updated elements. There are alsotools for reverse engineering existing user interfaces, e.g., HTML pages, to MARIAdescriptions (Bandelloni et al, 2008, pp. 285).</p><p>6.1.5 LZX</p><p>LZX is an XML based language used in the open source platform OpenLaszlo, whichprovides a framework for building rich client applications on the web, which can beexecuted as DHTML (dynamic HTML, i.e., HTML with AJAX) and JavaScript as</p></li><li><p>124 6 An Ontology of User Interfaces and Interactions</p><p>Fig. 6.2: A screenshot of the Teresa toolkit</p><p>well as a Flash application (Laszlo Systems, 2006). User interfaces can be written inthe declarative LZX language, which is then compiled to DHTML or Flash.</p><p>LZX has 54 types of basic components, starting from simple drawing elementsup to interactive elements such as buttons or input elds. Furthermore, there arespecialized components, e.g., for charting data, showing media, getting pictures froma users camera, etc.</p><p>LZX also supports dening behavior with JavaScript, based on events. Thereare 15 different predened event types to capture events from the mouse and thekeyboard, as well as other events, e.g., reacting to interactions with the browser theOpenLaszlo application runs in, or to predened timers. Furthermore, LZX has abasic notion of states an object can have, although the only predened states are fordening that an object is being dragged or being resized.</p><p>Languages similar to LZX are MXML, the language used by Adobe Flex (AdobeSystems Inc., 2011), and XAML developed by Microsoft (Microsoft Corporation,2011c).</p></li><li><p>6.1 Existing User Interface Description Languages 125</p><p>6.1.6 WAI ARIA</p><p>WAI ARIA (see also Sect. 4.2.3) is an initiative by the W3C for making web applica-tions more accessible (W3C, 2011a). It denes a set of roles that a UI element cantake. Those annotations can be embedded, e.g., in HTML pages, and may be used toassist users with cognitive disabilities to use web applications by providing additionalassistance, e.g., speech output or visual magnication. Those assistive technologieshave to rely on some abstract representation of the user interface components. WAIARIA is also available as an OWL ontology.</p><p>Consequently, WAI ARIA denes a hierarchy of UI elements, together with a setof properties and state information. The set of roles is divided into eleven abstractand 60 concrete roles. The class model of WAI ARIA contains some multipleinheritance relations and mixes concerns in some places, such as presentation formand contents. For example, the section class contains sibling subclasses marqueeand definition where one denes a form (i.e., text ying from side to side),while the other denes contents. Furthermore, it is does not follow a rigid ontologyengineering approach, but contains some questionable subclass relations. The toplevel consists of the three categories WINDOW, WIDGET, and STRUCTURE, but thereare many categories which are sub categories of more than one of those. For example,ALERT DIALOG is at the same time a sub category of WINDOW and of REGION (asub category of STRUCTURE), where the latter is explained to be a large perceivablesection of a web page or document, while the former follows the notion of beinga browser displaying such web pages and documents. The WAI ARIA ontology isnot general, but has a strong bias towards web based user interfaces, which makesit difcult to transfer it to other types of user interfaces. The hardware parts of userinterfaces are not contained in WAI ARIA, neither are user interactions.</p><p>Furthermore, WAI ARIA denes a set of different states and properties a UIcomponent can have, most of which are orthogonal. Although states and propertiesare n...</p></li></ul>


View more >