model-driven engineering of user interfaces: promises, successes, failures, and challenges

1 Keynote address – ROCHI’2008, Iasi, 17-18 September 2008

Model-Driven Engineering of User Interfaces:Promises, Successes, Failures, and Challenges

Jean Vanderdonckt

Université catholique de Louvain (UCL)Louvain School of Management (LSM)

Information Systems Unit (ISYS)Belgian Laboratory of Computer-Human Interaction (BCHI)

http://www.isys.ucl.ac.be/bchiPlace des Doyens, 1 – B-1348 Louvain-la-Neuve (Belgium)


Talk’s outline

• What is model-based UI design?– What is a UI model?– What do we need to get a quality UI model?– How many models do we need to characterize a UI?

• From model-based UI design to model-driven UI engineering– Three dimensions

• What are the models? Which language?• What is the methodological approach to manipulate them?• What is the tool support?

– Promises– Successes vs failures– Challenges

• What can we do today together?


What is the today’s situation regarding user interfaces?

• Technological aspects of user interfaces progress significantly faster than– Software engineering aspects

• It takes time to develop a user interface with a new device, a new interaction technique

• It takes more time to develop a toolkit• It takes even more time to rely on a model-driven approach

– Usability engineering aspects• New user interfaces are shipped with usability problems

because– Little or no experience– No past, no empirical evidence

• Empirical experiments require a lot of resources if done carefully

[Dragicevic et al.,2004]

http://www.dgp.toronto.edu/~dragice/foldndrop/foldndrop_UIST04.pdf


What is model-based UI design?

• Classical approch to developing UIs with a UI builder (IDE)– Describe it, prototype it, code it with trial and errors, repeat



• UI builders offer only partial solution

Desired Interface UI Builder Solution

Window Menu bar Palette (icons) Scrollbars Other widgets: drop-down list, etc.

Table with dynamic data Gantt chart Direct manipulation of widgets



• UI builders cannot produce their own UI (no bootstrapping)



• Classical approch to developing UIs– Advantages

• UI is graphical by nature • A UI can be

– Rapidly prototyped– Easily modified– Visually demonstrated

– Shortcomings• No structured method for layout and programming • Selection of widget can be inappropriate• Layout of widget can be tedious, repetitive • Problem of the spaghetti of callbacks• Any UI modification can lead to unstructured design



• Model-based UI design consists of– Describing a UI in a UI model– Relying on this UI model to drive the UI development lifecycle– While pursuing the following goals:

• To provide comprehensive UI development environments• To deliver robust development lifecycle support• To improve portability• To integrate usability with UI development

– A structure in 3 dimensions

Models

UsiXML Language

Step-wise method

Software toolssupport

involves

described in

Development methodology



• What is a model?– Webster’s definitions:

• « One who is employed to display clothes or other merchandise »

• « A set of plans for a building » (good analogy)• « A system of postulates, data, and inferences presented as

mathematical description of an entity or state of affairs »– Our definition

• A structured set of plans for developing a UI• A system of postulates, data, and inferences presented as a UI

description (or specification) that drives the UI development lifecycle



• What is a model?– A simple example for understanding purposes: a system login

– Finding the right model abstractions is NOT• Replicating code in another way• Capturing all physical properties

System loginSystem login

User name:

Password:

Enter Text

*******

Log in Cancel

Use this link to recover your password

I have lost my password

Help



• What is a model?– Finding the right model abstractions IS

• Turning important aspects into UI design options– E.g. LabelAlignement instead of absolute coordinates

• Deriving final properties from these options– E.g., compute coordinates from LabelAlignement = {left,

right, aligned, centred, justified}


User name:

Password:

Enter Text

*******

Log in Cancel



Help

LabelAlignement = left

ButtonPlacement = TotalEquil



• What is a model?– Finding the right model abstractions IS

• Describing UI widgets independently from technology• Why?


User name:

Password:

Enter Text

*******

Log in Cancel



Help

Identification label

Identification label

Input textInput text

Push buttonPush button



• What is a model?– Because different evolutions

time

Platform

User

Environment

Language



• What is a model?– Because different evolutions

WML 2.0WML 1.0

XML

XHTML

1986 1989 1997 1998 20031996

HTML 4.0HTMLSGML

VoiceXML2000



• What is a model?– Because same widget appears in many different technologies

– Because many different ways to reach the same goal (e.g., selector)

Check Box(Windows)

BoxArray(Garnet)

XmToggleButton(OSF-Motif)



• What is a model?– In order to derive UIs systematically

GrafiXML




GrafiXML




Edit the UsiXML

Edit the UsiXML

Show attributes

Show attributes

A click on the tree highlights the

corresponding UsiXML

A click on the tree highlights the

corresponding UsiXML

GrafiXML



• What do we need to get a quality UI model?– Abstractions are always limited, but extensible

Plateau effect:• Do not introduce too many,

otherwise too complex (<> simple)• Do not introduce too few,

otherwise too simplistic (<> simple)– C1: Need to ensure quality properties of a model

• Completeness• Consistency• Correction• Expressiveness• …

CUI Model

CIO

graphicalCIO

graphicalContainer graphicalIndividualComponent


What do we have so far?

• With this first set of abstractions, we go from Final UI (FUI) to Concrete UI (CUI)

• Consequently, a CUI is independent from– Any language: (X)HTML, Java, Visual Basic, C++– Any computing platform: Windows, Linux, MacOS

Concrete userInterface S

Final userInterface S


And now what?

• UIs are not only graphical, but could be vocal, tactile, 3D, haptic, an multimodal

• So, there is a need for more abstraction


User name:

Password:

Enter Text

*******

Log in Cancel



Help

Abstract ContainerAbstract Container

Abstract Individual Component with Output

Abstract Individual Component with Output

Abs. Ind. Comp. with InputAbs. Ind. Comp. with Input

Abs. Ind. Comp. with ControlAbs. Ind. Comp. with Control



• With this second set of abstractions, we go from CUI to Abstract UI (AUI)

• Consequently, a AUI is independent from– Any interaction modality



Abstract userInterface S


And now what?

• UIs, even if they are abstract, are structured according to the target system as opposed to be user-centered

• So, there is a need for an abstraction that is computing independent– Task and domain models (UML Class Diagram)

System Login

Enter information Check U+P

[]>>

Enter Username

Enter Password

|||

Check existence

Verify correspondence

|||



• With this third set of abstractions, we go from AUI to Task and Domain (T+D)

• Consequently, a T+D is independent from any implementation



Task and Domain S



Do we need more models?

• Users want to determine their path on each platform

[Forrester Research,2003]


Do we need more models?Demo

FlashiXML

http://www.samsungfuns.com/Wallscreen/photos/2/2/highlighted.aspx

http://www.panoramtech.com/products/facility.html



• Multiplicity of contexts of use

Location Role Device Experience

Sporting Multimedia Travel programme

Working Travel booking sitePowerful interface for complex operations

Travelling Booking notificationEverywhere connectivity for simple data exchange

Family TV is multi-media family device #1

http://images.google.com/imgres?imgurl=http://www.idealbodiesonline.com/images/gym_pics/In%2520the%2520gym-Biceps2.jpg&imgrefurl=http://www.idealbodiesonline.com/mystory.htm&h=202&w=280&sz=10&tbnid=uRs_PFdFw1kJ:&tbnh=78&tbnw=109&hl=en&start=8&prev=/images%3Fq%3DGym%26hl%3Den%26lr%3D%26sa%3DG



• Property of plasticity = adaptation to the context of use while satisfying predefined usability properties of interest

[Grolaux et al.,2002]

DemoFlexClock

E:\My Documents\Recherche\Multi-Platform & plasticity\FlexClock\flexclock.oza

http://www.isys.ucl.ac.be/bchi/publications/2002/Grolaux-Tamodia2002.pdf


So what we have now is



Task and Domain S


S=Source context of use

User S Platform S Environment S

[Calvary et al.,2003]

In GrafiXML

http://www.isys.ucl.ac.be/bchi/publications/2003/Calvary-IwC2003.pdf


Multi-platform for Emergency

• Three platforms– Pocket PC– Desktop PC– Wall Screens


Multi-platform for Emergency

• Model and method– Design the reference screen first– Refine the others screens later

• By applying graceful degradation• By applying transformation techniques


Graceful degradation

[Florins & Vanderdonckt,2004]


Transformation rules



Add all >>

Add >

<< Remove all

< Remove

>>

>

<<

<

>

<

Group box

Item 1Item 2Item 3Item 4

Item 5Item 6Item 7Item 8

Item 1Item 2Item 3Item 4Item 5Item 6Item 7

Item 1

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

Item 7


Environment T

What do we have now

• Cameleon Reference Framework

Final userInterface T

Concrete userInterface T

Task and Domain T

Abstract userInterface T

T=Target context of use



Task and Domain S


S=Source context of use

Reification

Abstraction

Reflexion

Translation

http://www.plasticity.org

UsiXMLunsupported

model

UsiXMLsupported

model

User S Platform S Environment S Platform TUser T

[Calvary et al.,2003]


Other challenges raised

• C2: Need to cover– Semantics:

metamodel as aUML Class Diagram

– Syntax:UsiXML language

– Stylistics:

• C3: Minimum amount of models• C4: Maximum amount of models

Root

insert search criteria

view results

aux show item

details

show list items select item

insert criteria

submit

edit

edit criteria

submit

view results

show

view detailsview items

show

select


Towards model-driven engineering

• Possible definition– MDA is an OMG initiative that proposes to define a set of non-

proprietary standards that will specify interoperable technologies with which to realize model-driven development with automated transformations. Not all of these technologies will directly concern the transformation involved in MDA. MDA does not necessarily rely on the UML, but, as a specialized kind of MDD (Model Driven Development), MDA necessarily involves the use of model(s) in development, which entails that at least one modeling language must be used. Any modeling language used in MDA must be described in terms of the MOF language to enable the metadata to be understood in a standard manner, which is a precondition for any activity to perform automated transformation.



• C5: Keep decision decision as SDLC is evolving– Support for annotation-based design


Example of preference-based design

• Goal = to input the temperature of a human body– Solution n°1: edit box– Solution n°2: list box– Solution n°3: drop down list– Solution n°4: field with scroll bar– Solution n°5: thermometer


How to argue for preference?

• Use the QOC notation

Question?

Criteria 1

Criteria 2

Criteria j

Criteria m

Option 1

Option 2

Option i

Option n

= negatively affects= positively affects

[McLean & Belotti,1998]


Preference example

Problem

Solution 1

Solution 2

(RE1)

(RE2)

(A1)

(A2)

(A3)

(A4)

(A5)

(A6)

Standard compatibility (e.g. Windows)

Cognitive respect

Minimal actions

Dialog representativity

Prompting

Clarity

suggests

suggests

avoids

contradicts

contradicts

respects

= for= against

(A1) = drop down list requires less space than radio buttons(A2) = some possible values become obsolete when they are infrequently used(A3) = drop down list shows better then current value than the possible values(A4) = radio buttons are faster and easier to manipulate than drop down list(A5) = radio buttons are recommended in Microsoft Windows style guide(A6) = radio buttons do not allow users to change the values

drop down list

radio buttons

(A4) is contradicted by (A2) and (A3): the widget should be more used for output than input.(A6) is contradicted by (A3): it is better to present all possible values than only one at a time(A5) is an autority argument, and can fall in front of (A1)+(A2)+(A3)

[Vanderdonckt,1997]


Theory of argumentation

Design problem Design option parameter(parameter, value)

Is described by

Is solved by

Accepted solution Set of admissiblesolutions

Solution 1 Solution 2 Solution n

Counter-argument(rebuttal, counter-claim)

Argument(ground, typically data)

can be rejectedbecause of

Can be acceptedthanks to

Is justified byDesign claim Warrant

Is reinforced by

Qualifiersatisfied

unsatisfied

Is related to

corresponds to

Is justified by

Is qualified by

[Toulmin,1975][Perelman, 1978]



• C6: Need to support (de)composition

DemoFormiXML


Towards Model-Driven Engineering

• C7: Support for multi-path development• Different types of engineering

– Forward engineering– Reverse engineering– Lateral engineering– Diagonal engineering (with or without shortcuts)

• Different approaches– Top-down– Bottom-up– Middle-out

• Different development paths– Example: Round-trip engineering = composition of

• Reification CUI -> FUI• Reflexion FUI -> FUI• Abstraction FUI -> CUI• Reflexion CUI -> CUI


Revisitation

Task &Concepts

Abstract UI

ConcreteUI

Final UI Reformating

TranscodingRecoding

Respecification

Retasking

Restructuration

Programunderstanding

Redocumentation

Reverse Engineering

Design recovery

Reengineering

Revamping

[Bouillon,2006]

http://www.isys.ucl.ac.be/bchi/publications/Ph.D.Theses/Bouillon-PhD2006.pdf


Towards Model-Driven Engineering

• C8: Support for multi-fidelity

DemoSketchiXML



• Other challenges (C13 -> C16)– High ceiling– Low threshold– Wide walls

Ca

pa

bili

ties

Resources (time, experience,…)

100%

50%

Ceiling

Threshold

Firs

t gen

erat

ion

Sec

ond

gene

ratio

n

MD

A C

AS

E to

ols

Inte

rfac

e bu

ilder

s an

din

tegr

ated

env

ironm

ents

Resource win for applicationssupported by MDA-compliant tools

Resource win for applicationssupported by first-generation



• Other challenges (C13 -> C16)– High ceiling– Low threshold– Wide walls

Capabilities

Resources(time, experience,…)

100%

50%

Ceiling

Threshold

Firs

t gen

erat

ion

Sec

ond

gene

ratio

n

Th

ird

gen

erat

ion

Inte

grat

ed D

evel

opm

ent E

nviro

nmen

ts

UI types

Walls



• C17: Effort incrementality and C19: rendering engines– UsiXML interpreter for

• Graphical user interfaces: XHTML, XUL, Java, Flash, Tcl-Tk• Vocal user interfaces: VoiceXML• Multimodal user interfaces: X+V• Haptic user interfaces: HaptiXML

– Support for• Distributed user interfaces• Mixed-reality user interfaces


Developed by Donatien Grolaux

• Problem: how to design a UI that takes care of multiple displays?

• Solution: Migration = DEMIPLAT• DistriXML = software architecture for migrating UIs from one

platform to another at run-time

Pencil

Palette

Painting

Paintingtool

[Grolaux & Vanderdonckt,2005]

http://www.isys.ucl.ac.be/bchi/publications/2005/Grolaux-Interact2005.pdf


Demonstration

[Grolaux & Vanderdonckt,2005]


Demonstration using two displays from two different computers


Example using a Pocket PC


UI Migration: DEMIPLAT

• Detach



• Detach - Migrate



• Detach - Migrate - Plastify



• Detach - Migrate - Plastify - Attach


This is not a floating toolbar

Process


Computer B

Process

This is migration !

Process

Computer A



• C18: More dynamic aspects– Mixed reality

Minorityreport


Conclusion

• What can be really supported?

0

2000

4000

6000

8000

10000

12000

14000

16000W

ord

KB

KB

Pa

rse

r

Qu

ery

Ne

two

rk o

ps.

UI

sta

tes

Pre

sen

tatio

n

Act

ion

s

Up

da

te

Inte

ract

ion

s

Application logic User interface

Generated code

Models

[Skezely,1996]


Conclusion

• Successes:– Work for very-well defined domains– Really fosters collaboration– Supports flexibility, maintainability (Belgium)

• Failures– Does not work for very complex, dynamic UIs– Takes a lot of resources for

• Models (some)• Method (more)• Tool (even more)

Support for mnemonics and

shortcuts

Support for mnemonics and

shortcuts


How to participate?

• Join the UsiXML Consortium today!– Anybody can be a member: observer, participant, developer– Send an e-mail to [email protected] and/or « Register » on the

web site: for this purpose, provide• Your firstname, last name, affiliation, desired level of

membership, and motivations– Think of your own research in terms of models, e.g.,

• Instead of evaluating a Graphical UI for a certain platform, conduct its evaluation on the corresponding CUI model: it is more general, you will gain more audience

• Instead of developing a separate tool, make it UsiXML-compliant: you will gain more audience, the results are more largely applicable

• Output– UsiXML session at conferences– UsiXML CD-ROM, etc.


How to participate?

• Develop web services for UsiXML• Open positions in UsiXML project (from April 2009)

– PhD students– Internships for MSc students


Thank you very much for your attention

For more information and downloading,http://www.isys.ucl.ac.be/bchi

http://www.usixml.orgUser Interface eXtensible Markup Language

http://www.similar.ccEuropean network on Multimodal UIs

http://www.isys.ucl.ac.be/bchi

http://www.usixml.org/

http://www.similar.cc/

model-driven engineering of user interfaces: promises, successes, failures, and challenges

Technology

based ui design

derive uis

driven engineeringother

quality ui

user interfaces

ui migration

driven engineering

transformation