1invited talk for ruipiacs’2006 workshop – geneva, switzerland, september 13 rd, 2006...

1 Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006

SketchiXML:an Informal Design Tool for

User Interface Early PrototypingAdrien Coyette, Jean Vanderdonckt, Quentin Limbourg

Université catholique de Louvain (UCL)Louvain School of Management (IAG)

Information Systems Unit (ISYS)Belgian Lab. of Computer-Human Interaction (BCHI)

http://www.isys.ucl.ac.be/bchi{coyette, vanderdonckt, limbourg}@isys.ucl.ac.be

What’s the situation of today?

• Interactive Software evolution: context of use =(U,P,E)

Platform

Environment

Language

What is the situation today?

• Technological aspects of user interfaces progress significantly faster than– Software engineering aspects

• It takes time to develop a user interface with a new device, a new interaction technique

• It takes more time to develop a toolkit• It takes even more time to rely on a model-driven approach

– Usability engineering aspects• New user interfaces are shipped with usability problems

because– Little or no experience– No past, no empirical evidence

• Empirical experiments require a lot of resource if done carefully

How to address this issue?

• Capture essence of concepts through models– Separation of concerns, Correlability– Parsability, editability– If possible, human readability

• Typical models Task & Concepts

Abstract UI

Concrete UI

Final UI

Task & Concepts

Abstract UI

Concrete UI

Final UI

Source platform Target platform

Task & Concepts

Abstract UI

Concrete UI

Final UI

Task & Concepts

Abstract UI

Concrete UI

Final UI

Typical models

Task & Concepts

Abstract UI

Concrete UI

Final UI

Task & Concepts

Abstract UI

Concrete UI

Final UI

Task & Concepts

Abstract UI

Concrete UI

Final UI

Task & Concepts

Abstract UI

Concrete UI

Final UI

textInput button button

Window

AICfacet=control

AbstractIndividualContainer

textInput button button

Window

AICfacet=control

AbstractIndividualContainer

Mappings across levels

[LimbourgLimbourg, 2006]

Environment T

Final userInterface T

Concrete userInterface T

Task and Domain T

Abstract userInterface T

T=Target context of use

Concrete userInterface S

Final userInterface S

Task and Domain S

Abstract userInterface S

S=Source context of use

Reification

Abstraction

Reflexion

Translation

UsiXMLunsupported

UsiXMLsupported

User S Platform S Environment S Platform TUser T

Our goals

• Objectives– Description of interactive systems

• Using pre existing domain specific meta-models• Used both at design and run time

– Capitalization• Properties• Transformations

Interactive system

Model of the IS

Model of the IS ‘

Interactive system ‘

Checks of properties

Transformation

Models, instances of Meta-Models described in MOF (even for properties and transformations…)

UsiXML

• UsiXML =– USer Interface exTensible Markup Language– XML-compliant specification language for user interfaces suitable

for any interface• Web• Java• Windows-based applications• Multimodal applications, 3D applications• Virtual, mixed reality applications

– http://www.usixml.org– Join the UsiXML Consortium by registering on line

All the models in UsiXML

The Concrete User Interface in UsiXML

Simple widgets (e.g.,

check box)

Containers (e.g., dialog

Behaviour ofConcrete

interaction objects Relationships

betweenConcrete

interaction objects

MDE based on UsiXML

Model to Model

PlatformIndependentModel (PIM)

PlatformSpecific

Model (PSM)Model to Code Source code

MDE Components

Techniques proposed based on UsiXML

ComputingIndependentModel (CIM)

Model to Model

UsiXML model:Abstract user

interface

UsiXML model:Concrete user

interfaceRendering

Final userinterface

UsiXMLmodels: task,

domain

Graphtransformations

The whole picture of UsiXML according to MDE

UsiXML model:Abstract user

interface

UsiXML model:Concrete user

interface

Rendering

Final userinterface

UsiXMLmodels: task,

domainGenerative

programming

Graphtransformations

Derivation rules

IdealXML

ReversiXML

FlashiXMLQtkXML

InterpiXML

VisualiXML

TransformiXML

GrafiXMLVisiXML

SketchiXMLFormiXML

KnowiXML

MethodiXML

Presentation plan

1.1.UsiXMLUsiXML1.1. Overall presentationOverall presentation

2.2. SketchiXMLSketchiXML1.1. MotivationsMotivations

2.2. Paper prototypingPaper prototyping

3.3. Computer Assisted PrototypingComputer Assisted Prototyping

4.4. SketchiXML Development SketchiXML Development

5.5. SurveysSurveys

6.6. ConclusionConclusion

Motivations

user interface determines how easily a user may control underlying functions of a computer program.

Thus, a program equipped with powerful functionalities and low quality user interfaces may be under-exploited or misused

Motivations

Additionally, the proportion of code dedicated to user interface in application development is considerable:

from 50% to 70 %50% to 70 % of the application code

[Myers and Al, 2000]

How to build a graphical user interface efficiently?

Start all programs JBuilder build your UI

Motivations

Even if a “final result” is quickly available, the UI Even if a “final result” is quickly available, the UI created is likely to require a huge set of expensive created is likely to require a huge set of expensive modifications… modifications…

There are no rewards in life for being the There are no rewards in life for being the first one with the wrong answer. first one with the wrong answer.

[SnyderSnyder, 1996]

Motivations

As it was the case in software engineering, HCI communities wanted to develop a well-structured method for developing user interfaces.

However, the systematization, and the reproducibility found in software engineering methods cannot be transposed equally in HCI: the development life cycle remains inherently open, ill-defined, and highly iterative

[LimbourgLimbourg, 2004]

Motivations

We consider that informal techniques based on low fidelity mock up is the best alternative for the early design phase.

It allows to cope with the fact the such process is eminently open, iterative and incomplete

[SumnerSumner, 1997]

Motivations

Such approach allows to reduce the time needed between the cycle

Permit to involve the end user in the process since few technical background is required

And rises as many usability problems than using high fidelity mock-ups

[Virzi and AlVirzi and Al, 1996]

Presentation plan

MotivationsMotivations

Paper prototypingPaper prototyping

Computer Assisted PrototypingComputer Assisted Prototyping

SketchiXML Development SketchiXML Development

SurveysSurveys

DemonstrationDemonstration

ConclusionConclusion

Paper prototyping

What is it used for?

Early design phaseGet a sense of the user's needs and goalsQuickly prune the tree of possible designsRemove “locked-in” choices

Paper prototyping

How to Create a paper prototype ?

PaperEraserPencils ScissorsGlue/TapePost-it

Paper prototyping

Paper prototypes are as much about:

seeing how the user uses the design as it is about seeing what ideas (user

and designer) come up DURINGDURING the design

Iteration is keyIteration is key

Design Tab

How to do it?How to do it?

Brainstorm, ConceptDesignTest

Paper prototyping

BrainstormBrainstorm Come up with all the tasks and goals you think the user will

need in the interface Come up with all the individual functions that will be

performed in the interface Generate a concept of how each goal will be accomplished

DesignDesign Create paper mockups of a series of screens representing

each feature One screen for each step in each task Generate a diagram of “ state transitions” from one mockup to

another[SnyderSnyder, 1996]

Paper prototyping

TestTestPick users and have them step through common tasksAs they move through the steps, describe what happens after each action, and show them the next screenWrite down mistakes, misunderstandings, problems, things that worked

IterateIterateUsing what you learned, redesign the paper prototype to improve the designRetest and redesign until all common problems are removed

Paper prototyping

Paper prototyping appears thus to be the a Paper prototyping appears thus to be the a natural answer:natural answer:familiar and unconstrained approach

fast to learn and quick to produce

focus on basic structural issues instead of unimportant details

it is very appropriate to convey ongoing, unfinished designs, and it encourages creativity,

can be performed collaboratively between designers and end-users

Design Tab

But drawbacks also exist…changes are hard to accomplish

The designer has to redraw the common features that the design retains

lack of interaction between the paper-based design and a user (a designer needs to"play computer" )

Can't emulate complex interactions

But...it isn't designed for those purposes:But...it isn't designed for those purposes:It is designed to solve problems before any It is designed to solve problems before any money or time is invested in a solutionmoney or time is invested in a solution

Paper prototyping

Designers almost never use paper prototyping in design projects

Why?Why?Doesn't impress?Too simple and cheap to do anything valuable?Probably a little of both

Paper prototyping

The idea of developing a computer-based tool for sketching UIs naturally emerged from these observations

Such tools would extend the advantages provided by sketching techniques by easily creating, deleting, updating or moving UI elements.

Some research was carried out in order to propose a hybrid approach, combining the best of the hand-sketching and computer assisted interface design, but as we will se trough the state of the art, this marriage presents a set of shortcomings.

Presentation plan

SurveysSurveys

Computer Assisted Prototyping

DENIM(James A. Landay, James Lin, Mark W. Newman, Jason I. Hong)

http://dub.washington.edu/denim/

Advantages:Advantages:

Support for scenario-based design Several levels of granularity Good documentation Toolbox of generic widgets Mature product, based on experimental

testing Zooming facility from local design (e.g.

a web page) to a global design (e.g., a portion of a web site or an entire web site)

Storyboarding facilities based on patterns

Shortcomings:Shortcomings:

No shape recognition and interpretation, thus loosing the effort

No code generation No preview mode Only dedicated to web sites

AdvantagesAdvantages

Performance (speed and accuracy) Multi-stroke gestures Recognizes rotated shapes Computationally light Open source Requires standard and freely available

libraries Extensible shape interpretation grammar

Shortcomings:Shortcomings:

No scenario editor Mono-window Only generates java (and no UI spec) Limited widget set Shape interpretation can only take as

input a construct made of a maximum of two vectorial shapes

No zoom

Presentation plan

SketchiXML DevelopmentSketchiXML Development

SurveysSurveys

SketchiXML Development

Avoidance of Effort loss

Large conceptual coverage

Robust recognition

Language neutrality

Ease of use (naturalness)

Flexible processing

Context independency

Requirements Requirements elicitation:elicitation:

Context independency

Well defined editing functionalities.

Save and export facilities.

Zooming capabilities

Robust scenario editor

Recognition flexibility …

Requirements Requirements elicitation:elicitation:

SketchiXML as a Multi-Agent SystemMulti-Agent System

Adaptive

Modular

Flexible

Software Agent •Implemented with/in software technologies •Environment : humans, machines, other software agents, platforms.

AgentAgent : : Individual who can act • autonomous, pro- active, adaptative with/in its

environment IntelligenceIntelligence

Multi-agent system: Multi-agent system: organization of individuals to organization of individuals to achieve particular, possible achieve particular, possible common goals.common goals.

I* representation of the Virtual I* representation of the Virtual mediator :mediator :

When a shape is provided, the mediator send it to the appropriate agents for recognition and integrate answers

Recognition engine :Cali libraryRecognition engine :Cali library

A fast, simple and compactA fast, simple and compact approach to identify Scribbles (multi-stroke geometric shapes)

Combines temporal adjacency, Fuzzy Logic and temporal adjacency, Fuzzy Logic and geometric featuresgeometric features to classify scribbles.

Recognizes elementary geometric shapes, such as Triangles, Rectangles, Diamonds, Circles… and some gesture commands, such as, Delete, Cross, WavyLine, Move, Copy and Tap.

Shapes are recognized independently of changes in recognized independently of changes in rotation, size or number of individual strokesrotation, size or number of individual strokes.

The recognizer has a recognition rate of 97%97% and a fast response

Using the cali library:– widgets have to be “constructed” based on

geometric primitives (elipse, rectangle, line, …)– each geometric primitive is entered (drawn)

separately

• The new approach:– user can give examples for each widget (training)– complete widget can be entered all at once– distance measure between trained and drawn

widgets based on stroke directions– Multi-strokes too (!= Rubine’s algorithm)

Recognition engine :Trainable Gesture RecognizerRecognition engine :Trainable Gesture Recognizer Handdrawn input (sketch) is

superimposed with a grid Each sampling point is mapped to

its closest grid node For adjacent nodes, a stroke

direction (out of 8 different directions) is taken

A string is generated from direction coding of input

Complex task of comparison of drawing inputs is reduced to simple calculation of string edit distances

Distance is shortest sequence of edit commands that transform s to t Simplest set of operations:

•Copy character from s over to t •Delete a character in s (cost 1) •Insert a character in t (cost 1) •Substitute one character for another (cost 1)

Levenstein Levenstein distance distance

The shape interpretationshape interpretation is based on the same mechanism:

When the program starts, the visual grammar is loaded

For each widget representation, an agent is created and try to build a widget according to his dedicated pattern

Each of these agents register to the mediator, and provide the kind of shape required according to its pattern

Drawing Canvas

Shapes Recognition

Shape Interpretation

UpdateBeliefs

UpdateUI

SketchiXML SketchiXML

processprocess

Presentation plan

SurveysSurveys

Surveys

Building the widget Building the widget catalogue catalogue

Surveys

Lisbox 3 Lisbox 1 Lisbox 2 Lisbox 4 Lisbox 5

Surveys

Check box Square Line

Elem ents

Recognition rate of the applicationRecognition rate of the application

Overall recognition rate for widget : 94.12% (only)

No difference between end user and computer scientist

Button Rectangle1 Rectangle2

Elem ents

Radio button Circle Line

Elem ents

Combo box Rectangle Triangle

Elem ents

Surveys

Evaluation of the usability of the application:Evaluation of the usability of the application:1. Evaluation of the time needed to draw a simple form with the

application:

The time needed decrease strongly as user get used to the application (learning effect)

End user are significantly faster than computer scientist to draw the same user interface

2. Evaluation of the application with a multiple choice questionnaire at the end of the test

Most of the participant enjoyed the test

Most of them found the application easy to use

Most of the participant were satisfied with the results (despite the low recognition rate)

Surveys

Presentation plan

SurveysSurveys

Demonstration

Presentation plan

SurveysSurveys

Conclusion

We have introduced a new and innovative sketching tool that generates a user, platform, and environment independent output

We have met most of the requirements that were identified as important shortcomings of existing tools

we have also conducted a survey on 60 peoples from different activity sectors with different backgrounds, in order to identify how these people would intuitively represent the widgets to be handled by SketchiXML

We have conducted a survey on 40 peoples with different background in order to test the application

SketchiXML extends a set of tools based on UsiXML, allowing to initiate the design process from the early design phase to the final concrete user interface for different platforms, with tools support for every stages

Conclusion

Meet all the requirements elicited in the state of the art and particularly the Development of a scenario editor allowing to represent transition between screen.

During the sketching process, the possibility to instantly switch to a runnable version of the current UI would be useful. Indeed, all informal design tools providing code generation allow easy switching from design to run mode, while SketchiXML requires to invoke a third application.

Thank you very much for your attention

For more information and downloading,http://www.isys.ucl.ac.be/bchi

http://www.usixml.orgUser Interface eXtensible Markup Language

http://www.similar.ccEuropean network on Multimodal UIs

http://www.usixml.org/index.php?view=page&idpage=29SketchiXML home page

1invited talk for ruipiacs’2006 workshop – geneva, switzerland, september 13 rd, 2006...

Documents

ihe tutorial geneva oct. 10th, 2006 ihe overview

united nations new york and geneva, 2006 - gdrc · ·...

2006-02 cad/cam rp (rapid...

national grid in singapore – moving forward egee 2006...

automated prototyping of user interfaces based on...

st22 revision proposal june-2006 wipo-sdwg meeting geneva

reference u/a for gcos seattle 2006 richard k. thigpen gcos...

communication and dissemination mechanisms geneva, 24 may...

16.682 - prototyping avionics spring 2006 · pdf...

prototyping with a purpose · prototyping with a purpose...

rapid prototyping capability project 2006

regional agreement, geneva 2006: plan modification …

new middle school geneva, il greg kemerer ae senior thesis...

systematic review protocol - geneva foundation for...

think game play! 2006-advanced-prototyping/ 016

virtual prototyping & rapid prototyping

lean prototyping: prototyping for lean process

2006 © switch grid activities at switch christoph witzig...

geneva, may 30, 2006 - bienvenue à...

1 prototyping tools khoo yit phang cmsc 434 october 26, 2006...