1invited talk for ruipiacs’2006 workshop – geneva, switzerland, september 13 rd, 2006...

1 Invited talk for RUIPIACS’2006 Workshop – Geneva, Switzerland, September 13 rd, 2006

SketchiXML:an Informal Design Tool for

User Interface Early PrototypingAdrien Coyette, Jean Vanderdonckt, Quentin Limbourg

Université catholique de Louvain (UCL)Louvain School of Management (IAG)

Information Systems Unit (ISYS)Belgian Lab. of Computer-Human Interaction (BCHI)

http://www.isys.ucl.ac.be/bchi{coyette, vanderdonckt, limbourg}@isys.ucl.ac.be


What’s the situation of today?

• Interactive Software evolution: context of use =(U,P,E)

time

Platform

User

Environment

Language


What is the situation today?

• Technological aspects of user interfaces progress significantly faster than– Software engineering aspects

• It takes time to develop a user interface with a new device, a new interaction technique

• It takes more time to develop a toolkit• It takes even more time to rely on a model-driven approach

– Usability engineering aspects• New user interfaces are shipped with usability problems

because– Little or no experience– No past, no empirical evidence

• Empirical experiments require a lot of resource if done carefully


How to address this issue?

• Capture essence of concepts through models– Separation of concerns, Correlability– Parsability, editability– If possible, human readability

• Typical models Task & Concepts

Abstract UI

Concrete UI

Final UI

Task & Concepts

Abstract UI

Concrete UI

Final UI

Source platform Target platform

Task & Concepts

Abstract UI

Concrete UI

Final UI

Task & Concepts

Abstract UI

Concrete UI

Final UI



Typical models

Task & Concepts

Abstract UI

Concrete UI

Final UI

Task & Concepts

Abstract UI

Concrete UI

Final UI


Task & Concepts

Abstract UI

Concrete UI

Final UI

Task & Concepts

Abstract UI

Concrete UI

Final UI


textInput button button

Window

AICfacet=control

AICfacet=control

AICfacet=control

AbstractIndividualContainer

textInput button button

Window

AICfacet=control

AICfacet=control

AICfacet=control

AbstractIndividualContainer


Mappings across levels

[LimbourgLimbourg, 2006]

Environment T

Final userInterface T

Concrete userInterface T

Task and Domain T

Abstract userInterface T

T=Target context of use

Concrete userInterface S

Final userInterface S

Task and Domain S

Abstract userInterface S

S=Source context of use

Reification

Abstraction

Reflexion

Translation

UsiXMLunsupported

model

UsiXMLsupported

model

User S Platform S Environment S Platform TUser T


Our goals

• Objectives– Description of interactive systems

• Using pre existing domain specific meta-models• Used both at design and run time

– Capitalization• Properties• Transformations

Interactive system

Model of the IS

Model of the IS ‘

Interactive system ‘

Checks of properties

Transformation

Models, instances of Meta-Models described in MOF (even for properties and transformations…)


UsiXML

• UsiXML =– USer Interface exTensible Markup Language– XML-compliant specification language for user interfaces suitable

for any interface• Web• Java• Windows-based applications• Multimodal applications, 3D applications• Virtual, mixed reality applications

– http://www.usixml.org– Join the UsiXML Consortium by registering on line

http://www.usixml.org/


All the models in UsiXML


The Concrete User Interface in UsiXML

Simple widgets (e.g.,

check box)

Containers (e.g., dialog

box)

Behaviour ofConcrete

interaction objects Relationships

betweenConcrete

interaction objects


MDE based on UsiXML

Model to Model

PlatformIndependentModel (PIM)

PlatformSpecific

Model (PSM)Model to Code Source code

MDE Components

Techniques proposed based on UsiXML

ComputingIndependentModel (CIM)

Model to Model

UsiXML model:Abstract user

interface

UsiXML model:Concrete user

interfaceRendering

Final userinterface

UsiXMLmodels: task,

domain

Graphtransformations



The whole picture of UsiXML according to MDE

UsiXML model:Abstract user

interface

UsiXML model:Concrete user

interface

Rendering

Final userinterface

UsiXMLmodels: task,

domainGenerative

programming



Derivation rules

IdealXML

ReversiXML

FlashiXMLQtkXML

InterpiXML

VisualiXML

TransformiXML

GrafiXMLVisiXML

SketchiXMLFormiXML

KnowiXML

MethodiXML


Presentation plan

1.1.UsiXMLUsiXML1.1. Overall presentationOverall presentation

2.2. SketchiXMLSketchiXML1.1. MotivationsMotivations

2.2. Paper prototypingPaper prototyping

3.3. Computer Assisted PrototypingComputer Assisted Prototyping

4.4. SketchiXML Development SketchiXML Development

5.5. SurveysSurveys

6.6. ConclusionConclusion


Motivations

user interface determines how easily a user may control underlying functions of a computer program.

Thus, a program equipped with powerful functionalities and low quality user interfaces may be under-exploited or misused


Motivations

Additionally, the proportion of code dedicated to user interface in application development is considerable:

from 50% to 70 %50% to 70 % of the application code

[Myers and Al, 2000]


How to build a graphical user interface efficiently?

Start all programs JBuilder build your UI

Motivations


Motivations

Even if a “final result” is quickly available, the UI Even if a “final result” is quickly available, the UI created is likely to require a huge set of expensive created is likely to require a huge set of expensive modifications… modifications…

There are no rewards in life for being the There are no rewards in life for being the first one with the wrong answer. first one with the wrong answer.

[SnyderSnyder, 1996]


Motivations

As it was the case in software engineering, HCI communities wanted to develop a well-structured method for developing user interfaces.

However, the systematization, and the reproducibility found in software engineering methods cannot be transposed equally in HCI: the development life cycle remains inherently open, ill-defined, and highly iterative

[LimbourgLimbourg, 2004]


Motivations

We consider that informal techniques based on low fidelity mock up is the best alternative for the early design phase.

It allows to cope with the fact the such process is eminently open, iterative and incomplete

[SumnerSumner, 1997]


Motivations

Such approach allows to reduce the time needed between the cycle

Permit to involve the end user in the process since few technical background is required

And rises as many usability problems than using high fidelity mock-ups

[Virzi and AlVirzi and Al, 1996]


Presentation plan

MotivationsMotivations

Paper prototypingPaper prototyping

Computer Assisted PrototypingComputer Assisted Prototyping

SketchiXML Development SketchiXML Development

SurveysSurveys

DemonstrationDemonstration

ConclusionConclusion


Paper prototyping

What is it used for?

Early design phaseGet a sense of the user's needs and goalsQuickly prune the tree of possible designsRemove “locked-in” choices



Paper prototyping

How to Create a paper prototype ?

PaperEraserPencils ScissorsGlue/TapePost-it



Paper prototyping



Paper prototyping

Paper prototypes are as much about:

seeing how the user uses the design as it is about seeing what ideas (user

and designer) come up DURINGDURING the design

Iteration is keyIteration is key



Design Tab

How to do it?How to do it?

Brainstorm, ConceptDesignTest



Paper prototyping

BrainstormBrainstorm Come up with all the tasks and goals you think the user will

need in the interface Come up with all the individual functions that will be

performed in the interface Generate a concept of how each goal will be accomplished

DesignDesign Create paper mockups of a series of screens representing

each feature One screen for each step in each task Generate a diagram of “ state transitions” from one mockup to

another[SnyderSnyder, 1996]


Paper prototyping

TestTestPick users and have them step through common tasksAs they move through the steps, describe what happens after each action, and show them the next screenWrite down mistakes, misunderstandings, problems, things that worked

IterateIterateUsing what you learned, redesign the paper prototype to improve the designRetest and redesign until all common problems are removed



Paper prototyping

Paper prototyping appears thus to be the a Paper prototyping appears thus to be the a natural answer:natural answer:familiar and unconstrained approach

fast to learn and quick to produce

focus on basic structural issues instead of unimportant details

it is very appropriate to convey ongoing, unfinished designs, and it encourages creativity,

can be performed collaboratively between designers and end-users


Design Tab

But drawbacks also exist…changes are hard to accomplish

The designer has to redraw the common features that the design retains

lack of interaction between the paper-based design and a user (a designer needs to"play computer" )

Can't emulate complex interactions

But...it isn't designed for those purposes:But...it isn't designed for those purposes:It is designed to solve problems before any It is designed to solve problems before any money or time is invested in a solutionmoney or time is invested in a solution


Paper prototyping

Designers almost never use paper prototyping in design projects

Why?Why?Doesn't impress?Too simple and cheap to do anything valuable?Probably a little of both



Paper prototyping

The idea of developing a computer-based tool for sketching UIs naturally emerged from these observations

Such tools would extend the advantages provided by sketching techniques by easily creating, deleting, updating or moving UI elements.

Some research was carried out in order to propose a hybrid approach, combining the best of the hand-sketching and computer assisted interface design, but as we will se trough the state of the art, this marriage presents a set of shortcomings.


Presentation plan





SurveysSurveys




Computer Assisted Prototyping

DENIM(James A. Landay, James Lin, Mark W. Newman, Jason I. Hong)

http://dub.washington.edu/denim/



Advantages:Advantages:

Support for scenario-based design Several levels of granularity Good documentation Toolbox of generic widgets Mature product, based on experimental

testing Zooming facility from local design (e.g.

a web page) to a global design (e.g., a portion of a web site or an entire web site)

Storyboarding facilities based on patterns



Shortcomings:Shortcomings:

No shape recognition and interpretation, thus loosing the effort

No code generation No preview mode Only dedicated to web sites



AdvantagesAdvantages

Performance (speed and accuracy) Multi-stroke gestures Recognizes rotated shapes Computationally light Open source Requires standard and freely available

libraries Extensible shape interpretation grammar



Shortcomings:Shortcomings:

No scenario editor Mono-window Only generates java (and no UI spec) Limited widget set Shape interpretation can only take as

input a construct made of a maximum of two vectorial shapes

No zoom


Presentation plan




SketchiXML DevelopmentSketchiXML Development

SurveysSurveys




SketchiXML Development

Avoidance of Effort loss

Large conceptual coverage

Robust recognition

Language neutrality

Ease of use (naturalness)

Flexible processing

Context independency

Requirements Requirements elicitation:elicitation:



Context independency

Well defined editing functionalities.

Save and export facilities.

Zooming capabilities

Robust scenario editor

Recognition flexibility …

Requirements Requirements elicitation:elicitation:



SketchiXML as a Multi-Agent SystemMulti-Agent System

Adaptive

Modular

Flexible



Software Agent •Implemented with/in software technologies •Environment : humans, machines, other software agents, platforms.

AgentAgent : : Individual who can act • autonomous, pro- active, adaptative with/in its

environment IntelligenceIntelligence

Multi-agent system: Multi-agent system: organization of individuals to organization of individuals to achieve particular, possible achieve particular, possible common goals.common goals.



I* representation of the Virtual I* representation of the Virtual mediator :mediator :

When a shape is provided, the mediator send it to the appropriate agents for recognition and integrate answers



Recognition engine :Cali libraryRecognition engine :Cali library

A fast, simple and compactA fast, simple and compact approach to identify Scribbles (multi-stroke geometric shapes)

Combines temporal adjacency, Fuzzy Logic and temporal adjacency, Fuzzy Logic and geometric featuresgeometric features to classify scribbles.

Recognizes elementary geometric shapes, such as Triangles, Rectangles, Diamonds, Circles… and some gesture commands, such as, Delete, Cross, WavyLine, Move, Copy and Tap.

Shapes are recognized independently of changes in recognized independently of changes in rotation, size or number of individual strokesrotation, size or number of individual strokes.

The recognizer has a recognition rate of 97%97% and a fast response



Using the cali library:– widgets have to be “constructed” based on

geometric primitives (elipse, rectangle, line, …)– each geometric primitive is entered (drawn)

separately

• The new approach:– user can give examples for each widget (training)– complete widget can be entered all at once– distance measure between trained and drawn

widgets based on stroke directions– Multi-strokes too (!= Rubine’s algorithm)



Recognition engine :Trainable Gesture RecognizerRecognition engine :Trainable Gesture Recognizer Handdrawn input (sketch) is

superimposed with a grid Each sampling point is mapped to

its closest grid node For adjacent nodes, a stroke

direction (out of 8 different directions) is taken

A string is generated from direction coding of input

Complex task of comparison of drawing inputs is reduced to simple calculation of string edit distances



Distance is shortest sequence of edit commands that transform s to t Simplest set of operations:

•Copy character from s over to t •Delete a character in s (cost 1) •Insert a character in t (cost 1) •Substitute one character for another (cost 1)

Levenstein Levenstein distance distance



The shape interpretationshape interpretation is based on the same mechanism:

When the program starts, the visual grammar is loaded

For each widget representation, an agent is created and try to build a widget according to his dedicated pattern

Each of these agents register to the mediator, and provide the kind of shape required according to its pattern



Drawing Canvas

Shapes Recognition

Shape Interpretation

UpdateBeliefs

UpdateUI

SketchiXML SketchiXML

processprocess


Presentation plan





SurveysSurveys




Surveys

Building the widget Building the widget catalogue catalogue


Surveys


Surveys

0

50

100

150

200

Lisbox 3 Lisbox 1 Lisbox 2 Lisbox 4 Lisbox 5

0

50

100

150

200


150

200

250

300



Surveys

0

20

40

60

80

100

Check box Square Line

Elem ents

succ

essf

ul

reco

gn

itio

n r

ate

Recognition rate of the applicationRecognition rate of the application

Overall recognition rate for widget : 94.12% (only)

No difference between end user and computer scientist

0

20

40

60

80

100

Button Rectangle1 Rectangle2

Elem ents

succ

essf

ul

reco

gn

itio

n r

ate

0

20

40

60

80

100

Radio button Circle Line

Elem ents

succ

essf

ul

reco

gn

itio

n r

ate

0

20

40

60

80

100

Combo box Rectangle Triangle

Elem ents

succ

essf

ul

reco

gn

itio

n r

ate


Surveys

Evaluation of the usability of the application:Evaluation of the usability of the application:1. Evaluation of the time needed to draw a simple form with the

application:

The time needed decrease strongly as user get used to the application (learning effect)

End user are significantly faster than computer scientist to draw the same user interface

2. Evaluation of the application with a multiple choice questionnaire at the end of the test

Most of the participant enjoyed the test

Most of them found the application easy to use

Most of the participant were satisfied with the results (despite the low recognition rate)


Surveys


Presentation plan





SurveysSurveys




Demonstration


Presentation plan





SurveysSurveys




Conclusion

We have introduced a new and innovative sketching tool that generates a user, platform, and environment independent output

We have met most of the requirements that were identified as important shortcomings of existing tools

we have also conducted a survey on 60 peoples from different activity sectors with different backgrounds, in order to identify how these people would intuitively represent the widgets to be handled by SketchiXML

We have conducted a survey on 40 peoples with different background in order to test the application

SketchiXML extends a set of tools based on UsiXML, allowing to initiate the design process from the early design phase to the final concrete user interface for different platforms, with tools support for every stages


Conclusion

Meet all the requirements elicited in the state of the art and particularly the Development of a scenario editor allowing to represent transition between screen.

During the sketching process, the possibility to instantly switch to a runnable version of the current UI would be useful. Indeed, all informal design tools providing code generation allow easy switching from design to run mode, while SketchiXML requires to invoke a third application.


Thank you very much for your attention

For more information and downloading,http://www.isys.ucl.ac.be/bchi

http://www.usixml.orgUser Interface eXtensible Markup Language

http://www.similar.ccEuropean network on Multimodal UIs

http://www.usixml.org/index.php?view=page&idpage=29SketchiXML home page

http://www.isys.ucl.ac.be/bchi

http://www.usixml.org/

http://www.similar.cc/

http://www.usixml.org/index.php?view=page&idpage=29

1invited talk for ruipiacs’2006 workshop – geneva, switzerland, september 13 rd, 2006...

Documents