1 workshop on model driven software engineering for ambient intelligence applications – darmstadt,...

1 Workshop on Model Driven Software Engineering for Ambient Intelligence Applications – Darmstadt, Nov. 7th, 2007

Distributed User Interfacesin Ambient Environment

Jean Vanderdonckt1, Hildeberto Mendonça1, José Pascual Molina Massó1,2

1Université catholique de Louvain (UCL)Louvain School of Management (LSM), Information Systems Unit (ISYS)

Belgian Laboratory of Computer-Human Interaction (BCHI)http://www.isys.ucl.ac.be/bchi

2Universidad de Castilla-La Mancha (UCLM)Laboratory of User Interaction & Software Engineering (LoUISE)

Instituto de Investigación en Informática de Albacete (I3A)http://www.i3a.uclm.es/

http://www.isys.ucl.ac.be/bchi

http://www.i3a.uclm.es/


Introduction (1/3)

• We live in an ambient, but distributed, world– Multiple devices, multiple locations, multiple users

A real distributed world A future (?) distributed world

Multiple PCs and Displays (% respondants)

0%

10%

20%

30%

40%

50%

60%

70%

80%

None Multiple monitorsattached to multiple

computers.

Laptop and desktopmonitor connected

together.

Dualmon or higher

Harris poll for multiple PCsand displays (July 2002)


Introduction (2/3)

• Distributed User Interfaces (DUIs) are a particular type of UIs whose portions are distributed across places or locations in a particular physical environment.

• Designing a DUI remains a complex problem which may prevent designers from exploring design issues– Too expensive to develop many alternatives– Hard to modify once coded

Rapid prototyping– Rapid prototyping of DUIs turns to be an important issue in AmI– Could be used as a vehicle for developing and demonstrating

visions of innovative DUIs– Could show various distribution configurations before going to full

implementation


Introduction 3/3

• Related work– DYNAMO-AID– MIGRATION project– Everywhere– 3DSim– VAQUITA

Our approach is different from previous pieces of work– It relies on a physical environment model, produced as a

virtual reality scene for rapid prototyping purposes– The environment is decomposed into surfaces, some being

interactive– These surfaces are attached to a service model, characterizing

the capabilities they can offer


The underlying model (1/5)

• Context of use: is decomposed into 3 aspects– User: population is expressed in terms of

stereotypes sharing the same attributes (e.g., preferences, age, skills)

– Platform: sofware –computing- and hardware –viewing- (e.g. mobile, laptop, desktop, wall screen)

– Environment: the physical context is represented as a virtual reality scene, inspired from/based on VRML/X3D

Context

UserStereotype Platform Location Environment

Hardwareplatform

Interactionsurface

Service

Event

Condition

Message

Role



• Interaction surface– Refers to any physical surface which can be “acted on or observed”

so as to support user interaction with a system, whether visible or embedded.

– 2 types:• Digital: a screen, a monitor, a wall display• Physical: a table equipped with a camera tracking system, a

real device

[Carpendale, 2006]

Context


Hardwareplatform

Interactionsurface

Service

Event

Condition

Message

Role



• Location– Positioning of interaction surfaces and platforms in the

environment– To be represented by potential positions of platforms in the

virtual reality scene

Digital interaction surface on the wall

Digital interaction surface on the desk

Context


Hardwareplatform

Interactionsurface

Service

Event

Condition

Message

Role



• Service– Any basic function provided by the interaction surface to the

designer who is prototyping the DUI

Services provided by this interaction surface:- cuiRendering: renders a CUI on this surface- cuiUnrendering: unrenders a previously rendered CUI from this surface- cuiAdaptation: adapts a CUI for this surface

Context


Hardwareplatform

Interactionsurface

Service

Event

Condition

Message

Role



• Event: Occurrence during service execution, e.g. widgetRenderingError

• Condition: Precondition, post-condition and integrity constraints

• Message: A container of information, used to signal events• Role: Responsible for perfoming services and raising events

Event: CUI rendered on this surfaceCondition: resolution is large enough to accommodate the CUIMessage: To cuiAdaptation service: needs to adapt to the new constraints

Context


Hardwareplatform

Interactionsurface

Service

Event

Condition

Message

Role


Software Tool for Rapid Prototyping 1/6

• A software tool named VUIToolkit has been developed above UsiXML and expanded with the previous environment model.

• VUIToolkit offers a set of VRML97/X3D PROTOs that transform the standard plain widgets into a truly 3D representation, referred as Virtualized User Interface.



• With this tool, the environment model is produced as a Virtual Reality scene, thus allowing the rendering of both software (e.g. widgets) and hardware (e.g. physical buttons), but just for rapid prototyping purposes.

• The tool provides basic services such as copy a CUI from one surface to another one, thus migrating it.



• When mapping a CIO from CUI to a VRML or X3D world, different cases are assessed:– Direct mapping.

• It is not possible to set a one-to-one mapping, as there is no VRML/X3D primitives that can be matched with a CIO. These Web3D languages specify basic elements that must be used together to create interactive elements.



• When mapping a CIO from CUI to a VRML or X3D world, different cases are assessed:– Direct mapping.– New mapping.

• There is a need to fill a gap by introducing a new widget by appropriate implementation.

• Some attributes defined in the CUI model are not used in the 3D world, while new attributes must be added in order to describe the interface elements in the 3D world, such as position and size.



• When mapping a CIO from CUI to a VRML or X3D world, different cases are assessed:– Direct mapping.– New mapping.– No possible mapping.

• There are CIOs whose VRML/X3D counterpart have not been done yet or are impossible or difficult to implement. An example of this case is the CIO box.



• Input/output mapping from desktop-oriented 2D GUI to VRML/X3D virtual world.– Digital Interaction Surface / Display = “Screen” PROTO.

• Rectangular area where screen units (pixels) are converted into world units (meters).

• Serves as a container of UI elements (children field).– Mouse = User’s pointing device.

• VRML/X3D sensors.– Keyboard?

• VRML: requires non-standard extensions (PG’s KbdSensor).• X3D: StringSensor.• Virtual keyboard.


Examples 1/6

• The user’s environment is a small office, with five different platforms


Examples 2/6

• User Interface Rendering: cuiRendering


Examples 3/6

• User Interface Migration: cuiMigration

cuiStoring

cuiUnrendering

cuiRetrieving

cuiRendering


Examples 4/6

• User Interface Adaptation: cuiAdaptation

1

2

3

Aspire 200015”, 16:101280x800

e7503,8”, 3:4240x320


Examples 5/6

• Other Services: uiPhysicalization, uiDigitization


Examples 5/6

• Other Services: uiPhysicalization, uiDigitization

Physical interaction surface

Digital interaction surface


Conclusion

• Benefits– Rapid prototyping of DUIs by manipulation in a virtual reality scene– Permanent maintaining of a model behind (CUI + context)– Rendering as a model-to-code service– Adaptation as a model-to-model service– Transition between digital and physical worlds is easy

• Shortcomings– Possible composition of services, but not implemented– Composition should be based on messages, but also on quality of

services• E.g., what is the closest surface to me that could render this

UI?– Manual operations trigger services: no algorithm for distributing UI

pieces on interaction surfaces– Rendering limited by widgets provided by VUIToolkit


How to model the quality of services provided by interaction surfaces

OwnesOwner*0..1

-isInvariant

QoSCharacteristic0..1

*Groupes

GroupedInTemplate Parent

Derivations Sub*

*

*

1

QoSParameter

Type

Parameter

1

*-statisticalQualifier-direction-unit : string

QoSDimension

referenced*

Type0..1

Typed*

DimensionOf1

Quantifier1..*

-Value

QoSDimensionSlotValue

Evaluate

1

*

-Value

QoSValue

Evaluates

Slot*

1 0..1ReferencedValue

-isQoSObservation

QoSContext

BasedOn*

*Context

-Qualification-EndToEndSource-EndToEndTarget-AllowedValues-LogicalOperator

QoSConstraint

OppositeAssociation**

Context

Supports*

1..*

ValidValues

Evaluates* 0..1

QoSContractQoSRequired QoSOffered

Previous

Next

0..1

0..1

ContractContract * ** *RequireAccorded OfferAccorded

QoSCompoundConstraint

SubConstraintsGlobalConstraints*

0..1

*

QoSCategory

QoSLevel

QoSCompoundLevel

-AdaptationActions : String

QoSTransition

AllowedSpaces

Levels

0..1

0..1

0..1GroupOfLevels

LevelsIncluded1..*

AllowedSpace*

*CurrentLevel

*

*

To

From

Source1

1Destination

• Allow utilization of QoS at various stages• Specifications of

– Users– providers

• Existing models– Q-WSDL, WSLA, DAML-QoS, etc.

• UML Profile for Modeling Quality of Service

– Metamodel– Characteristics quantified by

dimensions– Constraints specifying expected values– Context in which quality expressions

are involved– Levels that can be achieved under

constraints and context[OMG, 2006 ; Hersens, 2007]


Upcoming events…

Albacete (Spain), June 2008http://cadui2008.albacete.org/

http://cadui2008.albacete.org/


Danke shön! – Merci! – Gracias!

For more information and downloading,http://www.isys.ucl.ac.be/bchi

http://www.usixml.orgUser Interface eXtensible Markup Language

http://www.similar.ccEuropean network on Multimodal UIs

Special thanks to all members of the team!

http://www.isys.ucl.ac.be/bchi

http://www.usixml.org/

http://www.similar.cc/

1 workshop on model driven software engineering for ambient intelligence applications – darmstadt,...

Documents