1 phd public defense, 25 june 2008 a methodology for developing multimodal user interfaces of...

1 PhD Public Defense, 25 June 2008

A Methodology for Developing Multimodal User Interfaces of

Information Systems

Adrian Stanciulescu

Université catholique de Louvain (UCL)Louvain School of Management (LSM)

Information Systems Unit (ISYS)Belgian Laboratory of Computer-Human Interaction (BCHI)

http://www.isys.ucl.ac.be/bchi


Outline

• Context & Demonstration

• State of the art

• Thesis statement & Focus

• Methodology:

– Models & Language

– Method

– Tool support

• Validation

– External

– Internal

• Conclusion

• Future work


Context

• 5 senses: sight, hearing, touch, smell, taste• Different user-system interaction types

graphical vocal tactile olfactory gustatory

• Prevailing interaction: graphical

• Shortcomings:– restrictive (graphical)– less flexible than human-to-human

interaction– difficult to use in mobile

environments with new emerging devices


Multimodal systems

• New interaction paradigm:– Combines two or more individual interaction – Generic benefits: naturalness, flexibility, robustness, error

avoidance

• Specific benefits: – wide range of users (disabled persons)

– environment (mobile)

– platform (heterogenous devices)


DemonstrationInput: graphical (A) Input: vocal (A) Input: multimodal (E)


Outline

• Context



• Methodology:


– Method

– Tool support

• Validation

– External

– Internal

• Conclusion

• Future work


State of the art: Concerns

• Concerns – lack of: – structuring framework for MM UI development, support for MM I/O,

separation of modalities, combination of modalities, modality-independent models, extendibility for new modalities, method extendibility

Concerns of multimodal UIs

Chapter 2

Real world

Real world

State of the Art

(2) analyseChapter 1

Thesis statement:1. Models2. Method3. Tool support

Methodology

(1) define

Shortcomings

(3) identify

12 Requirements

(4) elicit

(5) guide (6) validate


State of the art: Shortcomings

• Survey of 8 languages and 11 development tools

• Shortcomings: lack of– MM application deployment, fast interaction, error recovery, platform mobility, usable

MM UIs, robust systems, device effectiveness, MM experience

• Multimodal Teresa[Paterno, 2004]– Design space with unexplicit

options and limited alternatives– Mix of modality dependent and

independent elements– CARE: no support for R and C– Pre-computed and hard-coded

transformations


State of art: Requirements

• Modeling:1. Support for MM I/O2. Separation of modalities3. Support for CARE properties in I/O4. Ability of modeling a UI independent of any modality5. Extendibility to new modalities6. Ontology homogeneity7. Human readability

• Method:8. Approach based on design space9. Method explicitness10.Method extendibility

• Tool:11.Machine processability of involved models12.Support for tool interoperability


Outline

• Context



• Methodology:


– Method

– Tool support

• Validation

– External

– Internal

• Conclusion

• Future work


Thesis statement & Focus

• Working hypotheses:– Model-based approach: a set of models specifying different abstractions of the final UI – Semi-automated transformational approach: rules manually selected and automatically

applied

• Focus:– Information systems– Predefined and constant contexts of use– Graphical, vocal and MM interactions– MM UIs familiar to vast majority of users and available on most platforms– Target audience: HCI reasearch comunity, professionals in the field of MM interaction

Define a design space-based method that is supported by model-to-model colored transformations in order to obtain multimodal user interfaces

of information systems from a task and a domain models.


Outline

• Context



• Methodology:


– Method

– Tool support

• Validation

– External

– Internal

• Conclusion

• Future work


Models & Language

• Cameleon Reference Framework:– Task Model – Domain Model– Abstract User Interface Model – Concrete User Interface Model– Mapping Model– Transformation Model

• UsiXML v1.8: – integrates our extensions– 3 dimensions of linguistics: semantics,

syntax, stylistics

Task & Concepts

Abstract User Interface

Concrete User Interface

Final User Interface

• extended CTT + extensions• modality independent vocabulary + extensions

• toolkit independent vocablary + extensions

• inter-model relationship mappings

• conceptualizes transformation rules + extensions

• UML class diagram


SemanticsVocal Concrete expansion

• Vocal Containers:– vocalGroup

– vocalForm

– vocalMenu

– vocalConfirmation

• Relationships:– vocalTransition– vocalAdjacency– vocalContainment– synchronization

• EventTypes: – error/help/noInput/ noMatch

• Vocal Individual Components:– vocalOutput:

• vocalFeedback• vocalPrompt• vocalMenuItem• audio

– vocalInput– grammar/part/item– vocalNavigation/submit– connect– record– vocalVar/setVar/resetVar– if/else/elseif– break/exit


SyntaxUsiXML language

• USer Interface eXtensible Markup Language

• Particular motivations:– Structured according to Cameleon Reference Framework– Ensures the independence of modality (Req.4. Ability to model modality-

independent UI )– Logically structures the transformation steps (Req. 9. Method

explicitness)– Flexibility for adding/deleting sub-steps (Req. 10. Method extendibility)– Transformational approach with transformations expressed in the same

formalism– Supported by tools processing its format (Req.11. Machine

processability of involved models)


Language: Stylistics

vocalGroup

A dashed rectangle to suggest the containment purposes and a callout symbol to indicate the vocal character.

vocalForm

A dashed rectagle to suggest the containment intentions, a user and a system icon next to a callout symbol.

vocalMenu

As it is a container that enables to select among different options and by analogy with the menu provided by the graphical toolkits the representation is composed of a blue dashed oval to suggest the containment purpouses, an overlaying callout symbol to indicate the vocal aspects and yellow ovals to indicate the vocal options.

vocalPrompt

A system icon and a callout symbol containing a question sign.

vocalInput

A user icon next to a callout symbol and a system icon.

?


Outline

• Context



• Methodology:


– Method

– Tool support

• Validation

– External

– Internal

• Conclusion

• Future work


Design space

Design option 1

Design option 2

Design option 3

Designoption 4

Design option 5

Designoption 6

Design option 7

Designoption n

Value 14

Value 13

Values 12

Value 11

Value 22

Value 21

Value 31 Value 32

Value 41

Value 42

Value 51

Value 52

Value n2

Value n3

Value n1

Value 63

Value 61

Value 62

Value 73 Value 72 Value 71


Design space (cont’d)

• Advantages– Clarifies the development process: structured, less design effort, consistent

results

– Every piece of development is reflected in a design option

– Abstractions covered by a software that do not require any further interpretation effort

• Rationale– Intrinsic: descriptive, comparative, generative

– Extrinsic:

• implementation and tool-independent: useful for any designer

• explicit guidance: flexibility to set the CARE properties

• simplifies the design decision: reduces the complexity of the design space


Design space (cont’d)

A structured combination of modality-independent design options having assigned a finite set of design option values that support the

stakeholder’s design decisions during the development life cycle of multimodal user interfaces

Design options for UIs

Sub-task presentation

Sub-task navigation

Navigation type

Control type

Sub-task guidance

Support for default value and unit

Answer cardinality

Answer order

Confirmation answer

Navigation and controltype

Sub-task triggering

Prompt

Input

Simpleoutput

Immediate feedback

Guidance

Definition, Rationale, Values



• Specifies the way in which the system is presenting the sub-tasks to the user.


Combined

One at once Many at once All at once

separated

extended task list

reducedtasklist

tabbedlist

singleexpansion

list

multipleexpansion

list

separatedlist

groupedlist

bulletedlist

orderedlist


Sub-task 1

Sub-task 2

Sub-task 3

Sub-task 1

Sub-task 2

Sub-task 3

Sub-task 1

Sub-task 2 Sub-task 3Sub-task 1

Sub-task 1

Sub-task 2

Sub-task 3Sub-task 1

Sub-task 2

Sub-task 3

Sub-task 1

Sub-task 3

Sub-task 2

Sub-task 1

Sub-task 2

Sub-task 3

Sub-task 1

Sub-task 2

Sub-task 3

1. Sub-task 1

2. Sub-task 2

3. Sub-task 3

Co

ncr

etiz

atio

in in

gra

ph

ical

ob

ject

sC

on

cret

izat

ion

in v

oca

l ob

ject

s

Sub-task 1

Sub-task 2

Sub-task 3

“Select an option”

?

“Which are my options?”

Sub-task 1

Sub-task 2

Sub-task 3

?“Sub-task 1”

?“Sub-task 2”

?“Sub-task 3”

Audio

Audio

Audio

Sub-task 1

Sub-task 2

Sub-task 3

?“Sub-task 1”

?“Sub-task 2”

?“Sub-task 3”

Audio (beep)

Audio (beep)

Audio (beep)

Sub-task 1

Sub-task 2

Sub-task 3

?“Sub-task 1”

?“Sub-task 2”

?“Sub-task 3”

Audio (“ 1 ”)

Audio (“ 2 ”)

Audio(“ 3 ”)

Sub-task 1

Sub-task 2

Sub-task 3

separatedextended

task list

reducedtasklist

tabbedlist

singleexpansion

list

multipleexpansion

list

separatedlist

groupedlist

bulletedlist

orderedlist

Sub-task 1

Sub-task 2

Sub-task 3

Sub-task 1

Sub-task 2


Sub-task 2

Sub-task 3


Design options for multimodal text field

• Prompting: multimodal (R)• Input: multimodal (E)• Immediate feedback: multimodal (R)• Guidance:

– Input: iconic (A)

– Immediate feedback: iconic (A)

Prompt: multimodal

Please specify your name

(vocal +graphical)1

Input: multimodal

Juan Gonzalez

2

(vocal +graphical)

Juan Gonzalez

Feedback:multimodal

Your answer was: Juan Gonzalez

3

(graphical + vocal)

Guidance for input: iconic

Guidance for feedback: iconic


Design Rationale Approach

• Supports different alternative solutions for a design issue by providing a justification and enabling its evaluation

• Advantages:– detects the concistency and completeness issues in the early design phases– clarifies the reasons for a particular design solution and enables to argue it

• Example: Dream/Team ([Laca05 ])– Team notation– Dream tool


• Existing shortcomings:– repeated common

parts– huge set of rules

• Contributions:– colored graph (rules) +

operations

• Benefits:– reduced number of

rules (250 -> 130)– high scalability

Colored ontology


abstractContainment

isComposedOf

isReifiedBy

vocalContainments

vocalContainments

vocalContainment

isReifiedBy

abstractContainment

isComposedOfisReifiedBy

abstractContainment

isReifiedBy

graphicalContainment

isComposedOf


isReifiedBy


abstractContainment

isReifiedBy isComposedOf

isReifiedBy

NAC

LHS

RHS

Rule for GUI Rule for VUI

isReifiedBy

NAC

LHS

RHS

• merging

Operations over colored rules:

• splitting

Rule for MMUI


Transformation rule catalog

• Complete and systematic arrangement of rules (130) into a structured catalog:– Transformation rules supporting design options

– Additional transformation rules


Sub-task 1

Sub-task 2

Sub-task 3

Sub-task 1

Sub-task 2

Sub-task 3

Sub-task 1

Sub-task 2 Sub-task 3Sub-task 1

Sub-task 1

Sub-task 2


Sub-task 2

Sub-task 3

Sub-task 1

Sub-task 3

Sub-task 2

Sub-task 1

Sub-task 2

Sub-task 3

Sub-task 1

Sub-task 2

Sub-task 3

1. Sub-task 1

2. Sub-task 2

3. Sub-task 3

Co

ncr

etiz

atio

in in

gra

ph

ical

ob

ject

sC

on

cret

izat

ion

in v

oca

l ob

ject

s

Sub-task 1

Sub-task 2

Sub-task 3

Sub-task 1

Sub-task 2

Sub-task 3

“Select an option”

?

“Which are my options?”

Sub-task 1

Sub-task 2

Sub-task 3

?“Sub-task 1”

?“Sub-task 2”

?“Sub-task 3”

Audio

Audio

Audio

Sub-task 1

Sub-task 2

Sub-task 3

?“Sub-task 1”

?“Sub-task 2”

?“Sub-task 3”

Audio (beep)

Audio (beep)

Audio (beep)

Sub-task 1

Sub-task 2

Sub-task 3

?“Sub-task 1”

?“Sub-task 2”

?“Sub-task 3”

Audio (“ 1 ”)

Audio (“ 2 ”)

Audio(“ 3 ”)

Sub-task 1

Sub-task 2

Sub-task 3

separatedextended

task list

reducedtasklist

tabbedlist

singleexpansion

list

multipleexpansion

list

separatedlist

groupedlist

bulletedlist

orderedlist

Sub-task 1

Sub-task 2


Sub-task 2

Sub-task 3

R2

R2

R5,R6

R5,R6

R7,R8

R7,R8R9,R10

R11,R12

R11,R12

R9,R10 R13,R14

R15,R16

R15,R16

R17,R18

R17,R18

R19,R20

R19,R20

R21,R22

R21,R22


Steps and sub-steps of the transformational approach

• Step 1: Task & Domain Models

• Step 2: From Task & Domain to AUI

• Step 3: From AUI

to CUI

• Step 4: From CUI

to FUI

sub-task presentationnavigation typecontrol type


Outline

• Context



• Methodology:


– Method

– Tool support

• Validation

– External

– Internal

• Conclusion

• Future work


TransformiXML tool

2

Step 2

TransformiXML tool

2

Step 3

VoiceXML Generator tool

4

GrafiXML tool

3XHTML+Voice Generator tool

5

Tool support MultimodaliXML

Task & Domain Models

Abstract UI Model

Concrete UI Model

(graphical)

Concrete UI Model(vocal)

Concrete UI Model

(multimodal)

XHTML+Voicecode

IdealXML toolStep 1

Step 4

VoiceXML code

Multimodal BrowserVoiceXML Browser

Web browser

XHTML code

1


Input: Graphical (A)Input: Vocal (A)Input: Multimodal


Outline

• Context



• Methodology:


– Method

– Tool support

• Validation

– External

– Internal

• Conclusion

• Future work


External validation: case studies

• Feasability of the approach - 3 case studies ( ≠ levels of complexity and ≠ design options):– 2 web-form: virtual polling application (low complexity), car rental application

(medium complexity)– 1 non web-form: map browsing application

Abstract User Interface

Task and Domain

Concrete User Interface(multimodal)

Concrete User Interface(vocal)

Concrete User Interface(graphical)

Final User Interface(graphical)

Final User Interface(vocal)

Final User Interface(multimodal)


Validation: Empirical assessment

• Assesment of 3 applications: – Session I (2 web-forms), Session II (1 non web-form)– Relative usability level of interaction modalities

• Input design option: G, V, MM (equivalence)

• Evaluation measures:– task completion time– error rate– interaction preference– task procentage completion– learning time– n° of mouse clicks


Results: Task completion mean time

• Web-form: – Efficiency (G > MM > V)– Experienced faster then non-experienced

• experience has a significant influence for G (t-Test analysis p=0.0339)• higher dispersion for V and MM


Results: Error rate

• Higher error rate for V and MM (synchronization and pronunciation errors )• Correlation (Pearson correlation analysis):

– task completion time (web form): V(Pearson= 0.75), true MM (Pearson= 0.73)– number of mouse clicks (web form): V(Pearson= 0.77), true MM (Pearson= 0.63)– task percentage completion (non-web form): V(Pearson= -0.72), MM (Pearson= -0.48)


Results: Interaction modality preference

• Satisfaction: strong preference for MM all commands conveyed MM [Oviatt, 1997]– web-form: G ≈ MM > V– non web-form: G > V > MM– mix of monomodal (radio buttons, list box) and multimodal (check boxes with numerous items /

combo boxes with items to select at the end) [Oviatt, 1999]


Outline

• Context



• Methodology:


– Method

– Tool support

• Validation

– External

– Internal

• Conclusion

• Future work


Internal validation: Requirements level of support

Req 1. Support for MM I/O

Req 2. Separation of modalities

Req 3. Support for CARE properties in I/O

Req 4. Ability of modeling a UI independent of any modality

Req 5. Extendibility to new modalities

Req 6. Ontology homogeneity

Req 7. Human readability

Req 8. Approach based on design space

Req 9. Method explicitness

Req 10. Method extendibility

Req 11. Machine processability of involved models

Req 12. Support for tool interoperability

Mo

del

ing

Met

ho

dT

oo

l


Outline

• Context



• Methodology:


– Method

– Tool support

• Validation– External– Internal

• Conclusion

• Future work


ConclusionMethodology barriers

• Ontological– Lack of model expressiveness

• Method– Difficulty in defining an orthogonal design space

– Difficulty in finding an appropriate level of generalization when defining rules

– Difficulty in managing large sets of transformation rules:

• High reusability

• Low reusability

• Tool– Low interoperability: high number of software modules


Conclusion (cont’d): Main Contribution

• Ontological contribution– Expanded task model to support the requirements of

MM UIs– Expanded vocal ontology– Multimodal instruction structure– Synchronization between the modalities– Formalization according to UsiXML syntax– Stylistics of vocal ontology



• Method contribution– Design space

• 16 design options: low treshold, high ceiling, wide walls

– Expanded model-to-model transformation

• colored concepts → colored graph → colored rules

• operations over colored transformation rules

– Transformation rule catalog: identify rules supporting the design options

– 4 transformational development steps: identify the involved design options

– Introduction of a new methodogical sub-step: synchronization of CICs



• Tool contribution– 5 interoperable software modules– Address specific steps of the development process– Proof of concept reasons


Outline

• Context



• Methodology:


– Method

– Tool support

• Validation

– External

– Internal

• Conclusion

• Future work


Future work

• Extend the methodology to support development of context-aware systems: dynamic migration from one modality to another based on a set of rules

• Design space improvement: reduce dependent dimensions, introduce new dimensions and new values

• Knowledge base of inference rules: recommandation of suitable design options

• New interaction modalities: – ontological extension– method extension– tool support extension


Future work (cont’d)

• Design space-based tool


Thank you !

Questions ?

1 phd public defense, 25 june 2008 a methodology for developing multimodal user interfaces of...

Documents

fast interaction

modalityindependent

set of models

domain models

human interaction bchihttp

ui independent

modelbased approach

ui development