het collectief centrum van de Belgische technologische industrie

Tom Tourwé & Elena Tsiporkova

Human-Machine Interaction Research @ Sirris

Agile development Variability Semantic technologies

Web 2.0 Decision support Innovation process support

Intelligent information retrieval Knowledge extraction

Data integration Business intelligence

Human machine interaction

Software & ICT

Non-profit & industry-owned

collective centre of the technology industry

HMI research @ Sirris

Multimodal

Interusable, multi-platform &

multi-device

Context-sensitive

Pro-active decision support

2 large industry-

driven European

R&D projects

Optimise the choices available

Pro-actively push relevant

information Present information in a user-specifc

and context-aware way

Provide the right dosage of

information at the right time Implement intention-aware adaptive

automation (trading of control)

Keep the user in control

ASTUTE

Pro-active decision support in data-intensive environments

• A decentralized solution is targeted where:

• the emergency workers are equipped with portable or embedded

devices capable of receiving, sending, and visualizing

dispatching events and context information such as annotated

geographical maps

• the emergency workers collaborate within their task force and

between different units backed by a central dispatching room

• a map-centric user interface provides the field workers with a

clear and up-to-date overview of all events.

Smart Emergency Dispatching

Dispatching room

Field workers

Adaptive Multimodal Interfaces

• The main goal of the user interface design task is to enable

applications to adapt to changing situational contexts, e.g.

• to send an alert to the commander when one of his firemen approaches

toxic substances in the building,

• through an optimal output modality, taking into account environmental

conditions such as noise and lighting level

• as well as an appropriate input modality, to allow the commander to

immediately take the appropriate action

Multimodal Interface Design

• The multimodal user interface design is supposed to provide

solutions to design problems such as:

• When to use certain modality?

• How to combine multiple modalities?

• How to adapt the modality according to its context of use?

• Formal principles and guidelines(1)

:

• design for the broadest range of users and contexts of use

• address privacy and security

• maximise human cognitive and physical abilities

• integrate modalities in a manner compatible with user preference,

context, and system functionality

• …

(1) L.M. Reeves, J. Lai, J.A. Larson, S. Oviatt, TS Balaji, S. Buisine, P. Collings, P. Cohen, B. Kraal, J.C. Martin, et

al. Guidelines for multimodal user interface design. Communications of the ACM, 47(1):57–59, 2004.

Theory (formal guidelines) vs. practice

• Different experts might approach the same interface design tasks in

different ways based on personal expertise, background and intuition

• Guidelines resulting from research do not capture the considerable

practical experience and expert knowledge that interface designers

rely on during their daily activities

• Existing formal guidelines mostly focus on high-level design

objectives

• are not specific to multi-modality

• do not include justification for the made recommendations

• lack information about how to move from guidelines to a concrete

implementation

Need for Knowledge Capture and Modelling

• Capture and document design best practices

• Can be used as a reference while designing, reducing time and

increasing quality of the design solutions

• Keep track of knowledge and expertise along projects

• Can be used for communication and education of team

(developers, designers, etc.)

• To be used and reused, going towards standardization

Ontology-driven Knowledge Modelling:

Levels of Modelling Abstraction

Application knowledge

Domain knowledge

Expert knowledge

Scenario-specific types of users, activities, tasks and

concrete working environment

HCI community: design guidelines and best

practices

Core domain concepts: factual information on users,

applications & devices

Capturing Domain Knowledge

Domain knowledge

Core domain concepts: factual information on users,

applications & devices

Domain knowledge is described via an ontology, a

formal representation of knowledge by a set of key

domain concepts and the relationships between

those concepts.

Modeling Design Guidelines

Expert knowledge

HCI community: design guidelines and best

practices

Design guidelines (expert

knowledge) are captured

via the Semantic Web Rule

Language (SWRL)

Application(?application), NoisyLocation(?location), used_in(?application, ?location) ->

cannot_use_modality(?application, audio_output),

cannot_use_modality(?application, voice_input)

Modelling Application Knowledge

Application knowledge

Scenario-specific types of users, activities, tasks and

concrete working environment

The original core ontology is complemented

and extended by creating an ontology with

relevant application-specific knowledge:

• types of users involved

fire fighters, fire commanders, fire station

dispatchers, air sampling collectors, emergency

communication managers, medical experts, company

employees

• their activities and tasks

fire fighting, locating water supplies, rescuing

company employees that could not leave a building,

logging relevant information, defining security

perimeters in the presence of dangerous substances

• working environment they are located in

an administrative office where the fire started, a

storage facility with smoke and high temperatures,

outside a building where dangerous substances

might be being spread in the air, inside a medicalised

tent.

Summary

• A semantic modelling framework

• allowing to capture the domain and expert knowledge available within the

Human-Computer Interaction (HCI) community

• supporting HCI designers in selecting the most appropriate (combination

of) modalities during their daily design tasks

• enabling the high-level concrete description of multimodal HMI design

patterns

• Work in progress

Smarcos

Smart composite human-computer interfaces

• Smarcos aims to ensure interusability of interconnected embedded

systems

• UI designs of new applications and services must accommodate

various devices, personal preferences, and adaptivity to different use

contexts to make the user experience both utilitarian and pleasant

Our contribution

Context-aware feedback delivery

• A technological framework for enabling multi-platform &

multi-device systems to adapt feedback delivery to

situational contexts

• send the right information

• at the right moment in time

• through a device that offers the optimal output modality for that

information

• as well as an appropriate input modality to allow the user to react

Requirements & challenges

• Measure the availability of the user for feedback (right moment)

• taking into account time of the day, current and future activities, social &

professional environment, …

• in order to determine whether to deliver feedback right away, or time-shift

it to a later moment in time

• Determine the available devices (right device)

• taking into account surrounding active devices, device characteristics,

available input/output modalities, …

• in order to select the most appropriate device for delivering feedback

taking into account its capabilities

• Adapt the message delivery (right information)

• taking into account the availability of the user, chosen modality, message

form and output device capabilities, …

• in order to tailor the content and functionality toward the selected device

capabilities

A declarative rule-based solution

• Determining timing, device and message format requires

modelling a complex piece of logic, i.e.

• under which combination of the many different context

parameters is a particular solutions prefered?

• a user’s current & future activities, and his location

• his physical & social environment (noise, light, temperature,

presences of colleagues, …)

• the devices surrounding him, their characteristics and capabilities

• the message’s urgency level

• Requires a reactive system that continuously monitors &

evaluates the context

• extend & complement the traditional imperative paradigm (if-

then-else) with a declarative rule-based paradigm

Attentive Personal Systems

Basic example scenario

The Smarcos system detects that a user has not taken his

medication and immediately sends an urgent pill reminder. The

user is currently commuting to the office and only carries his

mobile phone. The system decides to send a text message. It

knows that the user normally arrives after half an hour at his

desk, and when he starts using his computer, he gets another

reminder to which he needs to react.

When is a message urgent?

Message urgency = HIGH if

> The message needs to be delivered to the user in 10 minutes

or less

When do we disturb a user?

User availability = LOW if

> The user is in a meeting at the office

> The user is on the go (e.g. commuting to work)

How do we deliver an urgent message to a

user with low availability?

> send message to mobile phone immediately

> send reminder message to other device when available later

When do we deliver scheduled messages?

• The users availability is not LOW

• A device becomes available

When does a device become available?

Device = AVAILABLE if

• It is in the same location as the user

> It is a mobile device (always available)

Summary

• A declarative rule-based framework that

• continuously monitors & evaluates the context, and acts

correspondingly according to the defined strategies

• decouples complex logic from other application concerns, which

leads to easier maintainability and understandability

• can be evolved in a flexible and incremental way

• Integrated into a more complex technology stack to enable

context-aware feedback delivery

• sensor layer, interconnectivity layer, context interpretation layer,

…

More information

• Web links

• http://www.astute-project.eu

• http://www.smarcos-project.eu

• Contact details

• tom.tourwe@sirris.be

• elena.tsiporkova@sirris.be