Mobile Multimodal Interaction for Older and Younger Users: Exploring Differences and Similarities

Gianluca Schiavo, Michela Ferron, Ornella Mich & Nadia ManaFBK - Fondazione Bruno Kessler, Trento, Italy

{schiavo,ferron,mich,mana}@fbk.eu

RESULTS FROM A VIDEO ANALYSIS

A total of 1h 35’ video footage of task sessions was analyzed. Each of the 240 observed interactions (8 gestures x 30 participants) was annotated, considering: • interaction type (gesture-only, speech-only or multimodal – if both gesture and

speech input were performed) and temporal occurrence (simultaneously, gesture-first, vocal-first);

• time taken to complete the task;

• words used for the voice commands; and

• gesture features (qualitative description of how physical commands were executed).

MULTIMODAL INTERACTION FOR OLDER ADULTS

Multimodality (such as the combination of speech-based and gesture-based interaction) can make human-computer interaction easier and more natural. However, there are still some limitations to overcome to make multimodality also feasible to older adults.

COMBINING MID-AIR GESTURE AND VOCAL COMMANDS

The ECOMODE (Event-Driven Compressive Vision for Multimodal Interaction with Mobile Devices) project, funded by the EU H2020 ICT22 call for Multimodal and Computer Interaction, aims to tackle limitations of current automatic speech and mid-air recognition systems.

The project exploits the recent EDC (Event Driven Compressive) paradigm to realize a new generation of low-power multimodal human-computer interfaces for mobile devices.

INVESTIGATING DIFFERENCES AND SIMILARITIES BETWEEN DIFFERENT AGE GROUPSWe designed a Wizard of Oz study where participants of different ages were asked to take photographs with a tablet device using multimodal interaction (mid-air and vocal commands).

Young adultsMean age: 30 (3.7)(range 25-35 years)

Middle-age adultsMean age: 51 (2.9)(range 45-55 years)

Older adultsMean age: 69 (3.2) (range 65-75 years)

All age groups, including older participants, could combine modalities when interacting with the tablet device. However, we observed a tendency in older participants toward performing gestures instead of voice commands.

Older adults tend to perform more gesture-only interactions, and, when using multimodal commands, they gave more gesture-first commands compared to the other groups.

Regarding voice interaction, younger participants provided a higher number of correct voice commands compared to the other groups. However, even if they often forgot the assigned command, older and middle-aged adults used meaningful synonyms instead

COMMAND SET

EXECUTION TIME

www.ecomode-project.eu

TYPE OF INTERACTION

ThisworkissupportedbytheEUHORIZONprojectECOMODE(Event-Driven

CompressiveVisionforMultimodalInteractionwithMobileDevices)under

GrantAgreement644096.