Enabling enactive interaction in virtualized experiences

Stefano Tubaro and Augusto SartiDEI – Politecnico di Milano, Italy

Nome relatore

How do we enable virtualized enaction with How do we enable virtualized enaction with incomplete sensory stimulation?incomplete sensory stimulation?

Gaining enactive knowledge from a mediated experience

Enactive Knowledge:information gained through perception-action interaction

• usually refers to an environment or a process (task)• inherently multimodal as it requires the

coordination of the various senses• knowledge acquired in a mediated form

is usually in symbolic or iconic form• enactive knowledge is based on the

active use of our senses• direct and intuitive (based on experiece)• often acquired without being aware of it

• Research on enaction is mostly focused on developing enactive multimodal interfaces

Nome relatore

Realistic or plausible?

• Aside from videogames, where plausibility plays

a dominant role, enactive knowledge acquisition

is usually aimed at learning about reality.

This is why mediated enaction usually relies on

realism and sensorial transparency, which makes

its applicability rather limited

• More promising (for massive use) is to accept virtualized enaction to be

based on reduced sensory perception and develop

• methodologies that allow the user to develop reliable

mappings between real and virtual enactive knowledge

• applications that are able to guess whether such mappings are

correctly established in a closed-loop fashion

Nome relatore

An example: the virtual dressing room

The e-market of articles of clothing is slumping because it offers no virtual try-on at home

The impact of applications such as virtual mirrors has been modest because of high cost; modestrealism; and lack of multimodal interaction

Research is needed for developing virtual try-on applications based on physically and perceptually plausible interactional rendering with reduced sensory perception (audio-visual), and limited realism, as long as we know how to• render what is perceptually relevant for a virtual experience that

can be instinctly remapped onto real life by the user • correctly map haptic interaction onto audio-visual interaction

(sensory substitution through cross-modal processing)• correctly map real interaction onto virtual interaction exploiting the

user’s experiential knowledge

Nome relatore

An example: the virtual dressing room

The mappings between experiential memory and virtual experience need

be established in a closed-loop fashion by an intelligent HMI, which is

able to read metacommunication cues carried by posture, gestures, gait,

facial expression, voice prosody, etc.

The HMI must become enactiveThe HMI must become enactive and learn on the fly whether the and learn on the fly whether the

virtual experience is being virtual experience is being effectively remapped onto the effectively remapped onto the

real worldreal world

Nome relatore

How do we enable enactive virtualized interpersonal How do we enable enactive virtualized interpersonal interaction with incomplete sensory perception? interaction with incomplete sensory perception?

Next step: enactive interpersonal interaction in virtual spaces

Interpersonal interaction is based on a closed-loop exchange of• symbolic or iconic information: Semantic information exchanged through

speech, images, intentional gestures, display of clothes or specific objects, ...

• meta-communication cues: body language (face expression, gestures, body posture, gait, etc.), voice prosody ad expression, tactile cues ...

Interactional meta-communication is• purely enactive• inherently multimodal• instinctive (reduced awareness)

Nome relatore

Issues in virtualized interactional experiences

Two options for enactive interaction• Interaction conveyed through signals

• metainformation to be captured, transferred and rendered as transparently and completely as possible

• e.g. video and audio 3D• problems: lack of effective and inexpensive

haptic sensors/display• Interaction mediated by models

• e.g. 3D body models (avatars), interactional sounds, etc.

• needs: compensate for incomplete and imperfect sensing and displaying technology with intelligent sensors, affective and cross-modal processing, perceptual multimodal displays, etc.

Again, Again, the HMI must become enactivethe HMI must become enactive and be aware of and be aware of the impact of the virtual experience onto the usersthe impact of the virtual experience onto the users

Nome relatore

In perspective

• Natural human-machine interaction • Multimodal interaction at multiple

levels of abstracion: from physical to emotional

• Enactive machines• Machines that learn from the

interaction with humans• Machines that learn from multimodal

observations of human-to-human interaction

Point of convergence between• Research on intelligent sensing

• computer vision, audio and acoustic analysis and processing, pattern recognition, multimodal fusion, ...

• Research on multimodal sensors and displays• Research on behavioural psychology

(social learning theory), and sociology