IST-1999-20410

MEGA Multisensory Expressive Gesture Applications

Concertation MeetingBrussels, 3-4 October 2000

Antonio Camurri (DIST-University of Genoa)MEGA Project Coordinator

Summary

• MEGA: Concepts, Objectives and Mission

• The Consortium

• Project Overview: Workplan, WPs, and Deliverables

MEGA: Concepts, Objectives and Mission

• Concepts:

– Multisensory

– Expressive (emotion, kansei, affect)

– Gesture

– Applications

MEGA: Concepts, Objectives and Mission

• Objectives:– modelling and communication of expressive and

emotional content in non-verbal interaction by multi-sensory interfaces

– design of novel paradigms of interaction – analysis, synthesis and communication of

expressive content in networked environments

– implementation of the MEGA System Environment– development of artistic public events showing the

usefulness, flexibility and power of the MEGA System Environment

MEGA: Concepts, Objectives and Mission

• Mission:– Consolidating a third original European route in

affective computing (USA) and KANSEI (Japan).– Raise the competitiveness of European Industry on

• Digital Music Instruments • Entertainment and Edutainment• Multimedia: e.g. VEs, e-commerce, intelligent

interfaces, ...

by building the MEGA System Environment and showing its effectiveness in public events.

Participant no.

Participant name Participant short name

Country Status*

1 DIST - University of Genoa Lab. on Musical Informatics

DIST I C

2 DEI - University of Padova DEI I P

3 IPEM – University of Gent IPEM B P

4 KTH - Royal Institute of Technology, Stockholm

KTH S P

5 Uppsala University, Department of Psychology

UUP S P

6 Telenor TLN NO P

7 Generalmusic GEM I A

8 Consorzio Pisa Ricerche CPR I P

9 Eidomedia EM I A

Participants

Workplan: Workpackages

    

• WP1 Project management• WP2 Requirements • WP3 Analysis in human movement and gesture • WP4 Analysis in musical gesture performance • WP5 Synthesis in visual media and movement • WP6 Synthesis in music performance • WP7 MEGA System Environment • WP8 MEGA Multimedia Performance • WP9 Dissemination and implementation

Workplan: gantt

Year 1 Year 2 Year 3WP Title1 2 3 4 5 6 7 8 9 10 11 12 1314 15 1617 18 19 2021 21 2324 2526 27 2829 30 31 3233 34 3536

WP1 Project management * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *WP2 Requirements * * * * * * * * * *WP3 Analysis in movement * * * * * * * * * * * * * * * * * * * * *WP4 Analysis in music * * * * * * * * * * * * * * * * * * * * *WP5 Synth. in movem. and vis. * * * * * * * * * * * * * * * *WP6 Synth. in music * * * * * * * * * * * * * * * * * * *WP7 MEGA System Environ. * * * * * * * * * * * * * * * * * * * * * * * *WP8 MEGA Multim. Perform. * * * * * * * * * * * * * * * * *WP9 Dissemin. and Implement. * * * * * * * * * * * * * * *

MILESTONES * * *

- Spiral approach- Three phases, each concluded at a project milestone (13, 25, 36 months).

WP2 Requirements • WP Leader: IPEM

• State of the art, share of results among partners, Requirements and setup for the research, the MEGA System Environment and the MEGA Multimedia Performance products.

• Definition of a standard communication protocol for expressive data. The protocol is defined in this WP and is implemented in the WP7 (MEGA System Environment).

• Contributes from all the partners.

• D3 (month 6) Requirements

• D9 (month 14) First requirement revision

• D12 (month 26) Final requirements

WP3-6: Research on analysis and synthesis of expressivity

• WP3 Analysis in human movement and gesture (DIST) • WP4 Analysis in musical gesture performance (DEI)• WP5 Synthesis in visual media and movement (EM)• WP6 Synthesis in music performance (KTH)• Results: a selection of techniques and algorithms, integrated

as s/w modules in the MEGA System Environment (WP7). These developed tools and modules will be continuously integrated and updated in the EyesWeb architecture, i.e., WP3-6 will deliver tested library modules and associated manual pages using the standard adopted in the project

• Deliverables: Reports (D5, D13), S/w tools (D6, D14)

WP3 Analysis in human movement and gesture

• WP Leader: DIST

• To extract expressive content from movement patterns.

• To evaluate and compare the results and performance of the obtained methods and algorithms (DIST and KTH) with the results obtained by psychological analysis of movement (UUP) with selected groups of users, both non-expert and expert (dancers, choreographers).

• To develop a project reference archive of microdances and movement patterns (DIST, KTH, UUP), for both psychological analysis and automated analysis, using the standard adopted in the project (audiovisual, mocap data).

WP4 Analysis in musical gesture performance

• WP Leader: DEI Analysis of the different cues characterising an expressive

musical performance (UUP, DEI); Analysis of the correlation between cues and the instrument

played by the performer (DEI); Development of a model to extract automatically an

expressive content of a musical performance (IPEM); Development of tools implementing the model (DEI,

IPEM); Evaluation and test of the effectiveness of the model by

means of performances analysis with the implemented tools (KTH, UUP);

WP5 Synthesis in visual media and movement

• WP Leader: EM

• Techniques inspired to the audiovisual language in cinema (EM) and their utilisation in virtual environments (TNR) and in visual and robot-human interaction (DIST).

• Development of models and methods to synthesise expressive visual media outputs (EM).

• Expressive synthesis of movements of synthetic characters and expressive modulation of VEs and Mixed Reality envs (TNR), depending on the expressive communication goals and content.

• Expressive movement of small robots in concrete apps on robot-human interaction and mobile scenery (DIST).

• Integration with Dovre VE architecture (TNR).

WP6 Synthesis in music performance

• WP Leader: KTH

• Development of tools both in MIDI and in digital audio format (KTH, DEI), to be used for the project products (WP7-8).

• Synthesis on MIDI format based on Director Musices (KTH)

• Adapting and integrating audio electronics for real-time expressive processing (GEM)

WP7 MEGA System Environment

• WP Leader: DIST

• Deliverable D4: EyesWeb software release, including tutorial and reference manuals (month 6)

• Deliverable D7: 1st version of MEGA Sys Env. (12)

• Deliverable D10: 2nd version of the MEGA Sys Env. (24)

• Deliverable D15: Final product - MEGA Sys Env (35)

• DIST: EyesWeb and architecture CPR: Wireless Sensors

• TNR: Dovre IPEM: Mapping Strategies, Eval

• EM: Wireless Sensors

MEGA System Environment (WP7)• Main components:

- the EyesWeb software architecture (from DIST),- the (open) libraries of software modules concerning

analysis and synthesis of expressive gesture (WP3-6), - the implementation of the protocol for expressive

communication (WP2),- the h/w electronics and sensor systems for wireless non-

invasive real-time motion capture (adapted from existing systems by CPR, DIST, EM), integration of commercial sensor systems (e.g., radio baton) and audio electronics systems (by GEM).

WP8 MEGA Multimedia Performance

• WP Leader: IPEM• Main components:- the specification of the artistic use of the MEGA System

Environment (WP7) in terms of the expressive content analysis and synthesis (IPEM, DEI);

- the support to specific multimedia artistic events, based on the MEGA System Environment (DIST, KTH);

- the evaluation of the MEGA System Environment with respect to audience acceptance, effectiveness, usability, and flexibility (IPEM).

WP9 Dissemination and implementation• WP Leader: TNR• To set up the Liason Board• To disseminate products (WP7-8) and research results

(Workshops, book, Frankfurt and other exhibits, etc). • Internet-based dissemination and exploitation strategy by a

free distribution to users of the software (under certain constraints and on-line contract forms),

• To enable external users to develop new “plugins” of the MEGA System Environment.

• Evaluate (UUP) the results from the project from WP7-8 and the internet dissemination strategy, with respect to the planned outputs.

• D2: Dissemination and Use plan (6)

• D17: Technological Implementation Plan (36)

Dissemination (WP8 and WP9)• via public performances (WP8)

• An internet site providing the software and support to users (WP9):– s/w: the full MEGA System Environment, incl. a

Wizard enabling users to develop new system plugins– library of plugins deriving from WP3-6 (research)– full documentation in the standard adopted

• Workshops, special tracks and sessions at IEEE CS, ACM Conf, at CEC Workshops, at Frankfurt Messe 2001 and/or 2002 (WP9)

Deliverables list

Del.No.

Del. Name WP no. Leadparticipant

Estimateperson-months

Del. Type* Security** Delivery

(proj.Month)

D1 Project Presentation WP9 DEI 1 Report Pub. 6

D2 Dissemination anduse plan

WP9 IPEM 8 Report Int. 6

D3 Requirements WP2 KTH 7 Report Int. 6

D4 EyesWeb s/w releaseincl. tutorial andreference manuals

WP7 DIST 4 Prototype Int. 6

D5 Analysis andsynthesis ofexpressiveness:techniques andmapping strategies

WP3WP4WP5WP6

UUP 15 Report Int. 12

D6 Tools for automaticanalysis andsynthesis ofexpressive content inmusic and danceperformance

WP3WP4WP5WP6

KTH 25 Demonstration Rest.(***)

12

D7 First version of theMEGA Sys. Env.

WP7 DIST 10 Prototype Pub. 12

D8 Report and s/wpatches from the firstmultimedia event

WP8 UUP 9.5 Report Int. 12

D9 First revision ofrequirements

WP2 DIST 5 Report Int. 14

D10 2nd version of theMEGA Sys. Env.

WP7 DIST 11 Prototype Pub. 24

D11 Report and s/wpatches from thesecond multimediaevent

WP8 IPEM 12 Report Int. 25

D12 Final requirements WP2 IPEM 5 Report Int. 26

D13 Final report onanalysis andsynthesis ofexpressiveness:models, techniques,and integrationissues

WP3WP4WP5WP6

KTH 34 Report Int. 31

D14 Tools for automaticanalysis of theexpressive content inmusic and danceperformance

WP3WP4WP5WP6

DEI 45.5 Prototype Rest.(***)

31

D15 Final product of theMEGA Sys. Env.

WP7 DIST 9 Prototype Pub. 35

D16 Report and s/wpatches from thethird multimediaevent

WP8 IPEM 16 Report Int. 36

D17 TechnologicalImplementation plan

WP9 TNR 10 Report Int. 36