eNTERFACE 08 Project 1“MultiParty Communication with a

Tour Guide ECA”

Mid-term presentation

August 19th, 2008

Outline

• Project Summary (Repeating the lesson ;) )

• System configuration

• Work in progress

• Speech Regonition

• Nonverbal Input Handling

• Input Understanding

• Dialogue Management

• Character Animation Player

• Conclusion

Summary

We aim to create an ECA based system which will interract with two users by:

• Capturing their behaviors (speech, nonverbal behaviors)

• Combining and handling input data

• Generating and displaying response to these behaviors in the real time

• Input

• Speech recognition

• Nonverbal behavior detection (Face detection, Facial orientiation, Hand raising)

• Central Part

• Input Understanding, Dialogue Management

• Output

• ECA animation player

• Platform to provide communication between the components

System Configuration

Speech Recognition

• Two SR components will be running on different machines

• Keywords will be dynamically changed according the conversational state

• No significant overlapping between microphones (distance ~ 1.5 m)

• Implementation in progress:

• Keywords detection with confidence score and speech duration by using Loquendo API

• Implemented:

• System which uses OpenCV to calculate difference between the frames to detect the appearance of users

• To do:

• Hand raising to ask the question by using OpenCV (If time permits)

Nonverbal Inputs: Appearance of the User and Hand Raising

• Implemented:

• Facial Orientation Detection

• Work in progress :

• Face tracking: Users are starting mutual conversation or leaving the system?

Nonverbal Inputs: Face Orientation

Input Understanding Component

• Combines two verbal channels and image processing data to recognize users’ utterances

• Examples: who is speaking, who is adressee, are users’ starting mutual conversation...?

• Component is still in early stage since input components are not being implemented

Dialogue Management Component- Functionalities

• Make decisions “when and what to do to whom”:

• Handle multi-modal input events

• Handle user interruptions while the agent is doing something

• Keep a model of each user

• Keep the domain knowledge, discourse model, and context memory

• Generate multi-modal outputs

Dialogue Management Component- Progress

• Dialogue management component based on information state theory [Larsson’00] in still being implemented

• The progress of dialogue is represented by a set of variables

• These variables are updated or queried by a basic unit called dialogue move like ask, answer, repeat, inform

• Conversation strategies (plans) can be composed to respond to different circumstance

• Most appropriate plan are selected and scheduled by simple inference

Dialogue Management Component- Issues

• What to do when the agent is interrupted by the user?• What to do when speech recognition fails?• What to do when the users are talking to each other?• What to do when the users starts to talk to the system at the same time?• How ECA’s gaze direction can be controlled? • How grounding can be implemented using verbal and non-verbal information?• What to do when the response of the user is not what the system expected?

Character Animation Player

• Character animation (ECA) player generates speech and synchronized animation by using XML based GSML language• Multiple threads in animation player obtain speech and gesture synchrony• Set of animations implemented

•Gazing, beats, facial expressions (joy, sad...), symbolic gestures

•To do: construct ECA behaviors according the system scenario (canned behaviors, general states)

Character Animation Player: Example

Conclusion

• We are in (slow :S) progress• To integrate a baseline system we need to implement:

•SR component•Component to detect facial orientation•Input Understanding•DM component•Behavior specification script for character Player

• Expected Outcome: there are many...