1seminar „multimodale räume“ uni karlsruhe, 14.5.2003 the fame project acronym:facilitating...

Seminar „Multimodale Räume“Uni Karlsruhe, 14.5.2003

1

The FAME Project• Acronym: Facilitating Agent for Multicultural Exchange

• Partners:

Universität Karlsruhe, INPG Grenoble,UJF Grenoble, ITC-irst Trento,UPC Bacelona, SONY Europe, StuttgartATLAS Barcelona

• Project volume: 5.5 M Euro• Duration: 40 months, started October 2001

• More info: http://www.fame-project.org


2

The FAME ProjektFacilitating Agent for Multicultural Exchange

• Volume:• Duration: 40 months (since October 2001)

• Financial Volume: 5,5 Mio. €

• currently approx. 30 scientists

• Partners:Uni Karlsruhe , INPG Grenoble , UJF Grenoble,

UPC Barcelona , ITC-irst Trento ,

Stuttgart, Barcelona

www.fame-project.org


3

Project Goals• long term vision: facilitate communication between humans

• reduce the workload on the users of technical equipment• observe humans and their activities in an intelligent room

and serve as a context-aware information butler

• FAME project goal: provide and integrate core technologies (video and speech perception, augmented reality, translation, information retrieval) to show feasibility of the concept

• demonstrate system at fair• scenario 1 (lecture scenario):

one person is giving a talk or lecture or presentation• scenario 2 (meeting scenario):

several people are discussing / working on a common task


4

The FAME Showcasesscenario 1 (presentation) scenario 2 (meeting)

• use A/V equipment• intelligent cameraman• presentation tracking• summarisation + archiving• translation, crosslingual IR

• augmented reality• video-based activity tracking • topic spotting• information butler• service: planning of fair visit


5

The FAME Demonstrator(at Barcelona Fair „Forum of Cultures“ 2004)

FAME outside view reception by FAME-guy

meeting inside people mention topicsroom gives informationabout spotted topics


6


at the phicon wall gestures the projection table

multimodal input on tableoutput also on the wall interactive visit planning

borrow a camera forphotographs of the visit


7


Back from the visit in theFAME room: dowload ...

... and look at photos select, print, save photosusing phicon interaction

record testimony intelligent cameraman,presentation tracker

take home photos andinformation about FAME


8

• multimodal environment

• context-aware intelligent camera-manautomatically track people and their activities

• augmented reality environmentmove physical icons (phicons) on table/wall,and interact with projection on table/wall

• spontaneous speech recognition (with distant microphones)• translation and crosslingual information retrieval

in European-English, Catalan, and Spanish• dialog and context model

Important Components


9

Microphon-Array(Speaker Lokalization)Microphon-Array(Speaker Lokalization)

Smartboard asProjection WallSmartboard asProjection Wall

LivingroomgLivingroomg

IR-Remote ControlIR-Remote Control X-10X-10 IlluminationIllumination

Audio SignalsAudio Signals

Several BeamersSeveral Beamers 4 Cameras4 Cameras TV/VideoTV/Video

LoudspeakersLoudspeakers

MicrophonesMicrophones

Multimodal Environment at UKA


10

Augmented Reality Table

• project virtual reality on real table• move around physical icons (multiple users)• interact with projection

• select, move, rotate, resize, delete, change color• write on table, pass notes to others, point to items


11

Intelligent Camera Man

• follow speaker while talking and moving around

• detect interaction from audience

• zoom on area of intereste.g. when pointing somewhere or showing something


12

LectureSupporter

• track lecture or presentation

• operate FAME room equipmentby speech commands

• automatically switch slides during presentation

• automatically create transcript of lecture

• create summary, translate to other languages

• record and store all lectures in searchable database

• retrieve and browse through previously recorded lectures


13

100 Links100 Links

wichtige Wörter

wichtige Wörter

wichtige Wörter

TO THE RECOGNITION OF THECONTINUOUS SPEECH RECOGNITION IN NOISY AND PATTERN RECOGNITION NOT IN SECURING DUE RECOGNITION AND RESPECT

CLASS 32CLASS 14CLASS 57CLASS 6CLASS 70

top n

most frequent40k words

least frequent20k words

60k vocabularyHUB-4 corpus

trigramm- modelP(wn|wn-1,wn-2) 72 classesP(Cn|wn-1,wn-2)· P(wn|Cn)

add to classes

estimate

presentation-slides

important words

tf-idf

100 links±2 contexts

perplexity

scores

Adaptation Overview

20% fewererrors


14

Welcome in Barcelonain Summer 2004

1seminar „multimodale räume“ uni karlsruhe, 14.5.2003 the fame project acronym:facilitating...

Documents