machine-mediated communication 3d environments, (sensors, projectors: sight, sound, touch, other...
TRANSCRIPT
![Page 1: MACHINE-MEDIATED COMMUNICATION 3D environments, (sensors, projectors: sight, sound, touch, other sensory modes) information capture, display personal agents](https://reader030.vdocuments.net/reader030/viewer/2022033106/56649dba5503460f94aaa5e5/html5/thumbnails/1.jpg)
MACHINE-MEDIATEDCOMMUNICATION
3D environments,(sensors, projectors: sight, sound, touch, other sensory modes)
information capture, display
personalagents multimodal
fusion
storage
media conversion, multilingual translation
information representation
computationmanagement
architectures, protocols
agentsdB’s
libraries
storageWeb pages
grid-based computation
Other Sites
HumanInterface
GlobalNetwork
CommunicationMediator
security
security
![Page 2: MACHINE-MEDIATED COMMUNICATION 3D environments, (sensors, projectors: sight, sound, touch, other sensory modes) information capture, display personal agents](https://reader030.vdocuments.net/reader030/viewer/2022033106/56649dba5503460f94aaa5e5/html5/thumbnails/2.jpg)
Multimodal Human-Machine Interface
![Page 3: MACHINE-MEDIATED COMMUNICATION 3D environments, (sensors, projectors: sight, sound, touch, other sensory modes) information capture, display personal agents](https://reader030.vdocuments.net/reader030/viewer/2022033106/56649dba5503460f94aaa5e5/html5/thumbnails/3.jpg)
Force Feedback Glove
Gaze Tracker
Microphone
Array
Multimodal Human-Machine Interface
Real-world trial with NJ National Guard
![Page 4: MACHINE-MEDIATED COMMUNICATION 3D environments, (sensors, projectors: sight, sound, touch, other sensory modes) information capture, display personal agents](https://reader030.vdocuments.net/reader030/viewer/2022033106/56649dba5503460f94aaa5e5/html5/thumbnails/4.jpg)
Portable Interactive Command Console (PICC)
Steerable microphone arrayelement
Stereo face tracking cameras
Gaze tracker
Source locator microphone
Loudspeakers
Light pen
Flatpaneldisplay
Sensors
HQ/VEHICLE
Robotic Vehicles Emergency Responder in the Field
Internet
![Page 5: MACHINE-MEDIATED COMMUNICATION 3D environments, (sensors, projectors: sight, sound, touch, other sensory modes) information capture, display personal agents](https://reader030.vdocuments.net/reader030/viewer/2022033106/56649dba5503460f94aaa5e5/html5/thumbnails/5.jpg)
User interface for interaction and collaboration with robots and humans
NSF Equipment Grant EIA#98-18313 Center for Advanced Information Processing, Rutgers University, Piscataway, NJ 08854.
PI: J.L. Flanagan, co-PIs: J. Wilder, I. Marsic, M. Krane
![Page 6: MACHINE-MEDIATED COMMUNICATION 3D environments, (sensors, projectors: sight, sound, touch, other sensory modes) information capture, display personal agents](https://reader030.vdocuments.net/reader030/viewer/2022033106/56649dba5503460f94aaa5e5/html5/thumbnails/6.jpg)
“Move this to there.”
![Page 7: MACHINE-MEDIATED COMMUNICATION 3D environments, (sensors, projectors: sight, sound, touch, other sensory modes) information capture, display personal agents](https://reader030.vdocuments.net/reader030/viewer/2022033106/56649dba5503460f94aaa5e5/html5/thumbnails/7.jpg)
FRAMEWORK FOR MULTIMODAL LANGUAGE
SENSORY SIGNAL
IDENTIFY ACTION
ESTIMATE MEANING
GENERATE RESPONSE
FORM SENTENCE
CREATE “WORDS”
CLASSIFY SEGMENTS
SEGMENT CONTINUA
EXTRACT FEATURES
Transduce intelligence
Options library
Semantic rules; parse
sentence
Multimodal language
generation
Syntax rules; word n-grams
Lexicon; grammar;
symbol n-grams
Symbol library
Source – receiver
constraints
Transform domain
FUSION OF MODALITIES