Sensor-based Situated, Individualized, and Personalized Interaction

in Smart Environments

Simone Hämmerle, Simone Hämmerle, Matthias WimmerMatthias Wimmer, Bernd Radig, Michael Beetz, Bernd Radig, Michael Beetz

Technische Universität München – Informatik IXTechnische Universität München – Informatik IX

Sensor-based Situated, Individualized, and Personalized Interaction

in Smart Environments

SIP via sensorsSIP via sensors

Situation detection:Situation detection: information about persons: name, location, focus information about persons: name, location, focus

of attention, posture, motion,…of attention, posture, motion,…

Individualized settings:Individualized settings: desktop, avatar, desktop, avatar,

input settings (gestures, voice commands,…)input settings (gestures, voice commands,…)

Personalized settings: Personalized settings: user’s role, right management,…user’s role, right management,…

SIP detection using sensorsSIP detection using sensors more comprehensive SIP informationmore comprehensive SIP information more intuitive HCImore intuitive HCI

Our Test BedOur Test Bed

Sensors: cameras, microphones, laser-range-sensorsSensors: cameras, microphones, laser-range-sensorsActuators: monitor, speaker, video-wallActuators: monitor, speaker, video-wall

Scenarios:Scenarios: person localizationperson localization automatic loginautomatic login meeting remindermeeting reminder individualized individualized

gesture interactiongesture interaction

laser range sensor

camera

video wall

loud speaker

automatic

login

seamless track

identification

control video wall

using gestures

natural language

interaction

VideoVideo

Techniques (Computer Vision)Techniques (Computer Vision)

person detectionperson detectionOpenCV (Haar-Face-Detector)OpenCV (Haar-Face-Detector)

person recognitionperson recognitionOpenCV (Hidden Markov Models)OpenCV (Hidden Markov Models)

person trackingperson trackingdeveloped at TUMdeveloped at TUMlaser-scanner based multiple hypothesis tracking,…laser-scanner based multiple hypothesis tracking,…

gesture recognitiongesture recognitiondeveloped at TUMdeveloped at TUMmotion templates, multiple classifiers,…motion templates, multiple classifiers,…

mimic recognitionmimic recognitiondeveloped at TUMdeveloped at TUMpoint distribution model, optical flow,…point distribution model, optical flow,…

Techniques (others)Techniques (others)

natural language input natural language input Java Sphinx 4 (origin CMU, now open source)Java Sphinx 4 (origin CMU, now open source)

phonemes are already trained phonemes are already trained

we defined the words ( = concatenation of phonemes)we defined the words ( = concatenation of phonemes)

we defined the grammar ( = allowed sentences)we defined the grammar ( = allowed sentences)

natural language outputnatural language output provides the user with audio informationprovides the user with audio information

user can be mobileuser can be mobile

FreeTTS 1.2 (sourceforge)FreeTTS 1.2 (sourceforge)

Software architectureSoftware architecture

Multi Agent Framework

Loudspeaker Agent

Loudspeaker

Microphone Agent

Microphone Module

Microphone

D

Camera

Person Recognition

Agent

Person Recognition

Module

Robot

Robot AgentRobot Module

Illumination Agent

Illimination Module

LampBrain

Dis

patc

her

Knowledge Base Accessor

Knowledge Base

Command-Generator

Command-Generator

Command-Generator

Loudspeaker Module

FilterFilterFilter

D

Camera

Gesture Detection

Agent

Gesture Detection Module

semantic communication

proprietary communication

Dis

pa

tch

er

multi agent framework

ConclusionConclusion

Advantages using sensorsAdvantages using sensors additional and more exact context knowledgeadditional and more exact context knowledge unobtrusive systemunobtrusive system

Multi agent frameworkMulti agent framework distributed and scalable systemdistributed and scalable system simply extensible to further scenariossimply extensible to further scenarios

Overall semanticOverall semantic semantic agent communicationsemantic agent communication central aggregation of semantic context knowledgecentral aggregation of semantic context knowledge

Leads toLeads to more comprehensive SIP informationmore comprehensive SIP information seamless integration of SIP informationseamless integration of SIP information intuitive HCIintuitive HCI

Thank you!Thank you!

Setup & BenefitSetup & Benefit

sensors for detection of SIP context:sensors for detection of SIP context: camerascameras microphonesmicrophones laser-range-sensorslaser-range-sensors pressure-sensors, …pressure-sensors, …

sensors provide knowledge about the SIP contextsensors provide knowledge about the SIP context situation dependant servicessituation dependant services intuitive HCI (human computer interface)intuitive HCI (human computer interface)

application scenarios:application scenarios: support in meetings and presentationssupport in meetings and presentations intelligent House intelligent House external robot controlexternal robot control

Our Test BedOur Test Bed

Sensors: Cameras, Microphones, Laser-Range-SensorsSensors: Cameras, Microphones, Laser-Range-SensorsActuators: Monitor, Speaker, Video-WallActuators: Monitor, Speaker, Video-Wall

Scenarios: Scenarios: automatic loginautomatic login meeting remindermeeting reminder individualized gesture interactionindividualized gesture interaction

intuitive robot controlintuitive robot control person localizationperson localization

SensorsSensors

person recognitionperson recognition

(Bild)(Bild)

gesture recognitiongesture recognition

(Bild)(Bild)

KnowledgebaseKnowledgebase

Web Ontology Language (W3C)Web Ontology Language (W3C)