Sensor-based Situated, Individualized, and Personalized Interaction in Smart EnvironmentsSimone Hmmerle, Matthias Wimmer, Bernd Radig, Michael Beetz

Technische Universitt Mnchen Informatik IXSensor-based Situated, Individualized, and Personalized Interaction in Smart Environments

SIP via sensorsSituation detection:information about persons: name, location, focus of attention, posture, motion,Individualized settings:desktop, avatar, input settings (gestures, voice commands,)Personalized settings: users role, right management,

SIP detection using sensorsmore comprehensive SIP informationmore intuitive HCI

Our Test BedSensors: cameras, microphones, laser-range-sensorsActuators: monitor, speaker, video-wall Scenarios:person localizationautomatic loginmeeting reminderindividualized gesture interaction

Video

Techniques (Computer Vision)person detectionOpenCV (Haar-Face-Detector)person recognitionOpenCV (Hidden Markov Models)person trackingdeveloped at TUMlaser-scanner based multiple hypothesis tracking,gesture recognitiondeveloped at TUMmotion templates, multiple classifiers,mimic recognitiondeveloped at TUMpoint distribution model, optical flow,

Techniques (others)natural language input Java Sphinx 4 (origin CMU, now open source) phonemes are already trained we defined the words ( = concatenation of phonemes) we defined the grammar ( = allowed sentences)

natural language output provides the user with audio information user can be mobile FreeTTS 1.2 (sourceforge)

Software architecture

Dispatcher

multi agent framework

ConclusionAdvantages using sensorsadditional and more exact context knowledgeunobtrusive system

Multi agent frameworkdistributed and scalable systemsimply extensible to further scenarios

Overall semanticsemantic agent communicationcentral aggregation of semantic context knowledge

Leads tomore comprehensive SIP informationseamless integration of SIP informationintuitive HCI

Thank you!

Setup & Benefitsensors for detection of SIP context:camerasmicrophoneslaser-range-sensorspressure-sensors,

sensors provide knowledge about the SIP contextsituation dependant servicesintuitive HCI (human computer interface)

application scenarios:support in meetings and presentationsintelligent House external robot control

Our Test BedSensors: Cameras, Microphones, Laser-Range-SensorsActuators: Monitor, Speaker, Video-Wall

Scenarios: automatic loginmeeting reminderindividualized gesture interactionintuitive robot controlperson localization

Sensors person recognition

(Bild)

gesture recognition

(Bild)

Knowledgebase Web Ontology Language (W3C)

Possible Application scenarios: support in meetings and presentations intelligent house external robot control