field informatics human sensing accms, kyoto university yuichi nakamura 1/46 copyright (c) 2010...
TRANSCRIPT
Field InformaticsHuman Sensing
ACCMS, Kyoto University
Yuichi Nakamura
1/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Introduction to Field Informatics Chapter4
Human Sensing:Measuring Human Activities and Social Actions
• Human Activities– simple activities
• walking, eating, house keeping, etc.• simple tasks in daily life
– social actions• conversation, meeting, lectures, etc.• tasks with other people
– others• peoples in a panic
• Measuring what and how?• How store and retrieve the data?
2/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Overview: Purpose of Human Sensing
(a) External information– who did what, how,....
(b) Internal information– thought, intention, feeling,...– physiological conditions
(c) Communication– communicated information– communication intention
3/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Overview: Information on Humans
(a) External information– bodily movements, behaviors, ...– bodily characteristics
(appearance , sweat , smell , etc.)(b) Internal information
– psychological conditions (tension , fear , emotion , comfort/discomfort , etc.)
– physiological conditions (table 4-3)(c) Communication information
– verbal/non-verbal communication– interpersonal contact, interpersonal
distance, mutual interaction with group
4/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Overview: Tools for Human Sensing
(a) External information– physical sensors– esp. non-invasive, non-intrusive
sensors
(b) Internal information– physiological sensors– brain measurements– introspection, reflection
(c) Communication information– physical sensors– ethnography, ethnomethodology
5/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Information on Humans (examples)
(a) External conditions– bodily movements, behaviors, ...– bodily characteristics
(appearance , sweat , smell , etc.)
(b) Internal conditions– psychological conditions (strain , fear ,
emotion , pleasant , etc.)– physiological conditions
(c) Communication conditions– verbal communication– non-verbal communication
6/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Human positions and movements
• image sensors
• magnetic sensors• ultrasonic wave sensors• RFID
7/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Human Position
• Image sensors– real-time tracking– frequently used for
security and surveillance purpose
• Fish-eye lens and omni-directional cameras– omni-directional– low spatial resolution
8/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Face Detection
• Image sensors with image recognition software.– face detection– face identification
• Many embedded system, e.g., digital camera.
9/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
3D Measurements
• Stereo Vision
Multiple Stereo Video CameraStereo Still Camera
10/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Real-time Stereo Machine
• 1995 Carnegie Mellon University
11/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Magnetic Sensors (Motion Capture)
• Comparing with image sensors– better accuracy– no occlusion effect
• Characteristics– ▲ cost, size– × non-intrusive– ◎ accuracy– ◎ occlusion– ◎ lighting– ×other constrains (affected by metal)
12/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Eye Tracking, Gaze Tracking
• Head mount type– measuring eye ball
direction by projecting infrared light
• Table mount type– measuring the pupil
position by video camera(s)
13/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Eye/Gaze Tracking
• Gazing properties– a sequence of fixations– order, duration– movements, saccades
• Internal conditions– intention, attention,
interest
• Object’s characterisitics– features– characteristics
14/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Information on Humans (examples)
(a) External conditions– bodily movements, behaviors, ...– bodily characteristics
(appearance , sweat , smell , etc.)(b) Internal conditions
– psychological conditions (tension , fear , emotion , comfort/discomfort , etc.)
– physiological conditions (table 4-3)(c) Communication conditions
– verbal/non-verbal communication– interpersonal contact, interpersonal
distance, mutual interaction with group
15/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Physiological Conditions (examples)
• electrocardiogram (ECG), heart rate, blood pressure, pulse pressure, O2/CO2 concentration in the blood
• breathing rate , O2/CO2 concentration in breath
• electrooculogram (EOG) , blink , pupil size , focus
• electromyography (EMG) , evoked electromyography
• skin potential activity, flicker value, body temperature, facial skin temperature, perspiration , etc.
• electroencephalogram (EEG), magnetoencephalograpy (MEG), functional mgnetic resonance imaging (fMRI) ,near infrared spectroscoping topography (NIRS)
16/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Measuring Brain Activity
• electric activities of neurons
• magnetic field caused by electric activities
• blood flow and brain metabolism
17/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Electroencephalogram
• electrical alterations in accordance with neural activity
• small potential changes on the scalp
-
-50
0
I1
I 2
-50
-100
I1
I 2
-50
-50
-
18/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Measuring Method
• 10-20 system : distances between adjacent electrodes are either 10% or 20% of the total front-back or right-left distance of the skull. 鼻根部
Nasion
後頭結節Inion
左耳 右耳
19/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Brain Waves• delta wave δ : 1 ~ 3Hz • theta wave θ : 4 ~ 7Hz • alpha wave α : 8 ~ 13Hz • beta wave β : 14 ~ 30Hz • gamma wave γ : 30 ~ 64Hz • omega wave ω : 64 ~ 128Hz • rho wave ρ : 128-512Hz • sigma wave σ : 512-1024Hz
sleep
relaxed
active
exited
20/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Near Infra-Red Spectroscopic Topography (NIRS)
• Near infrared light (around 800nm) is projected and traverses the scalp and skull
• Reflectance from the brain are measured on the scalp
• Brain metabolism can be measured by the ratio of oxidized hemoglobin and deoxidized hemoglobin
Near Infrared Light
Brain
Scalp
Skull
21/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
MagnetoEncepharoGraph (MEG)
• Magnetic field arising from neural electrical activity
• Large-scale system with high-performance probes
• high temporal resolution
• noise elimination is a serious problem
22/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
functional Magnetic Resonance Imaging ( f MRI)
• Brain activity (blood flow, metabolism)
• Magnetic resonance difference between oxidized hemoglobin and deoxidized hemoglobin
23/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
ElectroMyoGraphy (EMG)
• Membrane potential changes in muscle contraction
muscle fiber
nerve muscle connection
muscle
motor nerve
24/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
surface EMG
AmplifierElectrodes
• Electric potential changes ( 10mV) ≦ on skin surface
• Electrodes and relatively simple electronic circuits
• problem: noise elimination, MU estimation
25/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
0.5 sec
1 m v
Multi Channel Measurement
26/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Example of EMG signal
0.5 sec
1 m v
Condition 1
Condition 2
similar motions with different conditions
27/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Subjective or introspective analysis for psychological conditions
• Conversation Analysis
• Protocol Analysis
• Narrative Analysis
28/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Protocol Analysis
Ask a subject to tell anything which comes up to the subject’s mind, and analyze the internal process of the subject’s.
1: Which one?2: I got it!3: Difficult to find, ...4: Hmm, push it, ... really?5: I’m afraid all are gone...
29/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Protocol Analysis
• think aloud method: speak synchronously what the subject thinks during actions
• retrospective report method: explain actions after it is finished
• In both methods, actions are takes in a video or some recording devices, and those data are minutely analyzed afterwards.
30/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
ref. Narrative Analysis
• The subject reconstructs real experiences as a personal story
• Originated from narrative therapy• Linear causality is a dominant feature of
narrative structure • A subject is prompted story telling by a
question addressing what to tell.– Type1: as less interruption as possible– Type2: guided by appropriate questions
31/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Information on Humans (examples)
(a) External conditions– bodily movements, behaviors, ...– bodily characteristics
(appearance , sweat , smell , etc.)(b) Internal conditions
– psychological conditions (tension , fear , emotion , comfort/discomfort , etc.)
– physiological conditions (table 4-3)(c) Communication conditions
– verbal/non-verbal communication– interpersonal contact, interpersonal
distance, mutual interaction with group
32/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Sensing of Communication Conditions
• Nonverbal information– 70 ~ 80 % of information is carried
through nonverbal behaviors
• Various kinds of nonverbal information– attitude, behaviors– body characteristics– perspiration , smell– clothes, accessories– others
33/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
ref. distance communication
• Asynchronous communication– e-mail– Web
• Realtime communication– chat– video conference– distance lecture
34/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Key Points on Human Sensing
• Objective measurements– physical sensors as much
as possible– non-invasive, non-intrusive
sensors• Multiple sensors
– synchronization– large amount of data
• Data handling– indexing– browsing– retrieval
35/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Some examples
• Smart meeting recording
• Lifelog
• Data Browsing
36/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Smart Meeting Recorder• Non-intrusive sensing and recording
– tracking each person from entering room to sitting down
– tracking each person’s face while talking
• Video capturing with typical picture compositions
カメラ制御コンポーネント
観測カメラ:人物の位置検出
制御指令・映像選択コンポーネント
撮影カメラ:首振りカメラによって追跡撮影 映像切替器
MPEG エンコード, HDD に録画
pan/tilt 制御
映像を提示• Two types of cameras
– sensing camera– contents capturing
camera• Contents capturing
cameras are guided by the sensing camera.
37/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Sensing camera:detecting participants positions
control, selecting views
capturing camera: tracking and capturing
video switching
MPEG encoding
pan/tilt control with tracking a face
38/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Automatic Editing Examples
39/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Smart Meeting Browser• Smart meeting browser with realtime meeting
capture• Toward realtime meeting support
40/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Lifelog (personal experience log)
video,sound,location,temperature,time,etc.
experiences
large amount of logs
automatic indexingstructure analysisefficient retrieval
browsing
memory aidseducation supportdisability supporthuman factor analysis
41/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
• Browsing– skimming
– gathering related actions
used it here
took it herecame into a room
went out a room
Lifelog (Personal View Records)
42/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Browsing
Browsing for indoor activities browsing by related events
43/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Data Indexing
• Meta-data– author, title, date, keywords, etc.– index, tag, ...
• Automated indexing by video, audio, and text processing.
• Examples– XML– MPEG7– ANVIL
44/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
ANVIL
45/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.
Example of Index<?xml version="1.0" encoding="ISO-8859-1"?><annotation> <head> <specification src="C:\Documents and Settings\nakamine\kijou_sagyou.spec" /> <video src="C:\Documents and Settings\nakamine\fish_nagai.mov" /> <info key="coder" type="String"> Our server </info> <bookmark name="scene01" time="73.43333" /> <bookmark name="scene02" time="234.89999" /> (...) <bookmark name="scene07" time="852.73334" /> <bookmark name="scene08" time="900.59998" /> </head> <body> <track name="situation" type="primary"> <el index="0" start="71.13333" end="203.5"> <attribute name="token">scene01</attribute> </el> <el index="1" start="203.5" end="234.33333"> <attribute name="token">show a sample</attribute> </el> <el index="2" start="234.89999" end="294"> <attribute name="token">scene02</attribute> </el>( 以下省略 )
46/46Copyright (C) 2010 Field Informatics Research Group. Kyoto University. All Rights Reserved.