mobile phones based continuous sensing systems
Post on 25-Feb-2016
35 Views
Preview:
DESCRIPTION
TRANSCRIPT
1
Mobile Phones based Continuous Sensing Systems
Kiran Rachurikkr27@cam.ac.uk
Computer LaboratoryUniversity of Cambridge
2
• Microphone
• Magnetometer
• GPS
• Bluetooth
• Accelerometer
• Camera
• Ambient light
• Proximity
Sensors in a Smart Phone
3
Microphone – Speaker Recognition
What Can be DoneCamera – Face Recognition
Accelerometer – Activity Recognition
4
Health monitoring
Application ScenariosSocial sensing
Location based services
5
Sense continuously
Energy-Accuracy trade-offs
Continuous Sensing Mobile Systems
6
Mobile Phone Limitations
Energy, processing, and memory constraints
Google Nexus One, and HDC HD2 are equipped with1GHz processor and 512MB RAM
Energy is still a scarce resource
7
EmotionSense
8
Emotions of users and how they are influenced
Social Psychology - Emotions
9
Speech patterns of users
Social Psychology - Speech Patterns
10
Oh yes, I was very happy today
Self reports ◦ But they are biased towards positive emotions
One-time behavioural study in laboratory◦ Users hide their natural behaviour
Social Psychology - Existing Methods
11
• Use Mobile phones• Powerful sensors• Ubiquitous• Unobtrusive
Our Solution
• Challenges• Sensors not built for this purpose• Battery powered• Processing• Main memory limitations• Privacy concerns
12
EmotionSense - Architecture
EmotionSense Manager
Inference Engine
SpeakerMonitor
HTTP Module
Declarative Database
Remote ServerColocationMonior
MovementMonitor
LocationMonitor
HTK
HTK: Hidden markov ToolKit
Implemented in PyS60 and Symbian C++
13
EmotionSense – Flow of Data
Classifiers Facts Base Inference Engine Actions Base
EmotionSense Manager
Sensor Monitors
Raw data
E.g. X Y Z of Accelerometer
E.g. User is MovingE.g. fact(Moving, True)E.g. fact(action, LocationSamplingInterval, 2)
14
Emotion & Speech Recognition
Collect voice data User/Emotion specific modelsTrain background GMM
[2] M. Liberman, K. Davis, M. Grossman, N. Martey, and J. Bell. Emotional prosody speech and transcripts, 2002.
Training Procedure
At Runtime
Record voice on phone Extract PLPs using HCopy Compare using HERest
[1] http://htk.eng.cam.ac.uk
GMM: Gaussian Mixture ModelPLP: Perceptual Liner PredictionHcopy, HERest: Tools of Hidden markov ToolKit (HTK)
Speaker Recognition: Participants voice dataEmotion Recognition: from libraryLoad speaker and emotion models on phone
15
Clustering of emotions
Emotion Categories
Broad Emotion Narrow Emotion
Happy Elation, Interest, Happy
Sad Sadness
Fear Panic
Anger Disgust, Dominant, Hot anger
Neutral Neutral normal, Neutral conversation, Neutral distant, Neutral tete, Boredom, Passive
(a) Used by psychologists (b) Improves accuracy
Why?
16
Optimizations
Adaptive framework
What About Energy?
17
Silence detection◦ train an additional GMM using silence audio
Comparisons driven by co-location information◦ A recorded audio sequence is compared only
with the models co-located users◦ This improves accuracy and saves energy
Speaker Recognition - Optimizations
18
Implemented using Pyke, a knowledge based inference engine [http://pyke.sourceforge.net/]
Activate GPS only when user is moving
Adaptation Framework
set_location_sampling_intervalforeach facts.fact($factName, $value) check $factName == 'Activity' facts.fact($actionName, $currentInterval) check $actionName == 'LocationInterval' $interval = update($value, $currentInterval)assert facts.fact('action', 'LocationInterval', $interval)
Facts Base Inference Engine Actions Base
E.g. fact(Moving, True)E.g. fact(action, GPS, ON)
19
Sensor Sampling
Time
Sleep Sense Sleep Sense Sleep
Events
Time
Sleep Sense
Events
Sense SleepSleep
20
Sensor Sampling Issues Continuous sampling
degrades battery life
Long sleep durations result in loss of sensor data
Not all sensors are similar
Accuracy varies with sensors and classifiers
21
Sampling Interval
Sleep Sense Once
Sleep 0 ∞Minimum Sampling Interval
Maximum Sampling Interval
Constant
22
Design Methodology
Missable Event
Sleep
Not all events are important
E.g.: Microphone recording when there is no audible sound
•Classify events as Unmissable and Missable•Use functions to control the sleep interval
Back-off Function
E.g.: f(x) = 2x, where x is sleep interval
UnMissable Event
Advance Function
E.g.: f(x) = x/2, where x is sleep interval
Sense
23
Back-off and Advance FunctionsType Back-off function Advance function
Linear
Quadratic
Exponential
Minimum N/A Minimum interval
Maximum Maximum interval N/A
€
kxkx
2xxe
€
ln(x)
x
x: sleep interval
24
Dynamic Adaptation
Dynamically switch functions from least to most aggressive
Missable Event Sleep
Sequence Count
Sense
Linear back-off function
Quadratic back-off function
Exponential back-off function
Update Sleep
Interval
< Linear Threshold
< Quadratic Threshold
> Quadratic Threshold
25
Micro-benchmarks
Social psychology experiment
Meeting experiment
Evaluation
26
Speaker Recognition - Benchmarks
Accuracy Effect of noise on accuracy
27
Speaker Recognition – Benchmarks
Energy consumption Latency
Why ?
a) Computationally intensive processingb) High latency
Then why compute locally ?
a) Privacy concernsb) Users can use their own sim cards
28
Emotion Recognition - Benchmarks
Accuracy Energy consumption
Clustering helps
29
Nokia 6210 Navigator mobile phones
18 participants, 10 days
Users filled in daily diary questionnaire
Voice data is discarded immediately
All computation performed locally on phone
Social Psychology Experiment
30
Emotion distribution similarity EmotionSense Self reports
Social Psychology Experiment - Results
Users indicated ``happy'' emotion to represent their mental state, and not necessarily verbal expression
31
Social Psychology Experiment - Results
Correlation with time of day and co-location
32
EmotionSense can also be used to analyze the speech patterns
Considerable amount of consistency in verbal behaviour
Social Psychology Experiment - Results
33
11 participants , 30 minutes discussion
We identified conversation leaders in each time slot of length 5 minutes
The result shows the top five most active speakers
Meeting Experiment
34
Continuous sensing mobile systems
EmotionSense◦ Speaker and Emotion Recognition◦ Sensor Monitors◦ Adaptive and Programmable Framework
Evaluation◦ Micro-benchmarks◦ Social Psychology Experiment◦ Meeting Experiment
Summary
35
Thank YouKiran Rachuri
Computer LaboratoryUniversity of Cambridge
kkr27@cam.ac.uk
EmotionSense http://www.cl.cam.ac.uk/research/srg/netos/emotionsense/
top related