A Framework for Energy Efficient Mobile Sensing for Automatic User State Recognition

Yi Wang, Jialiu Lin, Murali Annavaram, Quinn A. Jacobson,

Jason Hong, Bhaskar Krishnamachari and Norman Sadeh

OUTLINE

Motivation

EEMSS introduction

System modules

Case implementation

Performance evaluation

Conclusion

PEOPLE-CENTRIC MOBILE SENSING

Health monitoring

Keeping track of patients/elders

Social networking

Automated “Twittering”

Automated profile updates

Ringtone adjustment

Location based services

Mobile advertising

CHALLENGE: LIMITED BATTERY

Battery capacity of mobile device is low

Sensors are main source of energy usage

Blind sensing drains battery soon

Need intelligent sensor management

Energy Efficient Mobile Sensing System (EEMSS) is important

Possibly trade some detection accuracy for much longer device lifetime

AN EXAMPLE

Office Library

SENSOR MANAGEMENT METHODOLOGY

Only utilize a minimum set of sensors

Recognize state & detect state transition

E.g.: No need for GPS when indoor

Hierarchical management

Sensors are activated when necessary

E.g.: Accelerometer -> WiFi scan -> GPS

If multiple sensors achieve the same task: Use energy efficient ones

How to determine sensor energy efficiency?

DETERMINE ENERGY EFFICIENCY

Energy = Power drain × Operating duration

Sensor power consumptions on N95 devices:

Keep an eye on sensor operating duration

EEMSS SYSTEM ARCHITECTURE

USER STATE DESCRIPTION

Task 1: Identify the states to be detected Working/meeting/walking/driving outdoor …

Task 2: For each state define entry criteria Sensing thresholds from one or more sensors

E.g.: “Outdoor” + “High travel speed” = “Vehicle”

A state is entered when the criteria are satisfied

Task 3: For each state also specify the necessary sensors to be monitored Detect state transition

XML BASED STATE DESCRIPTOR

DESIGN BENIFITS

Scalable

Add or remove a state upon user’s interest

Different criteria for different individuals

Real-time update

User’s habits may change

Sensing criteria can to be refined in real-time

CLASSIFICATION MODULE

Classification algorithms are the key to high state recognition accuracy

Mobile phone is the only sensing resource

Capability limitations

Computing issues

Real-time

Computing power

REAL-TIME AUDIO RECOGNITION

Based on microphone sensing

Focus on the detection of speech

Audio features Energy Silence ratio SSCH peak1

3 outputs: Speech Loud/Noisy Silent

1 B. Gajic and K.K. Paliwal. “Robust speech recognition in noisy environments based on subband spectral centroid histograms”

SPEECH DETECTION PERFORMANCE

Algorithm is tested on 1085 speech clips

91.14 % are classified as speech

Complexity of algorithm is O(N2)

N: Number of frequency samples

Overall processing time of a 4 seconds sound clip on N95

~10 seconds

EEMSS CASE IMPLEMENTATION

Implemented and tested on Nokia N95

Sensors operated include:

Accelerometer, Microphone, WiFi detector, and GPS

User states are featured by:

Location, background sound, and user motion information

E.g.: “office + quiet + still” => User working in office

STATES INTERESTED

State Name State Features Sensors Monitored

Location Motion Background Sound

Working Office Still Quiet Accelerometer, Microphone

Meeting Office Still Speech Accelerometer, Microphone

Office_loud Office Still Loud Accelerometer, Microphone

Resting Home Still Quiet Accelerometer, Microphone

Home_talking Home Still Speech Accelerometer, Microphone

Home_entertaining Home Still Loud Accelerometer, Microphone

Place_quiet Some Place Still Quiet Accelerometer, Microphone

Place_speech Some Place Still Speech Accelerometer, Microphone

Place_loud Some Place Still Loud Accelerometer, Microphone

Walking Keep on changing Moving Slowly N/A GPS

Vehicle Keep on changing Moving Fast N/A GPS

SENSOR DUTY CYCLES

When activated, sensors are turned on and off periodically

Trade-offs in sampling:

Frequent sampling: Wastes energy and provides redundant information

Infrequent sampling: Saves energy but low state detection accuracy

Current implementation provides small event detection delay

EEMSS PERFORMANCE EVALUATION

We evaluate EEMSS in terms of: Ability to record one’s state in real time State recognition accuracy Energy efficiency

User study at CMU & USC with 10 users Each user carried Nokia N95 phone as daily

used cell phone with EEMSS running Ground truth was manually recorded by

each userFine-grained entries with time and state

records

STATE RECOGNITION RECORD

State records of two sample users

A CMU user A USC user

STATE RECOGNITION ACCURACY

Individual user state recognition accuracy for all 10 participants:

STATE RECOGNITION ACCURACY

Confusion matrix of recognizing “Walking”, “Vehicle”, and all other states*

All other states

Walking Vehicle

All other states 99.17% 0.78% 0.05%

Walking 12.64% 84.29% 3.07%

Vehicle 10.59% 15.29% 74.12%

* “All other states” include “Working”, “Meeting”, “Office_loud”, “Resting”, “Home_talking”, “Home_entertaining”, “Place_quiet”, “Place_speech”, and “Place_loud”.

DEVICE LIFETIME

Average device lifetime comparison

More than 75% gain compared to existing systems

EEMSS ENERGY USAGE AT A GLANCE

User worked in office, then walked to library and stayed (20 min empirical interval)

CONCLUSION

User state recognition based on mobile sensing is popular

Energy efficiency is required due to low device battery capacity

Our sensor management methodology: Utilizing minimum number of sensors to

accomplish sensing tasks

Manage sensors hierarchically

EEMSS achieves good state recognition accuracy and energy efficiency

THANK YOU!