Introduction to Mobile Sensing with Smartphones

Uichin LeeApril 22, 2013

iPhone 4 - Sensors

Applications

• Transportation– Traffic conditions (MIT VTrack, Nokia/Berkeley

Mobile Millennium) • Social Networking– Sensing presence (Dartmouth CenceMe)

• Environmental Monitoring– Measuring pollution (UCLA PIER)

• Health and Well Being– Promoting personal fitness (UbiFit Garden)

Citysense

MacroSense

CabSense

Eco-system Players

• Multiple vendors– Apple AppStore– Google Play (Android Market)– Microsoft Mobile Marketplace

• Developers– Startups– Academia– Small Research laboratories– Individuals

• Critical mass of users

Scale of Mobile Sensing

Sensing Paradigm

• Participatory: active sensor data collection by users– Example: managing garbage cans by taking photos – Advantages: supports complex operations– Challenges:

• Quality of data is dependent on participants

• Opportunistic: automated sensor data collection– Example: collecting location traces from users– Advantages: lowers burden placed on the user– Challenges:

• Technically hard to build – people underutilized• Phone context problem (dynamic environments)

SENSE

LEARN

INFORM, SHARE AND PERSUASION

Mobile Sensing Architecture

Mobile Computing Cloud

Sense

• Programmability– Managing smartphone sensors with system APIs– Challenges: fine-grained control of sensors, portability

• Continuous sensing– Resource demanding (e.g., computation, battery)– Energy efficient algorithms– Trade-off between accuracy and energy consumption

• Phone context– Dynamic environments affect sensor data quality– Some solutions:

• Collaborative multi-phone inference• Admission controls for removing noisy data

SENSE

LEARN

INFORM, SHARE, PERSUASION

Learn

• Integrating sensor data– Data mining and statistical analysis

• Learning algorithms – Supervised: data are hand-labeled (e.g., cooking, driving)– Semi-supervised: some of the data are labeled– Unsupervised: none of the data are labeled

• Human behavior and context modeling• Activity classification• Mobility pattern analysis (place logging)• Noise mapping in urban environments

SENSE

LEARN

INFORM, SHARE, PERSUASION

Learn: Scaling Models

• Scaling model to everyday uses – Dynamic environments; personal differences – Large scale deployment (e.g., millions of people)

• Models must be adaptive and incorporate people into the process

• If possible, exploit social networks (community guided learning) to improve data classification and solutions

• Challenges:– Lack of common machine learning toolkits for smartphones– Lack of large-scale public data sets – Lack of public repository for sharing data sets, code, and tools

SENSE

LEARN

INFORM, SHARE, PERSUASION

Inform, Share, Persuasion

• Sharing– Data visualization, community awareness, and social networks

• Personalized services– Profile user preferences, recommendations, persuasion

• Persuasive technology – systems that provide tailored feedback with the goal of changing user’s behavior– Motivation to change human behavior (e.g., healthcare,

environmental awareness)– Methods: games, competitions, goal setting– Interdisciplinary research combining behavioral and social

psychology with computer science

SENSE

LEARN

INFORM, SHARE, PERSUASION

Privacy Issues

• Respecting the privacy of the user is the most fundamental responsibility of a mobile sensing system

• Current solutions– Cryptography– Privacy-preserving data mining– Processing data locally versus cloud services– Group sensing applications is based on user

membership and/or trust relationships

Privacy Issues

• Reconstruction type attacks– Reverse engineering collected data to obtain invasive

information • Second-hand smoke problem– How can the privacy of third parties be effectively protected

when other people wearing sensors are nearby?– How can mismatched privacy policies be managed when

two different people are close enough to each other for their sensors to collect information?

• Stronger techniques for protecting people’s privacy are needed

Understanding Smartphone Sensors:accelerometer, compass, gyroscope,

location, etc

Smart Phone/Pad Sensors

GalaxyNexus iPhone4 iPhone5 Samsung

Galaxy S3SamsungGalaxy S4

Galaxy Tab/ iPad2

Accelerometer O O O O O O

Magnetometer (Compass) O O O O O O

Gyroscope O O O O O O

Light O O O O O O

Proximity O O O O O O

Camera O O O O O O

Voice O O O O O O

Pressure (Barometer) O O O

Humidity/IR/Temperature O

Smartphone Sensors

• Accelerometer• Magnetometer (digital compass)• Gyroscope• Light• Pressure (Barometer)• Proximity• Camera• Voice• Temperature/Humidity/IR Gesture (Galaxy S4)

Android APIs

• Package: android.hardware• Classes:– SensorManager – android service– Sensor – specific sensor– SensorEvent – specific event of the sensor = data

SensorManager

• Most sensors interfaced through SensorManager or LocationManager – Obtain pointer to android service using

Context.getSystemService(name)

– For name, use constant defined by Context class • SENSOR_SERVICE for SensorManager • LOCATION_SERVICE for LocationManager

• Check for available sensors using List<Sensor> getSensorList(int type)

– Type constants provided in Sensor class documentation

SensorManager

• Use getDefaultSensor(int type) to get a pointer to the default sensor for a particular type Sensor accel =

sensorManager.getDefaultSensor( Sensor.TYPE_ACCELEROMETER);

• Register for updates of sensor values using registerListener(SensorEventListener, Sensor, rate)

– Rate is an int, using one of the following 4 constants • SENSOR_DELAY_NORMAL (delay: 200ms)• SENSOR_DELAY_UI (delay: 60ms)• SENSOR_DELAY_GAME (delay: 20ms)• SENSOR_DELAY_FASTEST (delay: 0ms)

– Use the lowest rate necessary to reduce power usage• Registration will power up sensor:

mSensorService.enableSensor(l, name, handle, delay);

SensorManager

• Unregister for sensor events using unregisterListener(SensorEventListener, Sensor) or unregisterListener(SensorEventListener)

• Undregistering will power down sensors:mSensorService.enableSensor(l, name, handle, SENSOR_DISABLE)

• Perform register in OnResume() and unregister in OnPause() to prevent using resources while your activity is not visible

• SensorListener is deprecated, use SensorEventListener instead – See documentation for Sensor, SensorManager, SensorEvent and

SensorEventListener

SensorEventListener

• Must implement two methods onAccuracyChanged(Sensor sensor, int accuracy) onSensorChanged(SensorEvent event)

• SensorEvent – int accuracy – Sensor sensor – long timestamp • Time in nanoseconds at which event happened

– float[] values • Length and content of values depends on sensor type

API – Setup

public class MainActivity extends Activity implements SensorEventListener {..

private SensorManager sm = null;…

public void onCreate(Bundle savedInstanceState) {..

sm = (SensorManager) getSystemService(SENSOR_SERVICE);}

protected void onResume() {..List<Sensor> typedSensors = sm.getSensorList(Sensor.TYPE_LIGHT);// also: TYPE_ALLif (typedSensors == null || typedSensors.size() <= 0) … error…sm.registerListener(this, typedSensors.get(0),

SensorManager.SENSOR_DELAY_GAME);// Rates: SENSOR_DELAY_FASTEST, SENSOR_DELAY_GAME,// SENSOR_DELAY_NORMAL, SENSOR_DELAY_UI

}}

API – Processing Events

It is recommended not to update UI directly!

public class MainActivity extends Activity implements SensorEventListener {..private float currentValue;private long lastUpdate;…public void onSensorChanged(SensorEvent event) {

currentValue = event.values[0];lastUpdate = event.timestamp;

}..

}

API – Cleanup

public class MainActivity extends Activity implements SensorEventListener {…

protected void onPause() {…

sm.unregisterListener(this);}

…

protected void onStop() {…

sm.unregisterListener(this);}

..}

Accelerometer

Mass on spring

Gravity Free Fall Linear Acceleration Linear Accelerationplus gravity1g = 9.8m/s2

-1g1g

Accelerometer

STMicroelectronics STM331DLH three-axis accelerometer (iPhone4)

Accelerometer

• Sensor.TYPE_ACCELEROMETER• Values[3] = m/s2, measure the acceleration

applied to the phone minus the force of gravity (x, y, z)– GRAVITY_EARTH, GRAVITY_JUPITER,– GRAVITY_MARS, GRAVITY_MERCURY,– GRAVITY_MOON, GRAVITY_NEPTUNE

Compass

• Magnetic field sensor (magnetometer)

ZX

Y

XY

Z

3-Axis Compass?

Magnetic inclination

Horizontal

Gravity

Magneticfield

vector

Magnetic declination

Magneticnorth

Geographicnorth

Compass

Hall Effect 3-Axis Compass

Compass

• Sensor.TYPE_MAGNETIC_FIELD• values[3] = in micro-Tesla (uT), magnetic field

in the X, Y and Z axis• SensorManager’s constants– MAGNETIC_FIELD_EARTH_MAX: 60.0– MAGNETIC_FIELD_EARTH_MIN: 30.0

Orientation Sensor• Sensor.TYPE_ORIENTATION

– A fictitious sensor: orientation is acquired by using accelerometer and compass • Deprecated

– Now use getOrientation (float[] R, float[] result)• Values[3] – (Azimuth, Pitch, Roll)

– Azimuth, rotation around the Z axis• 0 <= azimuth <= 360, 0 = North, 90 = East, 180 = South, 270 = West

– Pitch, rotation around the X axis• -180 <= pitch <= 180• 0 = sunny side up• 180, -180 = up side down• -90 = head up (perpendicular) • 90 = head down (perpendicular)

– Roll, rotation around the Y axis• -90<=roll <= 90• Positive values when the z-axis moves toward the x-axis.

Orientation: Why Both Sensors?

• We need two vectors to fix its orientation! (gravity and magnetic field vectors)

Tutorial: http://cache.freescale.com/files/sensors/doc/app_note/AN4248.pdf

Gyroscope

• Angular velocity sensor– Coriolis effect – “fictitious force” that acts upon a freely moving object as

observed from a rotating frame of reference

Gyroscope

Gyroscope

• Sensor.TYPE_GYROSCOPE• Measure the angular velocity of a device• Detect all rotations, but few phones have it– iPhone4, iPad2, Samsung Galaxy S, Nexus S– Values[] – iPhone 4 gives radians/sec, and makes it

possible to get the rotation matrix

Accelerometer vs. Gyroscope

• Accelerometer– Senses linear movement, but worse rotations, good for tilt

detection, – Does not know difference between gravity and linear

movement• Shaking, jitter can be filtered out, but the delay is added

• Gyroscope– Measure all types of rotation– Not movement– Does not amplify hand jitter

• A+G = both rotation and movement tracking possible

How to Use the Data – Example

float[] matrixR = new float[9];float[] matrixI = new float[9];

SensorManager.getRotationMatrix(matrixR, matrixI,matrixAccelerometer, matrixMagnetic);

float[] lookingDir = MyMath3D.matrixMultiply(matrixR,new float[] {0.0f, 0.0f, -1.0f}, 3);float[] topDir = MyMath3D.matrixMultiply(matrixR,new float[] {1.0f, 0.0f, 0.0f}, 3);GLU.gluLookAt(gl,0.4f * lookingDir[0], 0.4f * lookingDir[1], 0.4f * lookingDir[2],lookingDir[0], lookingDir[1], lookingDir[2],topDir[0], topDir[1], topDir[2]);

Accelerometer Noise - Simple

• Exponential moving average

const float kFilteringFactor = 0.1f; //play with this value until satisfied

float accel[3]; // previous iteration

//acceleration.x,.y,.z is the input from the sensoraccel[0] = acceleration.x * kFilteringFactor + accel[0] * (1.0f - kFilteringFactor);accel[1] = acceleration.y * kFilteringFactor + accel[1] * (1.0f - kFilteringFactor);accel[2] = acceleration.z * kFilteringFactor + accel[2] * (1.0f - kFilteringFactor);

result.x = acceleration.x - accel[0];result.y = acceleration.y - accel[1];result.z = acceleration.z - accel[2];

Return result;

Accelerometer Noise – Notes

• If it is too slow to adapt to sudden change in position, do more rapid changes when angle(accel, acceleration) is bigger

• You can throw away single values that are way out of average.

• |acc| does not have to be equal to |g| !• Kalaman filters – too complicated?

Other sensors

• Light sensor• Proximity sensor• Pressure sensor• Temperature/Humidity/IR Gesture (Galaxy S4)

Light sensor

• Sensor.TYPE_LIGHT• values[0] = ambient light level in SI lux units• SensorManager’s constants

– LIGHT_CLOUDY: 100– LIGHT_FULLMOON: 0.25– LIGHT_NO_MOON: 0.001– LIGHT_OVERCAST: 10000.0 (cloudy)– LIGHT_SHADE: 20000.0– LIGHT_SUNLIGHT: 110000.0– LIGHT_SUNLIGHT_MAX: 120000.0– LIGHT_SUNRISE: 400.0

Proximity sensor

• Sensor.TYPE_PROXIMITY• values[0]: Proximity sensor distance measured

in centimeters (sometimes binary near-far)

Temperature sensor

• Sensor.TYPE_TEMPERATURE• values[0] = temperature

Pressure sensor

• Sensor.TYPE_PRESSURE• values[0] = pressure• no constants