Open Source Smartphone Libraries for Computational Social Science

@neal_lathia, k. rachuri, c. mascolo, g. roussosseptember 2013

“...each of these transactions leaves digital traces that can be compiled into comprehensive pictures of both individual and group behaviour...

“Computational Social Science” Lazer et. al

AccelerometerMicrophoneCameraGPSCompassGyroscopeWi-FiBluetoothProximityNFCLight

“... a number of challenges remain in the development of sensor-based applications [...] there is mixed API and operating system (OS) support to access the low-level sensors...”

“A Survey of Mobile Phone Sensing,” Lane et. al

● Battery-friendly sensor data collection● Triggering notifications● Data storage & transmission

“Reinventing the Wheel”

All smartphone-based research needs to begin by engineering solutions for:

if you don't like code, turn away now

● Current approach:– Requires treating each sensor with different code– Hides battery & energy efficiency requirements

● We have built Android ESSensorManager– Everything is a “sensor”– Simple API with two modes (get, subscribe)– API exposes battery issues to programmer

Sensor Data Collection

● Pull Sensors– Accelerometer, Location, Microphone– Wi-Fi, Bluetooth, Camera– Active apps, SMS/Call Log Content

● Push Sensors– Battery, Connection State

– Proximity, Screen– Phone Calls/SMS Events

Everything as a 'Sensor'

aim: get data in 2 lines of code.

// get the instance

ESSensorManager sm = ESSensorManager.getSensorManager(context)

// ask for some data

MicrophoneData data

= (MicrophoneData) sm.getDatafromSensor(

SensorUtils.SENSOR_TYPE_MICROPHONE)

Get data, fast

aim: quickly configure sensing & respond to battery

// make a subscription

int sid = sm.subscribeToSensorData(

SensorUtils.SENSOR_TYPE_MICROPHONE, listener)

// deal with data pushed to you

class Listener implements SensorDataListener

{

public void onDataSensed(SensorData d){..}

public void onCrossingLowBatteryThreshold(..) {..}

}

Get data, continuously

● “I need to ask the user for some interaction...”– Based on time of day, randomly– Based on a sensor event– ESTriggerManager

● “I need the data to come back to me!”– Without using up all of my participant's 3G

connection– ESDataManager

Problems #2 & #3

● JSON formatting– Flexible for various sensors

– Includes sensing configuration{

"zAxis":[9.959879,9.959879,....],

"senseStartTime":"17:07:00:281 16 05 2013 -0500 CDT",

"sampleLengthMillis":8000,

"xAxis":[0.11492168,0.11492168,0.0766144548,...],

"yAxis":[0.11492168,0.11492168,0.0766144548,...],

"sensorTimeStamps":[1368742020298,1368742020488,....],

"sensorType":"Accelerometer"

}

Data Management

● Simple on-phone data querying– Only queries local data

– “Give me all the accelerometer data from the last hour...”

Data Management

● Configurable Transfer Policy– Do not transfer (local storage only)

– Transfer immediately (or fail)

– Asynchronous transfer (Wi-Fi & timeout)

Data Management

● Student Project at Birkbeck College, London– Post-graduate students with programming

experience

– Sample Audio data from the environment– Measure noise pressure/audio features– Post data to COSM (Xively)

Preliminary Feedback

● Why wasn't it used?– Lack of experience; support for emulator

● Using the library– Easy, quick

– Substantially less code (~ better quality): focus is on app features, not sensor engineering

● What is missing?– Data filters; inference algos; simulated data

Preliminary Feedback

“it is useful for the research community to think about and propose sensing abstractions and APIs ...”

“A Survey of Mobile Phone Sensing,” Lane et. al

Open Source Smartphone Libraries for Computational Social Science

http://emotionsense.orghttps://github.com/nlathia/SensorManager https://github.com/nlathia/TriggerManager https://github.com/nlathia/SensorDataManager