Lecture 4: People-centric Sensing Cristian Borcea

Department of Computer Science

NJIT

2

Enable accurate real-time monitoring of the physical world

Consist of homogeneous, static, tiny sensors

Sink Internet, Satellite

Task Manager

End user

Sensor node

Water pollution in the river Fire in the forest Structural integrity of the bridge

How can we benefit from sensing in our

daily routine?

Leverage existing “infrastructure”: smart phones and vehicular systems No deployment cost

Excellent coverage: billions of devices distributed everywhere

3

Tornado

approaching!

Highway

Parking Area

Shopping

mall

Traffic jam!

Local fog

patches!

Free parking spot!

50%

discount!

People-centric: people are both consumers and providers of sensed data

On phones: camera, microphone, GPS, accelerometer, digital compass, light sensor, Bluetooth, [future: pollution sensors, medical sensors]

On cars: all car sensors can be made available to the vehicular system

Owners can provide information as well (i.e., human sensors)

4

Monitoring pollution in urban areas

Beneficiary: environment protection agencies

Monitoring vehicular traffic to detect/prevent traffic jams

Beneficiary: departments of transportation for planning, all of us for

improved traffic conditions

Finding the locations of potholes on the roads

Beneficiary: municipalities or departments of transportation

Determining the effect of lifestyle (meeting people, level

of exercise) on people’s wellbeing

Beneficiary: medical/social researchers, all of us

Documenting public events in real-time

Beneficiary: news outlets, all of us 5

On phones: free SDKs for development & app stores for deployment Could easily deploy sensing applications at scale

On cars: no standard vehicular system yet Care must be taken to ensure apps do not interfere with

car safety 6

Introduction

Smart phone sensing

Micro-Blog @ Duke

CenceMe @ Dartmouth

Activity recognition on smart phones

Vehicle sensing

Future challenges

7

Internet

Micro-Blog vision: virtual information telescope [Ref. 4]

8

Users encouraged to blog on mobile phones Video, audio, pictures, text, etc.

Micro-Blog phone client geo-tags blog

Uploads to server over WiFi/GPRS

Micro-Blog server positions blog on Google Maps

Internet users zoom into maps

Witness blog streams across the world

9

Virtual Telescope

Cellular, WiFi Visualization Service

Web Service

Phones People

Physical Space

Some queries participatory Is beach parking available?

Others are not Is there WiFi at the beach café?

Phones reply to query & reply posted on Google Map as new micro-blog

10

Content generated at a certain region is automatically

downloaded on user smart phone when user enters region

Example

User X creates micro-blog about restaurant food

“Floats” micro-blog at the restaurant

User Y arrives at restaurant

X’s micro-blog downloaded onto Y’s phone

Y can modify content and “re-float”

11

Continuous GPS (8 hours battery life) WiFi, GSM localization improves energy (16, 40 hours)

Degrades localization accuracy (40, 500m)

Solution: switch between them as function of app needs

WiFi

GSM GPS

Time (in minutes) 12

Phones need to continuously update their location Poses privacy risks

Pseudonyms insufficient

Proposed 3 blogging modes

Public, Social, Private

Users set privacy policy

In social mode, only those in social network view blogs

For querying

Privacy feasible through K-anonymity based solutions

13

14

Leverage sensors on the phones and social networks to provide context/activity inferences

15

16

Introduction

Smart phone sensing

Activity recognition on smart phones

Darwin phones @ Dartmouth

Vehicle sensing

Future challenges

17

Machine learning techniques applied on phone sensor data to

determine user activity

E.g., sitting, walking, running, in a car, etc.

Done on phone or server (on-line or off-line) function of amount of

resources required

Data exchange between nearby phones could enable group activity recognition

Examples of applications:

Is the user exercising enough? (sensors: GPS & accelerometer)

Has the user a good social life? (sensors: Bluetooth and microphone)

Deriving higher-level context information

▪ Is the user in an important meeting? (sensors: Bluetooth,

localization + social network) 18

19

Calibration, uncontrolled phone context, computational efficiency, battery power

Example of uncontrolled phone context: one activity, two different patterns

Cycling while phone in pants pocket (A) and backpack (B)

20

microphone

camera

GPS/WiFi/

cellular

air quality

pollution

social context

audio / pollution / RF

fingerprinting

image / video

manipulation

classification model evolution

classification model pooling

collaborative inference

Darwin

Applies distributed computing and collaborative inference to activity recognition

Why Darwin? Same classifier model for all users doesn’t work; even same classifier

for same user doesn’t work when changing context

Creating one classifier model per user doesn’t scale

21

4 phases

initial training (derive model seed)

classification model evolution

classification model pooling

collaborative inference

supervised

unsupervised

Darwin was tested on voice recognition 22

23

phone: feature extraction (low computation)

backend

backend: model training (high computation)

24

phone determines when to evolve

match?

NO

evolve (train new model using

backend as before)

training sampled

Speaker A’s model

Phone A Phone B

Phone C

Speaker B’s model

Speaker C’s model

Speaker C’s model

Speaker A’s model

Speaker B’s model

Speaker B’s model

Speaker A’s model

Speaker C’s model

Re-use already available classifiers instead of training a new model for each speaker Saves battery power and reduces delay

25

Phone A Phone B

Phone C

User A speaking!!!

A’s LI results:

Prob(A speaking) = 0.65

Prob(B speaking) = 0.25

Prob(C speaking) = 0.10

C’s LI results:

Prob(A speaking) = 0.30

Prob(B speaking) = 0.67

Prob(C speaking) = 0.03

B’s LI results:

Prob(A speaking) = 0.79

Prob(B speaking) = 0.11

Prob(C speaking) = 0.10

Local inference can be misleading

Use collaborative inference to reach common decision

26

Collaborative inference & classification model evolution boost performance The more phones the better 27

• Indoor, quiet scenario • 8 people around a table

Introduction

Smart phone sensing

Activity recognition on smart phones

Vehicle sensing

Pothole Patrol @ MIT

ParkNet @ Rutgers

Future challenges

28

Determine which roads need to be fixed Challenges

Differentiate potholes from other road anomalies (railroad crossings, expansion joints)

Cope with variations in detecting same pothole (speed, sensor orientation)

Vehicles have GPS and 3-axis accelerometer <time,location,speed,heading,3-axis acceleration>

Detection based on accelerometer data

Embedded Linux system + WiFi/Cellular installed in 7 taxis

29

Pothole Record

Clustering Algorithm

Cab 1

GPS

3 Axis Accelerometer

Location Interpolator

Pothole Detector

Cab 2

GPS

3 Axis Accelerometer

Location Interpolator

Pothole Detector

Central Server

30

Use classifier Hand-labeled data for training

Extend training set with loosely-labeled training data (type & frequencies, but not exact locations)

Clustering (at least K events must happen at same location with small error)

31

On labeled data:

32

On unlabeled data (with spot verification):

Provide a high-level view of parking availability to drivers on the road Single sensor vehicle senses multiple parking spots

33

34

Sensor vehicle instrumented with Computer & WiFi/cellular communication

Ultrasonic sensor: 20 readings per sec; range: 6.5m

GPS: 5 readings per sec

Webcam: capturing ground truth for evaluation

Obstacles: ‘dips’ in ultrasonic sensor readings Use classifier to differentiate parked cars from

other objects

35

36

Slotted (marked) spots

Unslotted spots Difficult to estimate if a car fits in the sensed spot

Introduction

Smart phone sensing

Activity recognition on smart phones

Vehicle sensing

Future challenges

37

Dependability How can “clients” trust data received from mobile sensors?

▪ How to validate these data? Should use reputation mechanism?

▪ How to determine ground truth?

Sensor calibration?

Privacy How to provide anonymity?

▪ How to balance the need for anonymity with the need for reputation?

Incentives Users motivated by self-interest for some apps (e.g., traffic jams)

Why should they provide data if there is no clear self-interest?

▪ Get paid? How to get paid while maintaining anonymity?

▪ Need for fair exchange protocol 38

Programmability Current systems are application-specific; How to quickly develop and

deploy new applications?

How to decompose global distributed sensing tasks into individual tasks?

▪ E.g., ensure spatial and temporal distribution of sensed data?

How can device owners control where, when, and how much resources are consumed for sensing?

Scheduling and real-time constraints What parameters should be considered in task scheduling?

▪ Payment, real-time constraints, spatio-temporal distribution of data providers?

▪ Should a provider be paid if the answer comes late? 39

1. Urban sensing @ UCLA: http://urban.cens.ucla.edu/

2. MetroSense project @ Dartmouth:

http://metrosense.cs.dartmouth.edu/

3. Urbanets: http://cs.njit.edu/~borcea/papers/ieee-pervasive07.pdf

4. Micro-Blog: http://synrg.ee.duke.edu/microblog.html

5. CenceMe:

http://www.cs.dartmouth.edu/~sensorlab/pubs/cenceme_sensys08.pdf

6. Darwin phones: http://www.ists.dartmouth.edu/library/478.pdf

7. Pothole patrol: http://db.csail.mit.edu/pubs/mobisys08.pdf

8. ParkNet:

http://www.winlab.rutgers.edu/~suhas/SuhasMathur_Mobisys2010.pdf

40

1. Two decades of mobile computing

2. Infrastructure support for mobility

3. Mobile social computing

4. People-centric sensing

5. Programming mobile ad hoc networks Migratory Services: context-aware client-service programming

Spatial Programming: location-aware imperative programming

Contory: SQL-like declarative programming

6. Vehicular computing and networking

7. Privacy and security in mobile computing

41