1 surroundsense: mobile phone localization via ambience fingerprinting

1

SurroundSense: Mobile Phone Localization via Ambience Fingerprinting

2

Context

Pervasive wireless connectivity+

Localization technology=

Location-based applications

3

Location-Based Applications (LBAs)

For Example: GeoLife shows grocery list when near Walmart MicroBlog queries users at a museum Location-based ad: Phone gets coupon at Starbucks

iPhone AppStore: 3000 LBAs, Android: 500 LBAs

4

Most emerging location based apps do not care about the physical location

GPS: Latitude, Longitude

5

Most emerging location based apps do not care about the physical location

Instead, they need the user’s logical location

GPS: Latitude, Longitude

Starbucks, RadioShack, Museum, Library

6

Physical Vs Logical

Unfortunately, most existing solutions are physical

GPS GSM based SkyHook Google Latitude

RADAR Cricket …

7

Given this rich literature,

Why not convert from Physical to Logical Locations?

8

Physical LocationError

9

Pizza HutStarbucks

Physical LocationError

10

Pizza HutStarbucks

Physical LocationError

The dividing-wall problem The dividing-wall problem

11

SurroundSense:A Logical Localization Solution

12

It is possible to localize phones by sensing the ambience

It is possible to localize phones by sensing the ambience

Hypothesis

such as sound, light, color, movement, WiFi …

13

It is possible to localize phones by sensing the ambience

It is possible to localize phones by sensing the ambience

Hypothesis

such as sound, light, color, movement, WiFi …

14

Multi-dimensional sensing extracts more ambient information

Any one dimension may not be unique, but put together, they may provide a

unique fingerprint

15

SurroundSense

Multi-dimensional fingerprint Based on ambient

sound/light/color/movement/WiFi

Starbucks

Wall

Pizza Hut

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

16

BB AACC DDEE

Should Ambiences be Unique Worldwide?

FFGG

HHJJ

II

LLMMNN

OO

PPQQ

QQ

RR

KK

17

Should Ambiences be Unique Worldwide?

BB AACC DDEE

FFGG

HHJJ

II

KKLL

MMNNOO

PPQQ

QQ

RR

GSM provides macro location (strip mall) SurroundSense refines to Starbucks

18

Economics forces nearby businesses to be diverse

Not profitable to have 3 adjascent coffee shopswith same lighting, music, color, layout, etc.

SurroundSense exploits this ambience diversity

Why does it work?

The Intuition:

19

++

Ambience Fingerprinting

Test Fingerprint

Sound

Acc.

Color/Light

WiFi

LogicalLocation

Matching

FingerprintDatabase

==

Candidate Fingerprints

GSM Macro Location

SurroundSense Architecture

20

Fingerprints

Sound:(via phone microphone)

Color:(via phone camera)

Amplitude Values-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1

N

orm

aliz

ed

C

ount

0.14

0.12

0.1

0.08

0.06

0.04 0.02

0

Acoustic fingerprint (amplitude distribution)

Color and light fingerprints on HSL space

Lightn

ess

1

0.5

0

Hue

0

0.5

1 00.2

0.4

0.6

0.8

1

Saturation

21

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Moving

22

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Queuing Seated

Moving

23

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Pause for product browsing

Short walks between product browsing

Moving

24

Fingerprints

Movement: (via phone accelerometer)

Cafeteria Clothes Store Grocery Store

Static

Walk more Quicker stops

Moving

25

Fingerprints

Movement: (via phone accelerometer)

WiFi: (via phone wireless card)

Cafeteria Clothes Store Grocery Store

Static

ƒ(overheard WiFi APs)

ƒ(overheard WiFi APs)

Moving

26

Discussion

Time varying ambience Collect ambience fingerprints over different time

windows

What if phones are in pockets? Use sound/WiFi/movement Opportunistically take pictures

Fingerprint Database War-sensing

27

Evaluation Methodology

51 business locations 46 in Durham, NC 5 in India

Data collected by 4 people 12 tests per location

Mimicked customer behavior

28

Evaluation: Per-Cluster Accuracy

Cluster

No. of Shops

1 2 3 4 5 6 7 8 9 10

4 7 3 7 4 5 5 6 5 5

A

ccura

cy (

%)

Cluster

Localization accuracy per cluster

29

Evaluation: Per-Cluster Accuracy

Cluster

No. of Shops

1 2 3 4 5 6 7 8 9 10

4 7 3 7 4 5 5 6 5 5

Fault tolerance

A

ccura

cy (

%)

Cluster

Localization accuracy per cluster

30

Evaluation: Per-Cluster Accuracy

Cluster

No. of Shops

1 2 3 4 5 6 7 8 9 10

4 7 3 7 4 5 5 6 5 5

A

ccura

cy (

%)

Cluster

Localization accuracy per cluster

Sparse WiFi APs

31

Evaluation: Per-Cluster Accuracy

Cluster

No. of Shops

1 2 3 4 5 6 7 8 9 10

4 7 3 7 4 5 5 6 5 5

No WiFi APs

Acc

ura

cy (

%)

Cluster

Localization accuracy per cluster

32

Evaluation: Per-Scheme Accuracy

Mode WiFi Snd-Acc-WiFi Snd-Acc-Lt-Clr SS

Accuracy 70%

74% 76% 87%

33

Evaluation: User Experience

Random Person Accuracy

Average Accuracy (%)0 10 20 30 40 50 60 70 80 90 100

1 0.9 0.8 0.7 0.6 0.5

CD

F

0.4 0.3 0.2 0.1 0

WiFISnd-Acc-WiFiSnd-Acc-Clr-LtSurroundSense

34

Limitations and Future Work

Energy-Efficiency Continuous sensing likely to have a large energy

draw

35

Limitations and Future Work

Energy-Efficiency Continuous sensing likely to have a large energy

draw

Localization in Real Time User’s movement requires time to converge

36

Limitations and Future Work

Energy-Efficiency Continuous sensing likely to have a large energy

draw

Localization in Real Time User’s movement requires time to converge

Non-business locations Ambiences may be less diverse

37

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

Electroic compasses can fingerprint layout Tables and shelves laid out in different orientations Users forced to orient in those ways

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

Limitations and Future Work

38

Ambience can be a great clue about locationAmbient Sound, light, color, movement …

None of the individual sensors good enoughCombined they may be unique

Uniqueness facilitated by economic incentive

Businesses benefit if they are mutually diverse in ambience

Ambience diversity helps SurroundSenseCurrent accuracy of 89%

Conclusion

39

Questions?

40

SurroundSense evaluated in 51 business locationsAchieved 89% accuracy

But perhaps more importantly, SurroundSense scales to any part of the world

Summary

41

SurroundSense

Today’s technologies cannot provide logical localization

Ambience contains information for logical localization

Mobile Phones can harness the ambience via sensors More sensors, beter reliability

Evaluation results: 51 business locations, 87% accuracy

SurroundSense can scale to any part of the world

42

43

Evaluation: Per-Shop Accuracy

Localization Accuracy per Shop

Average Accuracy (%) 0 10 20 30 40 50 60 70 80 90 100

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

C

DF

WiFiSnd-Acc-WiFiSnd-Acc-Clr-LtSurroundSense

44

Architecture: Filtering & Matching

Candidate Fingerprints

45

Fingerprinting Sound

Fingerprint generation : Signal amplitude Amplitude values divided in 100 equal intervals Sound Fingerprint = 100 normalized values

• valueX = # of samples in interval x / total # of samples

Filter Metric: Euclidean distance Discard candidate fingerprint if metric > threshold г

Threshold г Multiple 1 minute recordings at the same location di = max dist ( any two recordings )

г = max ( di of candidate locations )

46

Fingerprinting Motion

Fingerprint generation: stationary/moving periods Sitting (restaurants, cafes, haircutters ) Slow Browsing (bookstores, music stores, clothing) Speed-Walking (groceries)

Filter Metric:

Discard candidate fingerprints with different classification

static

moving

t

tR =

0.0 ≤ R ≤ 0.2 sitting0.2 ≤ R ≤ 2.0 slow browsing

2.0 ≤ R ≤ ∞ speed-walking

47

Fingerprinting WiFi

Fingerprint generation: fraction of time each unique address was overheard

Filter/Ranking Metric Discard candidate fingerprints which do not have similar

MAC frequencies

48

Fingerprinting Color

Floor Pictures Rich diversity across different locations Uniformity at the same location

Fingerprint generation: pictures in HSL space K-means clustering algorithm Cluster’s centers + sizes

Ranking metric

49

Thinking about Localizationfrom an application perspective…

50

Emerging location based apps need place of user, not physical location

Starbucks, RadioShack, Museum, Library

Latitude, Longitude

51

Emerging location based apps need place of user, not physical location

Starbucks, RadioShack, Museum, Library

Latitude, Longitude

We call this Logical Localization …We call this Logical Localization …

52

Can we convert from Physical to Logical Localization?

53

Can we convert from Physical to Logical Localization?

State of the Art in Physical Localization:1. GPS Accuracy: 10m 2. GSM Accuracy: 100m3. Skyhook (WiFi+GPS+GSM) Accuracy: 10m-

100m

State of the Art in Physical Localization:1. GPS Accuracy: 10m 2. GSM Accuracy: 100m3. Skyhook (WiFi+GPS+GSM) Accuracy: 10m-

100m

54

Widely-deployable localization technologies have errors in the range of several meters

Widely-deployable localization technologies have errors in the range of several meters

Can we convert from Physical to Logical Localization?

State of the Art in Physical Localization:1. GPS Accuracy: 10m 2. GSM Accuracy: 100m3. Skyhook (WiFi+GPS+GSM) Accuracy: 10m-

100m

State of the Art in Physical Localization:1. GPS Accuracy: 10m 2. GSM Accuracy: 100m3. Skyhook (WiFi+GPS+GSM) Accuracy: 10m-

100m

55

Physical Vs Logical

Lot of work in physical localization

GPS (5m accuracy, outdoor only) GSM based (high coverage, 200m accuracy) SkyHook (40m accuracy, need war-driving) Google Latitude (similar)

RADAR (5m accuracy, need indoor mapping) Cricket (install per-room devices) …

Each of them has distinct problems … Each of them has distinct problems …

56

Physical Vs Logical

Lot of work in physical localization

GPS (5m accuracy, outdoor only) GSM based (high coverage, 200m accuracy) SkyHook (40m accuracy, need war-driving) Google Latitude (similar)

RADAR (5m accuracy, need indoor mapping) Cricket (install per-room devices) …

Each of them has distinct problems … Each of them has distinct problems …

57

Even if we assume the best of all solutions

(5m accuracy indoors)

logical localization still remains a challenge.

58

Contents

SurroundSense

Evaluation

Limitations and Future Work

Conclusion

59

Contents

SurroundSense

Evaluation

Limitations and Future Work

Conclusion

60

Contents

SurroundSense

Evaluation

Limitations and Future Work

Conclusion

61

Contents

SurroundSense

Evaluation

Limitations and Future Work

Conclusion

62

Contents

SurroundSense

Evaluation

Limitations and Future Work

Conclusion

63

Evaluation: Per-Cluster Accuracy

Cluster

No. of Shops

1 2 3 4 5 6 7 8 9 10

4 7 3 7 4 5 5 6 5 5

A

ccura

cy (

%)

Cluster

Localization accuracy per cluster

Multidimensional sensing

64

Limitations and Future Work

Energy-Efficiency

Localization in Real Time

Non-business locations

65

Limitations and Future Work

Energy-Efficiency Continuous sensing likely to have a large energy

draw

Localization in Real Time

Non-business locations

66

Limitations and Future Work

Energy-Efficiency Continuous sensing likely to have a large energy

draw

Localization in Real Time User’s movement requires time to converge

Non-business locations

67

Limitations and Future Work

Camera inside pockets Detect phone when out of pocket Takes pictures when camera pointing downward