25/7/2014

1

LiveLabs: Building An In-Situ Real-Time Mobile Experimentation Testbed

Talk at HotMobile 2014Santa Barbara, Feb 27th 2014

WHAT PROBLEM DOES LIVELABS SOLVE?

Amazing mobile sensing app!!MobiSys deadline < 3 weeks!How to test with real users?

Lets just test it with lab usersClaim it is “real-world”Reviewers won’t know better

Wow! They did know better!Need access to real venuesWith real users on real devices

HOW???

25/7/2014

2

LIVELABS IN ACTION (WHEN OPERATIONAL)

30,000 opt-in consumers

Retail & Consumption

Leisure & Tourism

Telco & IDM

Multiple Urban Venues & Lifestyle Verticals

Mall@Singapore SentosaChangi AirportSMU

Resource-efficient deep context

collection

Real-time mobile analytics & insights

Real World experimentation

LIVELABS: PARTICIPANTS & VENUES

25/7/2014

3

LIVELABS DATA FLOW

Internet Cloud

Real-timeAnalytics Server

ExperimentationServer

Results Server

Investigators

LiveLabs Urban Lifestyle Innovation Platform

LiveLabs ContextCollection application Installed in smart phones

External Analytics Providers(eg. LARC,

IBM, Accenture,..)Specify

Interventions

SPECIFYING EXPERIMENTS : CURRENT THOUGHTS

What location (e.g., SMU, 1st

floor of Mall) does the participant need to be on

STATIC Demographic criteria (age, gender, occupation, etc.)

DYNAMIC Context Criteria (e.g.., person sitting down in the foodcourt)

Intervention—e.g., a pop-up “Notification” on screen with a discount coupon

A Simple Web-based Interface forSpecifying Targeted, Context-Based Lifestyle Experiments/Services

25/7/2014

4

CURRENT STATUS

• LiveLabs@SMU operational since Sep 2012 (actually Apr 2012) • Approx. 2000 participants signed up; approx. 700 active participants• Data collection for Android and iOS platforms deployed

• Campus-wide Indoor Location Tracking• Longitudinal traces of over 4000+ individual devices using server-side location• Controlled activation of fine-grained client-side location (Android)

• Developed Analytics over Mobile Data• Queuing Detection: Research prototype tested• Group Detection: Under active R&D

• Interventions/Promotions• Merchant promotions provided to participants via in-house built SMUddy App• First end-to-end experiment to run next week!

• Fair Amount of Visibility These Days ( Double edged)• Appeared in slides by Samsung US Researcher yesterday morning (woo hoo!!)

1. Deep, energy-efficient, continuous, context collection

2. Continuous indoor location tracking in public spaces

3. Derive Deep Analytics from Context

4. Run automated social experiments on mobile devices

5. Handle transient network traffic loads

LiveLabs: Key Component Technologies

• Clients for Android, iOS, Phone8 .• Server-controlled capture of phone events (e.g., SMS, URLs) & sensor data

• Client-side +-3m accuracy for Android.

iOS

• Client-side +-3m accuracy for Android.• Server-side tracking for all platforms (e.g., iOS, Phone 8) ~ inter-AP distance

• Real-time Queue Detection System.• Detection of Dynamic Groups from Spatiotemporal trajectories

• Intervention Management Portal (v1)

ads/promotions.

• Intervention Management Portal (v1) allows location & time-based delivery of ads/promotions.

• Use of TV Whitespace and real-time RF Mapping technologies under investigation

Key Research Challenges Current Innovations/Capabilities

25/7/2014

5

RESEARCH HIGHLIGHT 1: QUEUE DETECTION

• New urban applications

• “Find the vendor withshortest waiting time”

• “Provide discounts tocustomers with long expected wait times”

• “Trivial” with fine-grained location info

• Not possible in practice

• Solution: Use accelerometer and infer queuing

• Does it actually work?

Measured Service Times are Highly Variable

QUEUE DETECTION PERFORMANCE

Still Possible to Achieve Good Results

F&B Outlet on Campus F&B Outlet on Campus

Tested on 15 different occassions at various locations (coffee shops, movie theaters, taxi stands, etc.) in Singapore and Japan

25/7/2014

6

RESEARCH HIGHLIGHT 2: GROUP DETECTION

• Fast and accurate group detection is very useful!• For timely recommendations and contextual reasoning

GROUP DETECTION PERFORMANCE

• Existing Solutions are not very effective• Trajectory analysis? Limited location infra + collocation of non-group

people

• Bluetooth scans? High power use + miss-detection (esp. when crowded)

• Semantic transition and sensor-driven features are effective!

VenueRecall(%)

Precision(%)

CoEX Mall 61.16 92.15

PlazaSing 68.48 97.39

* at 10 minutes latency

25/7/2014

7

ALL THAT GLITTERS IS NOT GOLD

• Taken a long *long* time to get to this stage• Idea was conceived in late 2009

• Funded in 2011

• Launched in 2012

• First “complete” test with actual users Next Week

• Quite a few technical and administrative challenges

• Most are “Obvious” in hindsight

• Let me share our pain so that you won’t have to go through it

CHALLENGE 1: INDOOR LOCATION• Our naïve initial position

20 years of work in this area. Must be solved! Lets just reuse what others have done

• Realisation: How come the results are not great? Heterogeneity of devices and environments makes the accuracy much *MUCH* worse!

RSSI is a terrible input Closed platforms make sensor fusion quite hard

• Big takeaway (wisdom so far) Inter-AP distance accuracy possible with low energy

across all devices (server-side techniques) Better accuracy very hard without significant energy and

programming costs (sensor fusion etc.)

25/7/2014

8

CLIENT SIDE ANOMALIES (AP INCONSISTENCIES)

Procedure: Use Same Device (Galaxy S 4) at the Same Place at the Same Time Measuring the Same Set of APs

Implication: Fingerprints / Models Need to be quite Dynamic!

AP 1 : Day 1 Highest AP 2 : Day 2 Highest AP 3 : Day 3 Highest

CLIENT SIDE ANOMALIES (PROXIMITY EFFECTS!)

Procedure: Measure each device individually at the same spot for 2 mins. Then measure both devices side by side for 2 mins.

Implication: Not Sure! Makes location much harder for groups!

Expected Result S4s Have Switched Gains!

S3s and S4s Have The Same Effect

25/7/2014

9

SERVER SIDE IS NOT MUCH BETTER!! (SIGH..)

Procedure: Measure RSSI at AP over a period of 3 hours. Device is connected and left at main screen (allowed to lock / power down)

Implication: Some devices are very hard to track! (lack of updates)

Some devices are quite noisy (errors in location)

Note 2: Lots of Updates + Stable

HTC One: Lots of Updates but NOISY!!

iPhone 5S: Few Updates & Noisy!

CHALLENGE 2: CONTINUOUS SENSING• Our naïve initial position

Lets turn everything on at full rate and collect the kitchen sink!

How hard can it be??

• Answer: Very Hard!!

Energy cost of individual sensors is large

Energy cost of multiple sensors may not be linear

Energy cost of multiple tasks is dominated by the most expensive task

Heterogeneity of devices and tasks makes situation much *much* harder!

25/7/2014

10

ENERGY COST OF INDIVIDUAL SENSORS

0

20

40

60

80

100

120

140

160

slowest slow fast fastest

Po

we

r C

on

sum

pti

on

(m

W)

Sensing Rate (4 default modes on android)

AccelerometerGyroscopeCompass

IT GETS WORSE WITH PROCESSING & STORAGE!!

0

50

100

150

200

250

300

350

400

450

500

slowest slow fast fastest

Po

we

r C

on

sum

pti

on

(m

W)

Sensing Rate (4 default modes on android)

Accel with Internal Flash StorageAccel w/o Internal Flash StorageLight with Internal Flash StorageLight w/o Internal Flash Storage

2x higher

5x higher!!

25/7/2014

11

ENERGY COSTS MAY NOT BE LINEAR

0

200

400

600

800

1000

1200

1400

slowest slow fast fastest

Po

we

r C

on

sum

pti

on

(m

W)

Sensing Rate (4 default modes on android)

All inertial sensors (accel, gyro, compass)Inertial + location + others (pressure, light)

no differenceLarge sub linear increase

Large non linear increase

CHALLENGE 3: PARTICIPANT RETENTION• Our naïve initial position

This is so cool & We LOVE IT!! Surely, everyone will opt in and stay!

• Realisation: No they won’t!

Drop out rate was very high at the start

Even with payment incentives!! (not obvious initially)

One main reason was lack of compelling usage

Solution: Develop Cool in-house customised apps

Challenge: Cool is relative and developing apps requires a lot more work

25/7/2014

12

COOL APP 1: SMU BUDDY (SMUDDY)

Heatmaps Showing Free Spots on Campus

Friend finder + location and time based

messaging

Exclusive promotions from Campus vendors

0

200

400

600

800

1000

1200

1400

1600

1800

2000

Au

g 1

3A

ug

13

Se

p 1

3S

ep

13

Se

p 1

3S

ep

13

Se

p 1

3O

ct 1

3O

ct 1

3O

ct 1

3O

ct 1

3N

ov

13

No

v 1

3N

ov

13

No

v 1

3D

ec

13

De

c 1

3D

ec

13

De

c 1

3Ja

n 1

4Ja

n 1

4Ja

n 1

4Ja

n 1

4Ja

n 1

4F

eb

14

Fe

b 1

4F

eb

14

Fe

b 1

4

No

. Of

Pa

rtic

ipa

nts

Month of the Year

Registered Users

Users Installing LiveLabs Applications

THE SLOW GROWTH TO “SUCCESS”Opportunity!!<small print>Or major problem <\small print>

25/7/2014

13

MOST SERIOUS CHALLENGE: ADMINISTRATION• Our (Archan and myself) naïve initial position

We can run everything (people, vendors, research) ourselves

• Realisation: No we can’t

This lab is like a startup. Way too many moving parts

Almost burnt out handling everything

Production is quite different from research Client management takes *SOOO* much time!

Solution: Hire dedicated PMs, lab manager (accountant), business development managers

LIVELABS: LESSONS LEARNED UP TO NOW (SUMMARY)

• Technical• Indoor Location Tracking is Not a Solved Problem

• Too many real-world anomalies with existing techniques

• Cannot do Continuous Mobile Sensing• Large amounts of low fidelity sensing with burst of high fidelity sensing

• The Tail Really Does Matter!• Venue operators prefer solutions with no fluctuation (even if base is

worse)

• Administration• Attracting Participants is Easy, Retention is Hard!

• Need to find what motivates participants to stay on (apps in our case)

• Production, Research, and Administration Do Not Mix!• Needed separate teams for each to ensure quality and prevent burnout

25/7/2014

14

ACKNOWLEDGEMENTS

Building something like LiveLabs requires standing on the shoulders of many other people

• Faculty – Archan Misra, Youngki Lee

• Post-docs – Rijurekha Sen, Victor Lu

• Ph.D. Student – Kartik Muralidharan, Sougata Sen, Joseph Chan, Nguyen Huynh

• Professional Staff – Jonathan Wang, Kenneth Fu, Kazae Quek, Sipei Huang

• Engineering – Swetha Gotipati, Le Gai Hai, Kasthuri Jayarajah, William Tan, Jeena Sebastian, Vignesh Subbaraju, Nguyen Minh, KohQuee Boon, Sriguru Nayak

• Many more people + interns have worked with us over the years

SUMMARY

• LiveLabs aims to change 4 real-world venues into living testbeds

• Real people with real devices in real environments performing real actions in real time

• We are live at SMU and going live at the airport soon

• 2000 sign ups at SMU

• 1st end-to-end experiment going live next week

• Free to use and open to all (at least at SMU)

25/7/2014

15

FOR MORE DETAILS & TO USE LIVELABS

Come talk to myself, Archan, or Youngki.

Or contact me at rajesh@smu.edu.sg and/or visit

http://www.livelabs.smu.edu.sg

Come test your applications, sensing technologies, and other mobile phone related solutions with us

We are hiring! Post-docs, research engineers, and Ph.D. students (in all areas of systems development and

research)

Please contact me if you are interested.