scott davidoffdissertation defense familycontrolsmarthome the project on routine as resource for the...

Scott DavidoffDissertation Defense

THE PROJECT ON

Routine as Resource for the Design of Learning SystemsScott Davidoff Dissertation Defense


A pattern of behavior that is followed repeatedly, but is subject to change if conditions change

Winter 1964

“

“”2


Koestler 1967

Routines reduce our attention needs

3


Zerubavel 1981

Reduce attention needed for tasks

4


Wakkary & Maestri 2007

Free attention for bigger challenges

5


Wolin + Bennett 1984

6


The cyclic nature of routine makes it a natural target for machine learning


Machine learning can use this order

8


Suchman 1983, Tolmie et al. 2002

Idiosyncrasies are hard to model

9


Certain human routines can be modeled, increasing the scope of activity recognition

Machine learning

Opportunities

10


The ability to use learned routines in end-user applications would solve a variety of human problems

HCI

Opportunities

11


Dual-income family logistics

12


Sequence of place and transportation (rides) that occur on daily, weekly, and seasonal cycles

“

“”13


Darrah et al. 2000

Managing details can be difficult

14


Frissen 2000

Routines give a feeling of control

15


Beech et al. 2004

Life does not always follow routines

16


Breakdowns lead to loss of control

17


Perry et al. 2001, Ling + Campbell 2003

A constant need to follow updates

18


Gneezy + Rustichini 1998, Darrah 2009

A constant source of anxiety

19


We can learn a model of family logistical routines, and present that information to families to help them feel more in control of their lives

20

Scott DavidoffDissertation Defense 21


2We can use sensing and modeling to synthesize missing information resources

3Show how to use the model and evaluate the impact of the information

1We can use fieldwork to identify missing but needed information resources

Fieldwork Modeling Validation

23


Create an ordered list of places and rides

Attend to the details of a plan as it unfolds

Planning and coordinationLogistics

Coordinate

Plan

24


Fieldwork1

25


1805

22 Davidoff et al Ubicomp 2006Months

Months

Months0606

12Families

Families

Families

Davidoff et al Ubicomp 2007

Davidoff, Dey + Zimmerman CHI 2009

45 Months24Families

26


Semi-Structured Interviews

Davidoff et al. Ubicomp 2006

27

Scott DavidoffDissertation Defense scott davidoff, min kyung lee, anind dey + john zimmerman

Needs Validation



User Enactments

29


User Enactments


30


GPS

Phone Calls

Email

SMS

CalendarsDavidoff, Dey + Zimmerman CHI 2009

31


528 phone interviews

109 activity Interviews

108 calendar months


32


Less than 20% of days go exactly as planned1Routines are not documented

People have incomplete knowledge of other people’s routines

People make plans that depend on incorrect information

2

3

4

33


Family E

Routine

Deviation, Scheduled

Deviation, Unscheduled

35


Family EAll Families

Routine

Deviation, Scheduled

Deviation, Unscheduled

36


Less than 20% exactly as planned

37


Routineness of Family E Activities

38



39



40


Routines are not documentedFieldwork Finding


41



August September October November

42


People have incomplete knowledge of other people’s routines

Fieldwork Finding


43



44



45



46



47



48



49



50



51


People make plans that depend on incorrect informationFieldwork Finding


52


Orthodontist

Scouts

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

View of Plan: Dad

Check-up

Track Practice

53


Orthodontist

Scouts

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

View of Plan: S16

Paper Route

Track Practice

54


Orthodontist

Scouts

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

Who Will do the Paper Route?

Check-up

Track Practice

55


Information gaps can break down coordination

No resources exist to find needed information


56


Modeling2

57


2 31 Ride Detection

Driver Prediction

Predict Lateness

58


In Situ Observation

In Situ Observation

Maintain Current Behaviors

Maintain Current Behaviors

Ubiquitous Sensing

Ubiquitous Sensing

GPS

59


Parent drives kid to an activity

Parent drives kid from activity

Pick-ups and drop-offsRide

Pick-up

Drop-off

60


Ride Detection


Day Care

Work

Using GPS to Sense a Drop-Off

Parent Child 62

t3 t4 t5 t6t2t1 t7


Day Care

Work


Parent Child 63

t3 t4 t5 t6t2t1 t7


Day Care

Work


Parent Child 64

t3 t4 t5 t6t2t1 t7


t3

Day Care

t4 t5 t6t2

Work

t1 t7


Parent Child 65


Day Care

Work

Using GPS to Sense a Pick-Up

Parent Child 66

t3 t4 t5 t6t2t1 t7


Day Care

Work


Parent Child

t3 t4 t5 t6t2t1 t7


GPS Ride Detection EvaluationFamily Precision Recall

How many of the sensed ridesare right

How many of the total ridesare sensed?

Pick-up Drop-off Pick-up Drop-off

A .991 .987 .912 .910

B .966 .962 .979 .981

C .913 .824 .971 .921

D .878 .873 .980 .944

E .931 .684 .959 .985

Mean .936 .866 .960 .950

.901 .955

69


Smarter power management

Wi-Fi, Bluetooth, cell tower ID

Apply heuristics to modelingCultural norms, Individual Behavior

Single Location Sensor

GPS Sampling Rate

70


Driver Prediction


Name Meaning Values

Ln Location of pick-up or drop-off Place ID

RType Ride type Pick-up, Drop-off

DoW Day of week 0,1,2,3,4,5,6

ToD Discretized time of day (15 min) 1,2,3…96

drivert-j Driver for the last 5 rides to Ln Mom, Dad

Driver distribution model [0,1]

Driver Prediction Model Features

72


Driver Prediction Performance x Number of Training Weeks of Data

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

73



1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

74



1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

1 24

75


Lateness Prediction


TnowTideal

Lateness in Minutes

Number of Examples

Distribution of Parent Late Arrivals

N = 83


Name Meaning Values

R Whether the parent remembers

True, False

Lchild Location of the child Place ID

Lcurr Current location of a parent

Place ID

J Driver prediction model Mom, Dad

78


Name Meaning Values


True, False



Place ID


T If the parent is traveling True, False

Lstart Starting location of a parent

Place ID

79


Name Meaning Values


True, False



Place ID




Place ID

Empirical cumulative distribution (ecdf) of

on-time arrivals to Lchild

at time TnowTideal

[0,1]

80


Name Meaning Values


True, False



Place ID




Place ID



at time TnowTideal

[0,1]

D Destination of a parent Place ID

81


Name Meaning Values


True, False



Place ID




Place ID



at time TnowTideal

[0,1]

D Destination of a parent Place ID

82


tideal - 30

A’ = 0.659

.6

.5


tideal

A’ = 0.801


tideal +10

A’ = 0.826



86


Validation3

87


Do parents understand the view?

Do parents perceive that the information is valuable?

Do parents feel more in control having this view?

89


1257

45 Davidoff et al Ubicomp 2006Davidoff et al Ubicomp 2007Davidoff, Dey + Zimmerman CHI 2009Davidoff et al. CHI 2010Months

Months24

24Families

Families

Families

Davidoff et al. Ubicomp 2011(in preparation)

90


Davidoff et al. Ubicomp 2011 in preparation

Experience prototype: Doctor

91


Davidoff et al. Ubicomp 2011 in preparation

Experience prototype: Kitchen

94


It's nice to have a map of the day. I don't literally write all this stuff down, so sometimes it's just hard to keep in my head.

Visual Distills Details

“

95

−P2“”


Because the calendar [is] all words and numbers, so you have to really think about …how long everything takes…

Visual Simplifies Calculation

“

96

−P10


…With the visual you can see who's doing what at what time. You can make a decision about who's activity to change.

Visual Simplifies Calculation

97

−P10“”


I would be able to know where everyone was gonna be, instead of having to ask around.

Visual Clarifies Intentions

“”98

−P11

“


Venkatesh + Davis 2003Technology Acceptance Model-3

99


…helps me do my job “as a parent.”

…would want my family to use it.

…more in control of details, more ready for changesPerception of Control

Behavioral Intention

Perceived Usefulness

100




Perception of Control

Supports Planning

Supports Awareness

7

6

5

4

3

2

1



102


Contributions

103


Suchman 1983, Tolmie et al. 2002

Certain routines can be modeled

104


2 31 Ride Detection

Driver Prediction

Predict Lateness

105


Commodity GPS with real power constraints

106


Breakdowns lead to loss of control

108




Perception of Control

Supports Planning

Supports Awareness

7

6

5

4

3

2

1


Gneezy + Rustichini 1998, Darrah 2009

A constant source of anxiety

110


Empower Parents


111


Future Work

112


Improve algorithms

Use algorithms to enable other family coordination systems

Add additional sensor inputs

Extend to other forms of routine

113

Scott DavidoffDissertation Defense Neustaedter + Brush CHI 2006 114


6pmOrthodontist6pmOrthodontist

Neustaedter + Brush CHI 2006 115


6pmOrtho6pmOrthoPaper RoutePaper Route

Neustaedter + Brush CHI 2006 116

Scott DavidoffDissertation Defense Brown et al. Ubicomp 2007

Baseball

117


Baseball

Brown et al. Ubicomp 2007 118


The project on


Thanks for your time, brain power, and support


Thanks to our sponsors

NSF IIS 1017429

Google Research Award


Thanks to our supporters


Thanks to my collaborators


Thanks to my committee


Thanks to my advisors


Questions


Image Credits

127


QUESTIONSLIDES

128


ROUTINE MODELINGRELATED WORK

129


FORMULAS

130


Driver Distribution

131€

φ =

driver = P1Ln ,Rtype,DoW

∑

driver = P1Ln ,Rtype,DoW

∑ + driver = P2Ln ,Rtype,DoW

∑


LWDT

132€

d(y)LWq =

y ie− j

i xij −q j( )

2

∑

i∈labeled data,i≠q

∑

e− j

i xij −q j( )

2

∑i=q∑


Bayesian Network

133

€

P(Rd | Lstartd ,Lstart

m ,Lcurrd ,Lcurr

m ,Lchild ,Td ,Tm ,φ)


RIDE SENSING TECHNICAL VERSION

134


Davidoff, Dey & Zimmerman CHI 2011

135


Two Kinds of Drop-Offs

136


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


t3t2 ,t1 ,

137


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


t3t2 ,t1 ,

138


Two Kinds of Pick-Ups

139


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


t3t2 ,t1 ,

140


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


t3t2 ,t1 ,

141


5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15


142


5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15

€

States = Ln,T |CoT,else{ }Using GPS to Sense a Drop-Off

143


5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15

€

States = Ln,T |CoT,else{ }

€

People = P,C{ }


144


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15

€


€

People = P,C{ }


145


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15

€


€

People = P,C{ }


€

(t1,P,CoT)∧(t1,C,CoT)∧

146


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15

€


€

People = P,C{ }


€

(t2,P,Ln )∧(t2,C,Ln )∧

€


147


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15

€


€

People = P,C{ }


€

(t2,P,Ln )∧(t2,C,Ln )∧

€

(t3,P,¬CoT)∧(t3,C,Ln )

€


148


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15

€


€

People = P,C{ }


€

(t2,P,Ln )∧(t2,C,Ln )∧

€

(t3,P,¬CoT)∧(t3,C,Ln )

€


149


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15

€


€

People = P,C{ }

€

(t1,P,¬CoT)∧(t1,C,Ln )∧


150


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15

€


€

People = P,C{ }

€

(t1,P,¬CoT)∧(t1,C,Ln )∧


151


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15

€


€

People = P,C{ }

€

(t1,P,¬CoT)∧(t1,C,Ln )∧

€

(t2,P,Ln )∧(t2,C,Ln )∧


152


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15

€


€

People = P,C{ }

€

(t1,P,¬CoT)∧(t1,C,Ln )∧

€

(t2,P,Ln )∧(t2,C,Ln )∧

€

(t3,P,CoT)∧(t3,C,CoT)


153


P C

5:15

Day Care

5:30 5:45 6:005:00

Work

4:45 6:15

€


€

People = P,C{ }

€

(t1,P,¬CoT)∧(t1,C,Ln )∧

€

(t2,P,Ln )∧(t2,C,Ln )∧

€

(t3,P,CoT)∧(t3,C,CoT)


154


SCHOOL HALF DAY BREAKDOWN

155


Baseball

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

View of Plan: S16

Track Practice

156


Baseball

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

View of Plan: S16

Track Practice

157


Baseball

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

Dad Started Baseball League

Track Practice

158


OTHER APPLICATIONS

159


Orthodontist

Scouts

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

Detect conflict with unlabeled routine

Track Practice

Check-up

Paper Route

160


Baseball

School

Dad Work

Home

4:00 5:00 6:00 7:003:002:00 8:00

S16Dad

Dad Started Baseball League

Track Practice

161


ACTIVITY SCATTERPLOT

162


RIDE SENSING ERRORS

163


Smarter power managementGPS Sampling Rate

164



Wi-Fi, cell tower, bluetoothSingle Location Sensor

GPS Sampling Rate

165


5:15

Home

5:30 5:45 6:005:00

Work

4:45 6:15

Simultaneous Departure Errors

School

Parent Child 166



Wi-Fi, cell tower, bluetooth

Apply heuristics to modelingCultural norms, Individual Behavior

Single Location Sensor

GPS Sampling Rate

167


1:00pm

Home

2:00 3:00 4:0012:00

Work

11:00 5:00

Lost Signal During Daytime

School

Parent Child 168


DRIVER PREDICTION ERRORS

169



170


USER ENACTMENT PICS

171


User Enactments


172


User Enactments


173

scott davidoffdissertation defense familycontrolsmarthome the project on routine as resource for the...

Documents

machine learning slide

zimmerman chi

families davidoff et

calendar months davidoff

learned routines

peoples routines fieldwork

peoples routines people

anind dey