context-aware systems in the wild

Context-Aware Systems Context-Aware Systems in the Wildin the Wild

Dr. Anind K. DeyDr. Anind K. DeyAssistant ProfessorAssistant ProfessorHuman-Computer InteractionHuman-Computer InteractionCarnegie Melon UniversityCarnegie Melon University

22

OutlineOutline

BackgroundBackground Ubicomp and context-awarenessUbicomp and context-awareness

Context-aware infrastructure Context-aware infrastructure Context-aware applicationsContext-aware applications New area: Usability in context-awarenessNew area: Usability in context-awareness

Change in approachChange in approach Change in infrastructure support and Change in infrastructure support and

interfaceinterface

44

Overview of UbicompOverview of Ubicomp

Computing embedded into everyday Computing embedded into everyday objects and environments, enhancing objects and environments, enhancing everyday activityeveryday activity

55

Context-Aware ComputingContext-Aware Computing

Context:Context: situational elements relevant to situational elements relevant to interaction between user, application, interaction between user, application, environmentenvironment

Context-awareness:Context-awareness: situationally appropriate; situationally appropriate; apps adapting to context, increasing value to apps adapting to context, increasing value to usersusers Using sensors and actuators to improve human-Using sensors and actuators to improve human-

computer interaction and (computer-mediated) computer interaction and (computer-mediated) human-human and human-environment interactionhuman-human and human-environment interaction

Examples: Examples: tour guide, reminders, diary retrievaltour guide, reminders, diary retrieval

66

Historical perspectiveHistorical perspective

Weiser’s 1991 visionary Scientific American Weiser’s 1991 visionary Scientific American article: “Computer for the 21st century”article: “Computer for the 21st century” Sal’s alarm clock and coffeemakerSal’s alarm clock and coffeemaker Windows: Electronic trails of neighbors, kids, Windows: Electronic trails of neighbors, kids,

weatherweather Rear view mirror: coffee shopRear view mirror: coffee shop Search for details in meeting informationSearch for details in meeting information

Amazing amounts of contextAmazing amounts of context

But how do you build?But how do you build?

77

10 years of context-aware 10 years of context-aware computing researchcomputing research

Driven by a single problemDriven by a single problem

Reminder to buy milkReminder to buy milk

When to deliver: not When to deliver: not time/location specifictime/location specific

How to deliver: appropriate How to deliver: appropriate modalitymodality

88

Context ToolkitContext Toolkit

1997: difficult to build simple 1997: difficult to build simple context-aware applicationscontext-aware applications No abstractions for acquiring and using No abstractions for acquiring and using

context from sensors or controlling context from sensors or controlling actuatorsactuators

Context coming from a number of Context coming from a number of distributed sourcesdistributed sources

No principles for designing applicationsNo principles for designing applications

99

Design processDesign process

1.1. SpecificationSpecification

2.2. AcquisitionAcquisition

3.3. DeliveryDelivery

4.4. ReceptionReception

5.5. ActionAction

1. Specification2. Acquisition

3. Action

1010

Thesis: Context ToolkitThesis: Context Toolkit

Context Toolkit to support programmers in Context Toolkit to support programmers in building context-aware applications more easilybuilding context-aware applications more easily

Widget

Sensor

Widget

Application Application

Interpreter InterpreterAggregator

Sensor

ContextArchitecture

DiscovererService

Actuator

1111

Some SuccessSome Success

First of a series of toolkits that actually First of a series of toolkits that actually makes it easier to build context-aware makes it easier to build context-aware applications applications

Toolkit in use by over 600 academic and Toolkit in use by over 600 academic and corporate institutionscorporate institutions http://contexttoolikit.sourceforge.nethttp://contexttoolikit.sourceforge.net

Lots of applications createdLots of applications created CHI 99, ISWC 99, UBICOMP 00, UBICOMP 01, UIST 02, CHI 03, HCI CHI 99, ISWC 99, UBICOMP 00, UBICOMP 01, UIST 02, CHI 03, HCI

Journal 01, ToCHI 04Journal 01, ToCHI 04

1212

OutlineOutline

Motivation and backgroundMotivation and background Ubicomp and context-awarenessUbicomp and context-awareness

Context-aware infrastructureContext-aware infrastructure Context-aware applicationsContext-aware applications Usability in context-awarenessUsability in context-awareness

1313

Context-Aware Context-Aware InfrastructuresInfrastructures

CTK: privacy, uncertainty, end-user supportCTK: privacy, uncertainty, end-user support Nexus: context modeling, representation, and Nexus: context modeling, representation, and

spatiotemporal processingspatiotemporal processing JCAF: quality of context and context modelingJCAF: quality of context and context modeling Context Fabric: privacy, and modeling of Context Fabric: privacy, and modeling of

context entitiescontext entities PersonisAD: context models of entities and PersonisAD: context models of entities and

links between them, support for scrutabilitylinks between them, support for scrutability Context Spaces: context modeling, Context Spaces: context modeling,

inferencing of situations and probabilistic inferencing of situations and probabilistic reasoningreasoning

1414

OutlineOutline



1515

Context-Aware ApplicationsContext-Aware Applications

What is a context-aware application?What is a context-aware application? App that uses context to perform some App that uses context to perform some

behavior/service for its user(s)behavior/service for its user(s) 3 types of behaviors:3 types of behaviors:

Display contextDisplay context Automatically execute/adapt servicesAutomatically execute/adapt services Tag captured information for easier Tag captured information for easier

retrievalretrieval

1616

Directly displaying contextDirectly displaying context

1717

Execute/Adapt ServicesExecute/Adapt Services

Select and perform a relevant service Select and perform a relevant service from multiple servicesfrom multiple services

Change how a service behaves or is Change how a service behaves or is executedexecuted

1818

Tag InformationTag Information

Use context to aid in context-based Use context to aid in context-based retrievalretrieval

May be easier to remember context May be easier to remember context than specific informationthan specific information

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.QuickTime™ and a

TIFF (LZW) decompressorare needed to see this picture.

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

1919

So are We Done?So are We Done?

Can support building and executing Can support building and executing of context-aware appsof context-aware apps

Are building interesting appsAre building interesting apps

But, we still don’t have widespread But, we still don’t have widespread deploymentdeployment

Commercially available apps are Commercially available apps are really quite simplereally quite simple

2020

Back to the Milk ReminderBack to the Milk Reminder

Just making it easier to build Just making it easier to build applications is only part of the applications is only part of the problemproblem Once you know what context to capture Once you know what context to capture

and how to use it, easier to executeand how to use it, easier to execute But how do you figure this out?But how do you figure this out?

Lots of problems to be addressed on Lots of problems to be addressed on the user sidethe user side

2121

OutlineOutline



2222

What are the Impediments?What are the Impediments?

Cost and difficulty in deploying Cost and difficulty in deploying infrastructureinfrastructure

Applications are of limited valueApplications are of limited value Not modeling the right tasksNot modeling the right tasks

Applications lacking in usabilityApplications lacking in usability Apps lacking in support for end-user Apps lacking in support for end-user

control control Apps lacking in intelligibilityApps lacking in intelligibility

2323

Control in UbicompControl in Ubicomp

Performed two studies on how people Performed two studies on how people respond to context-aware applicationsrespond to context-aware applications

Each study: ~20 people, diary, interviewsEach study: ~20 people, diary, interviews Presented variety of applications, different Presented variety of applications, different

levels of interactivity and elicited responseslevels of interactivity and elicited responses Serious issues with lack of control and privacySerious issues with lack of control and privacy Despite these, wanted to use these Despite these, wanted to use these

applicationsapplications

UBICOMP 03, INTERACT 03UBICOMP 03, INTERACT 03

2424

Usability is KeyUsability is Key

Dourish, Abowd and Mynatt, Dourish, Abowd and Mynatt, Bellotti and others: lack of Bellotti and others: lack of control in these control in these environmentsenvironments

Information collected, Information collected, synthesized and used synthesized and used implicitlyimplicitly

How do I know what’s going How do I know what’s going on? (intellgibility)on? (intellgibility)

How do I change what’s How do I change what’s going on? (control)going on? (control)

Who gets this information? Who gets this information? (privacy)(privacy)

Is this another way to SPAM Is this another way to SPAM me? (overload)me? (overload)

Rich Gold “Dancing Toaster”

2525

Research Focus: UsabilityResearch Focus: Usability

2626

Research Focus: Feeling in Research Focus: Feeling in ControlControl

End-User Intelligibility and End-User Intelligibility and Control: decide what actions an Control: decide what actions an environment should take and environment should take and understand environment stateunderstand environment state CHI 04, Pervasive 06, Ubicomp 06, CHI 07, CHI 04, Pervasive 06, Ubicomp 06, CHI 07,

Ubicomp 07Ubicomp 07

2727


Privacy: decide what information to release Privacy: decide what information to release whenwhen CHI 03, Personal and Ubiquitous Computing Journal 04CHI 03, Personal and Ubiquitous Computing Journal 04

2828


Peripheral Displays: Peripheral Displays: present present information without overwhelminginformation without overwhelming CHI 03, CHI 04, Pervasive 04, UIST 04, UIST CHI 03, CHI 04, Pervasive 04, UIST 04, UIST

05, CHI 0605, CHI 06

2929

User-Centered Approach:User-Centered Approach:

Improved modeling is required: real situations, Improved modeling is required: real situations, real users, real problems --> real benefitsreal users, real problems --> real benefits

EthnographyEthnography Need findingNeed finding Sense user behavior of interestSense user behavior of interest Create models of user behaviorCreate models of user behavior Present information or execute services based on modelsPresent information or execute services based on models Study impact:Study impact:

Domain impactDomain impact Impact on infrastructure designImpact on infrastructure design Mental modelsMental models Interactive techniques, tools for control, feedback, Interactive techniques, tools for control, feedback,

modeling, evaluation…modeling, evaluation…

3030

ChallengesChallenges

Leveraging real human context and Leveraging real human context and realizing that you’re not really realizing that you’re not really getting at human intentiongetting at human intention Star Trek doorsStar Trek doors

Anyone remember the original Star Trek Anyone remember the original Star Trek series?series?

Japanese doorsJapanese doors Wash doors?Wash doors?

3131

ChallengesChallenges

Leveraging real human context and Leveraging real human context and realizing that you’re not really realizing that you’re not really getting at human intentiongetting at human intention ClippyClippy

3232

End User Control and End User Control and IntelligibilityIntelligibility

Control: real situations are dynamic Control: real situations are dynamic and cannot always be planned for and cannot always be planned for a a prioripriori Need to allow constant adaptationNeed to allow constant adaptation

Home: new people, new organization, new Home: new people, new organization, new rolesroles

Programmer can’t possibly hope to cover Programmer can’t possibly hope to cover these situationsthese situations

Must support end-users in maintaining controlMust support end-users in maintaining control Need support for thisNeed support for this

3333

Control of Smart HomesControl of Smart Homes

3434

End User Control and End User Control and IntelligibilityIntelligibility

Intelligibility: users will reject Intelligibility: users will reject systems they do not understand, systems they do not understand, particularly when these systems particularly when these systems make mistakesmake mistakes Amazon recommender, ClippyAmazon recommender, Clippy Complex systems need to describe their Complex systems need to describe their

behavior to some degreebehavior to some degree

3535

Study of Mental ModelsStudy of Mental Models

One group received up to 3 features that best explain current One group received up to 3 features that best explain current predictionprediction

CHI 07CHI 07

3838

Research ProblemResearch Problem

Disconnect: what should smart homes do and howDisconnect: what should smart homes do and how

3939

Research ProblemResearch Problem

Support what families value: time, activities, relationshipsSupport what families value: time, activities, relationships

4040

Focus on Dual-income Focus on Dual-income FamiliesFamilies Largest segment of US population and growingLargest segment of US population and growing

Live logistically complex lives that drive Live logistically complex lives that drive aggressive and experimental use of aggressive and experimental use of communication technologycommunication technology

4141

Why Families Feel Out of Why Families Feel Out of ControlControl

Swamped with responsibilities from Swamped with responsibilities from kids activities and jobskids activities and jobs

4242

Findings: Master “Busyness”Findings: Master “Busyness”

Parents want to master managing numerous activitiesParents want to master managing numerous activities

4343

Challenge: “Wicked Challenge: “Wicked Problem” of Activity Problem” of Activity

ManagementManagement

4444

Soccer ExampleSoccer ExampleGAME OR PRACTICE? HOME OR AWAY? WHAT TIME?

4545

WHO PICKS UP? WHO DROPS OFF? WHERE?

Soccer ExampleSoccer Example

4646

SHIN GUARDS, KNEE PADS. CLEATS OR FLATS?


4747

PRACTICE UNIFORM? HOME OR AWAY UNIFORM?


4848

CLEAN CLOTHES THE NIGHT BEFORE


4949

JUICE BEFORE OR AFTER? ORANGES AT HALFTIME?


5050

LOST ON THE CALENDAR

BreakdownsBreakdowns

5151

LAST MINUTE CARPOOL DECISIONS


5252

MOM’S OUT OF TOWN SO DAD’S IN CHARGE


5353



5454



5555

CREEPING RESPONSIBILITY


5656

CREEPING RESPONSIBILITY


5757

UNPREDICTABLE ORANGES


5858

SICK CHILD


5959

CASCADE EFFECTS


6060

Mapprentice: Route Planning

GPS-enabled cars In-car Support

Current routing systems are impoverished

Don’t take into account: time/day, weather, traffic

Don’t take knowledge of the user into account

Can build much smarter systems

Model driving behavior, use contextual cues, and produce better driving directions

Predict destination and route, suggest alternate routes to take care of errands, or to improve path

Diagnose capabilities of elder drivers and produce more appropriate routes

Web Support

6161

IMPACT: Improving and Motivating Physical Activity using ContexT

Great job!!

Great job!!

GPS-enabled phone Activity monitor Visualization

60% of Americans overweight/obese

Leading cause is lack of physical activity

Ethnography reveals subjects want expert advice, have difficulty with motivation and understanding data from exercise devices

Our solution: real-time feedback and point-of-decision prompts via mobile phone, expert agents to guide reflection and provide motivation, context

Motivate lifestyle change to be more physically active

6262

Context-Based Memory Aids for Peoplewith Mild Cognitive ImpairmentsDey

12 million Americans have Alzheimer’s/MCI:

memory loss, financial impact and caregiver burden

Study of patient/caregiver dyads to understand types of memory impairment, impacts of impairment, and role of technology

Our solution: rich capture of everyday experiences, integrate with contextual cues for enhanced playback and review, and integrate use into daily life

More accurate/complete account of life, reduced burden, increased care, and greater independence

rich multimedia experience

unobtrusive sensors engaging memory exercise

6363

SummarySummary

Despite all the work in making context-aware Despite all the work in making context-aware applications easier to build, there’s lots to be doneapplications easier to build, there’s lots to be done

Context is only an abstraction for user Context is only an abstraction for user motivation/intentmotivation/intent

Context is a part of everyday life and activitiesContext is a part of everyday life and activities A number of difficult usability and technical issues A number of difficult usability and technical issues

that need to be addressedthat need to be addressed Model user needs and practicesModel user needs and practices Support end-user controlSupport end-user control Support intelligibility of end-user systemsSupport intelligibility of end-user systems

Still no solution to the milk reminder problemStill no solution to the milk reminder problem

6464

Questions?Questions?

[email protected]@cs.cmu.edu

www.cs.cmu.edu/~anindwww.cs.cmu.edu/~anind

context-aware systems in the wild

Documents

contextaware systems

quality of context

years of context

context models of entities

context modelingcontext

aware infrastructuresctk

aware applicationswhat

aware computing researchdriven