creating context-aware applications

1

Context-Aware ApplicationsWorkflows, Challenges, and Opportunities

Simon GuestDistinguished Engineer

Neudesic, [email protected]

Image Attribution: http://www.flickr.com/photos/48600098314@N01/2612632787

2

Workflow for Determining Context

3

1. Determine SensorsSensors Critical to Determine Context

• Hard Sensors – GPS, Accelerometer, Gyro, Temperature, Humidity, etc.

• Soft Sensors – Calendar, Facebook Friends, Profile, Likes/Dislikes, History, etc.

• Mobile Devices – Arrays of sensors, with new ones being introduced all the time (e.g. fingerprint democratized with Android)

• Sensor Co-processors – Sensors moving to co-processors means lower power consumption (which means that sensors can be on all of the time)

• Bluetooth LE – Greatly extends the the range and types of sensors (e.g. indoor proximity and awareness)

4

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

etc.

5

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

etc.

2. Sensor AggregationWe Don’t Need *All* of the Raw Sensor Data to Determine Context

• Augmented Sensors – Algorithm that augments or abstracts existing sensors (e.g. “Shaking” using the accelerometer)

• Extended to Motion APIs – Google introducing standing, walking, cycling, driving “sensors” in Android 4.3

• Vendor Opportunities – Multiple vendors building SDKs for sensor aggregation – inc. Semusi (Male/Female “sensor” from accelerometer and gyro)

6

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

etc.

7

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

etc.

etc.

etc.

8

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

etc.

etc.

etc.

3. Send to Non-Relational Data StoreGoal is to capture aggregated sensor data quickly and efficiently

• Schema-less– No need to care about schema or structure• Fire and Forget – No need to send confirmation back to client, or

handle missed messages or retries• Scale and Partitions – Data store should be a single entry point that

automatically handles scale and partitioning

9

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

etc.

etc.

etc.

10

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

etc.

etc.

etc.

11

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

etc.

etc.

etc.

4. Determine Perceived ContextAnalyze data store to determine one or more perceived contexts

• Simple Matches– Single sensor data can often reveal simple matches (e.g. movement or in/out geo-fenced location)

• Combined Matches – Multiple parallel or serialized sensors to reveal more complex matches (e.g. movement with external GPS, combined with loss of signal and walking activity might indicate parked in underground parking structure).

• Learned Matches – Multiple sensor data used to reveal matches through patterns (e.g. driving detected at same time every day might indicate commuting, which is turn can infer place of work)

• Timing – Context often derived from “time blocks” of sensor data – e.g. what is happening now vs. what has happened over the last 30 minutes

12

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

etc.

etc.

etc.

13

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

etc.

etc.

etc.

Asleep

etc.

Make determination

Query

14

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

etc.

etc.

etc.

Asleep

etc.

Make determination

Query

5. Triggers on Context ChangeTriggers used to monitor change in context

• Change in Context– Move from one perceived context state to another (e.g. from in office to commuting)

• External Data Source– Triggers often rely on external data source (which can be interpreted as another soft sensor). Could include environmental data, move listings, weather forecasts, etc.

• Machine Learning– Often benefits from some kind of ML to help determine change in context across multiple sensors, especially in recommendations space

15

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

etc.

etc.

etc.

Asleep

etc.

Make determination

Query

16

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

etc.

etc.

etc.

Asleep

etc.

Make determination

Query

Context Change(Triggers)

ExternalData

Flippingchannels

TimingLoop

Better movie

Startedcommute

House too cold

17

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

etc.

etc.

etc.

Asleep

etc.

Make determination

Query


ExternalData

Flippingchannels

TimingLoop

Better movie

Startedcommute

House too cold

6. Invoke FunctionsCertain Triggers Used to Invoke Functions

• Take Action– Functions take action based on triggers. Invoke an action that may or may not involve some user input.

18

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

etc.

etc.

etc.

Asleep

etc.

Make determination

Query


ExternalData

Flippingchannels

TimingLoop

Better movie

Startedcommute

House too cold

19

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

etc.

etc.

etc.

Asleep

etc.

Make determination

Query


ExternalData

Flippingchannels

TimingLoop

Turn on heating

Make recommendation

Better movie

Startedcommute

House too cold

Adaptive application

Proactive application

Functions

20

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

etc.

etc.

etc.

Asleep

etc.

Make determination

Query


ExternalData

Flippingchannels

TimingLoop

Turn on heating

Make recommendation

Better movie

Startedcommute

House too cold



Functions

7. User Interaction and Feedback LoopInteract with user to notify them of any action, and/or provide feedback

• User Notification– Notify the user of the result from a function• User Interaction/Confirmation – Get confirmation from the user that

a function should really happen, or provide a recommendation.• Feedback Loop – Provide option for user to submit feedback based

on the function (like, dislike, rating, etc.)

21

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

etc.

etc.

etc.

Asleep

etc.

Make determination

Query


ExternalData

Flippingchannels

TimingLoop

Turn on heating

Make recommendation

Better movie

Startedcommute

House too cold



Functions

22

GPS

Hard Sensors

Accel.

Temp

Noise

Step

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Geo

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

etc.

etc.

etc.

Asleep

etc.

Make determination

Query


ExternalData

Flippingchannels

TimingLoop

Turn on heating

Make recommendation

Better movie

Startedcommute

House too cold



User Interaction

Decision Making Feedback

Functions

23

Types of Context-Aware Applications

24

Types of Context-Aware ApplicationsAdaptive

• Act on behalf of the user• Try to adapt to the user’s

context• Often based using single-trigger

• Turn on heating when leave for home

• Phone to silent in meetings• Offer music library when get on

bus

• Need to be careful if building adaptive UI, especially if hiding/showing features

Proactive

• Involving the user• Often tied to a recommendation

engine• Often requires user-interaction• Often uses multiple triggers

• Recommend movie to watch when watching TV

• Display offer based on history when walking through grocery store

• UI needs to be unobtrusive, and easy to dismiss

25

Challenges When Developing Context-Aware Applications

26

ChallengesContext Mismatch

• Sometimes the perceived context is just wrong…• “You really think I want to do this now?!?!”• Multiple contexts help accuracy, and feedback mechanism is critical

Being Creepy

• Sharing everything that you know about the context/trigger can appear creepy

• “Your Mom’s birthday is tomorrow, and you’ve missed the last three years in a row. Do you want to pull into the 7-Eleven ahead? They have flowers on sale…”

• Reveal only enough to let the user deduce the same context, and feedback again is critical

Being Annoying

• Interruptions should follow normal human behaviors• “I wanted to let you know that I’ve turned the heating on” – “Not

now, I’m driving!”• Knowing the context of the recipient during the UI loop is also

important

27

Opportunities

28

Examples of Opportunities

Retail

• Opportunistic product offers while shopping

• Shopping recommendations based on current context

Real Estate

• Contextual awareness of prospective buyers searching for properties

Gaming

• Using context to enhance the gaming experience of patrons

Field Employees

• Context-aware applications that helps field-based employees become more productive

Home Automation

• Context-aware applications that interact with other systems in the home

Travel Applications

• More intelligent travel applications through context

29

References

30

References• Shilt, Adams, Want (1994): Proceedings of the Workshop on Mobile

Computing Systems and Applications • http://www.interaction-design.org/references/conferences/

proceedings_of_the_workshop_on_mobile_computing_systems_and_applications.html

• Swati A. Sonawane: Context-Aware Computing• http://www.slideshare.net/swatibaiger/context-aware-computing-

14084995• Albrecht Schmidt (2013): Context-Awareness, Context-Aware User

Interfaces, and Implicit Interaction • http://www.interaction-design.org/encyclopedia/context-

aware_computing.html

http://www.interaction-design.org/references/conferences/proceedings_of_the_workshop_on_mobile_computing_systems_and_applications.html




http://www.slideshare.net/swatibaiger/context-aware-computing-14084995



http://www.interaction-design.org/encyclopedia/context-aware_computing.html



31

Thank You

Simon GuestDistinguished Engineer

Neudesic, LLC

[email protected]@simonguest

creating context-aware applications

Technology