the assisted cognition project henry kautz, dieter fox, gaetano boriello lin liao, brian ferris,...
TRANSCRIPT
![Page 1: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/1.jpg)
The Assisted Cognition Project
Henry Kautz, Dieter Fox, Gaetano BorielloLin Liao, Brian Ferris, Evan Welborne
(UW CSE)
Don Patterson(UW / UC Irvine)
Kurt Johnson, Pat Brown, Mark Harniss(UW Rehabilitation Medicine)
Matthai Philipose(Intel Research Seattle)
![Page 2: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/2.jpg)
Trend 1: Sensing Infrastructure
Robust direct-sensing technologyo GPS-enabled phoneso RFID tagged productso Wearable multi-modal sensors
Rapid commercial deployment
![Page 3: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/3.jpg)
Trend 2: Healthcare Crisis
Demand for community integration of the cognitively disabledo 100,000 @ year disabled by traumatic brain injuryo 7.5 million in US with mental retardationo 4 million in US with Alzheimer’s
Family burnout Nationwide shortage of professionals
![Page 4: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/4.jpg)
Assisted Cognition Technology to support independent living
by people with cognitive disabilitieso at homeo at worko throughout the community
byo Understanding human behavior from sensor
datao Actively prompting and advisingo Alerting human caregivers when necessary
![Page 5: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/5.jpg)
Building Partnerships
UW Assisted Cognition seminaro CSE, medicine, nursing, Intel
ACCESSo UW CSE & Rehabilitation Medicineo Grant from NIDDR (Dept. of Education)o Help cognitively disabled use public
transportationo Prototype: Opportunity Knocks
Intel Proactive Health efforto Computing for wellness & caregivingo Promote partnerships with government,
universities, healthcare organizationso Intel Seattle: sensors for activity tracking
![Page 6: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/6.jpg)
Example
Way-finding Assistanto Help user travel throughout community
On foot Using public transportation
o Detect user errors Proactively help user recover “You missed your stop, so get off at the next
stop and then wait for the #16 bus...”
o Potential users TBI, MR, mild memory impairment
![Page 7: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/7.jpg)
Example ADL Assistant
o Activities of daily living Eating, bathing, dressing, ... Cooking, cleaning, emailing, ...
o Monitoring Changes in ADLs signal changes in health
o Reminding / prompting “Time to take your blue meds”
o Step-by-step guidance “Turn on the tap ... now pick up the brush
...”
o Potential users Disabled, ordinary aging
![Page 8: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/8.jpg)
General Model
usermodel
common-sense
KB
geospatialDB
wearablessensors
environmentalsensors
interventiondecisionmaking
userinterface
caregiveralerts
![Page 9: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/9.jpg)
General Model
common-sense
KB
geospatialDB
wearablessensors
environmentalsensors
interventiondecisionmaking
userinterface
caregiveralerts
physical motion& position
cognitive state
goals
activity
![Page 10: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/10.jpg)
Deciding to Intervene
A = system intervenes
G = user actually needs help
![Page 11: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/11.jpg)
ACCESSWay-finding Assistant
supported by
National Institute on Disability & Rehabilitation Research
DARPA IPTO
![Page 12: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/12.jpg)
The Need: Community Access for the Cognitively
Disabled
![Page 13: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/13.jpg)
Problems in Using Public Transportation
•Learning bus routes and numbers
![Page 14: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/14.jpg)
Problems in Using Public Transportation
•Learning bus routes and numbers
•Transfers, complex plans
![Page 15: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/15.jpg)
Problems in Using Public Transportation
•Learning bus routes and numbers
•Transfers, complex plans
•Recovering from mistakes
![Page 16: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/16.jpg)
Result
•Need for extensive life-coaching
•Need for point-to-bus service
![Page 17: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/17.jpg)
Result
•Need for extensive life-coaching
•Need point-to-bus service
•Isolation
![Page 18: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/18.jpg)
Current GPS Navigation Devices
Designed for drivers, not bus riders!o Should I get on this bus?o Is my stop next?o What do I do if I miss my stop?
Requires extensive user inputo Keying in street addresses no fun!
Device decides which route is “best”o Familiar route better than shorter one
“Catastrophic failure” when signal is lost
![Page 19: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/19.jpg)
New Approach
User carries GPS cell phone System infers transportation mode
o Position, velocity, geographic information
Over time, system learns about usero Important places o Common transportation plans
Breaks from routine = possible user errorso Ask user if help is needed
![Page 20: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/20.jpg)
GPS readingzk-1 zk
Edge, velocity, positionxk-1 xk
k-1 k Data (edge) association
Time k-1 Time k
mk-1 mk Transportation mode
tk-1 tk Trip segment
gk-1 gk Goal
ck-1 ck Cognitive mode { routine, novel, error }
User Model
![Page 21: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/21.jpg)
Error Detectio
n: Missed
Bus Stop
![Page 22: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/22.jpg)
GPS camera-phone “Knocks” when there is
an opportunity to help
o Can I guide you to a likely destination?
o I think you made a mistake!
o This place seems important – would you photograph it?
Prototype: Opportunity Knocks
![Page 23: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/23.jpg)
Status
User needs study Algorithms for learning and
predicting transportation behavioro Best paper award at AAAI-2004
Proof of concept prototype Now: user interface studies
o Modality: Audio, Graphics, Tactile, ...o Guidance strategies: Landmarks,
User frame of reference, Maps, ...
![Page 24: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/24.jpg)
ADL Monitoring from RFID Tag
Data
UW CSE
Intel Research Seattle
demo at Intel this afternoon
![Page 25: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/25.jpg)
Object-Based Activity Recognition
Activities of daily living involve the manipulation of many physical objectso Kitchen: stove, pans, dishes, …o Bathroom: toothbrush, shampoo, towel,
…o Bedroom: linen, dresser, clock,
clothing, … We can recognize activities from a
time-sequence of object touches
![Page 26: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/26.jpg)
Sensing Object Manipulation
RFID: Radio-frequency identification tagso Smallo Long-lived – no
batterieso Durable
Easy to deploy Bracelet touch sensor Wall-mount
movement sensor
![Page 27: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/27.jpg)
Example Data Stream
![Page 28: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/28.jpg)
Example Activity Model
![Page 29: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/29.jpg)
Creating Models of ADLs
Hand-built Learn from sensor data Mine from natural-language texts All of the above...
![Page 30: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/30.jpg)
Experiment: Morning Activities
10 days of data from the morning routine in an experimenter’s homeo 61 tagged objects
11 activities o Often interleaved and interruptedo Many shared objects
Use bathroom
Make coffee Set table
Make oatmeal
Make tea Eat breakfast
Make eggs Use telephone Clear table
Prepare OJ Take out trash
![Page 31: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/31.jpg)
DBN with Aggregate Features
88% accuracy6.5 errors per episode
![Page 32: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/32.jpg)
Improving Robustness
Tracking fails if novel objects are used
Solution: smooth parameters over abstraction hierarchy of object types
![Page 33: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/33.jpg)
Status
Accurate tracking of wide variety ADLs Active collaboration with Intel Current work
o Detecting user errors in ADL performanceo Learning more complex ADLs
Preconditions/effects Multi-tasking Temporal constraints
o Reminding & prompting
![Page 34: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/34.jpg)
Concluding Remarks
Research on Assisted Cognition going great guns at UW and (a few) other universitieso CMU / Pitt / U Michigan (Nursebot,
Autominder – M. Pollack)o Georgia Tech (Aware Home, G.
Abowd)o MIT (House N, Stephen Intille)
![Page 35: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/35.jpg)
Some Thoughts on Funding
Getting funding for work in this area is currently challengingo We were fortunate once with NIDRR,
but less than 1% of their budget is for research
o NIH & NIA spend relatively little on caregiving research New NIH “Roadmap” for interdisciplinary
exploratory research completely leaves out caregiving!
o NIN has good people, but no real money
![Page 36: The Assisted Cognition Project Henry Kautz, Dieter Fox, Gaetano Boriello Lin Liao, Brian Ferris, Evan Welborne (UW CSE) Don Patterson (UW / UC Irvine)](https://reader034.vdocuments.net/reader034/viewer/2022051416/56649ea35503460f94ba7d3c/html5/thumbnails/36.jpg)
Some Thoughts on Funding
Getting funding for work in this area is currently challengingo NSF supports some of the underlying,
multi-use technology, but not medically-oriented applications Exception: helping disabled use computers
o Industry support is vital, but more for collaboration than actual dollars Good industry grant = 1 grad student
o There’s a gap waiting to be filled...