ICST 2015

What Companions Know and Remember

Maria Wolters@mariawoltersSchool of InformaticsSchool of Philosophy, Psychology, and Language ScienceUniversity of Edinburgh

ForgetIT Team: Robert Logie, Elaine Niven (Edinburgh), Heiko Maus, Sven Schwarz (DFKI)

CADENCE: Johanna Moore, Myroslava Dzikovska, Jonathan Kilgour, Sarah MacPherson (Edinburgh), Marek Grzesz, Jesse Hoey (Waterloo)

Companions need to focus on what matters, not on what they can store.

If companions are to learn what matters, they need to be good students.

Example Case Study

Leafy Green General Practice

gives all their patients a health care companion app

that provides electronic record access, and facilitates communication with the practice.

Old Style Knowing and Remembering

❖ Dawn Jones is Patient ID SCO4711. She has Type II diabetes.

❖ Dawn last saw Dr Smith three months ago, on July 23 at 2.15pm; as a result, her medications were changed

Is That Enough?

❖ The caller is Dawn Jones, Patient ID SCO4711. She has Type II diabetes.

❖ Dawn is quite chatty. It’s hard to get a straight Yes or No out of her.

❖ Dawn last saw Dr Smith three months ago, on July 23 at 2.15pm; as a result, her medications were changed

❖ Dawn normally goes every quarter for a check up. Almost all of her appointments are with Dr Miller, whom she trusts and likes.

The Additional Edge

❖ Know:

❖ how people talk, not just what they are likely to talk about

❖ to what extent people can and will adapt to computer voice interfaces

❖ Remember: remember the gist, forget the details

Companions and Knowing

❖ Interaction style (MATCH project)

❖ How to Help (CADENCE project)

1. Wolters MK, Georgila K, MacPherson S, Moore J. Being Old Doesn’t Mean Acting Old: Older Users' Interaction with Spoken Dialogue Systems. ACM Trans Access Comput. 2009;2(1):1–39.2. Maria K. Wolters, Jonathan Kilgour, Sarah E. MacPherson, Myroslava Dzikovska, and Johanna D. Moore. 2015. The CADENCE Corpus: A New Resource for Inclusive Voice Interface Design.

In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 3963-3966. http://doi.acm.org/10.1145/2702123.2702372

MATCH: Appointment Scheduling

❖ Older (50+) and younger (18-30) participants schedule one appointment each using 9 different voice interfaces, fill in long usability questionnaire, and then recall agreed health care professional, date, time, and location

❖ Voice interfaces vary by number of options presented (1, 2, 4) and confirmation (no, implicit, explicit)

❖ Wizard-of-Oz with human speech recognition and natural language understanding

Interaction Analysis

❖ All interactions were transcribed orthographically

❖ Semi-automatic dialogue act annotation

❖ dialogue act = speech act (request, confirm, …) + modifier (half day, time, person)

❖ Clustering of behaviour based on statistics (frequency of dialogue acts, turn length, type/token ratio …)

Differences in Interaction Style❖ Cluster 1 - Factual:

very few social dialogue acts, short utterances, to the point.

❖ Cluster 2 - Social: more polite, fairly chatty, give reasons why they can’t make certain dates, …

❖ Most younger people, and 1/3 of the older people were factual, the remaining older people were more social.

❖ Users with a social interaction style did not change it over the course of 9 dialogues

Open Questions from MATCH

❖ Why do Social people fail to adapt?

❖ no adverse effects (perfect speech recognition)

❖ social cognition (can’t pick up cues)

❖ don’t want to

❖ What happens if we force Social people to become Factual?

CADENCE: Intelligent Cognitive Assistant

❖ Idea: The ICA watches people perform tasks and provides help if needed

❖ requires a model of users’ abilities, a task model, plans for achieving goals

❖ Origin: COACH Handwashing assistant (Mihailidis / Hoey and collaborators)

CADENCE Task

❖ Participants with (n=10) and without (n=44) cognitive impairment use a very simple tablet app to look up details of appointments

❖ A simulated ICA (Wizard of Oz) is available to help

❖ system versus user initiative

❖ neutral versus encouraging

Wizard of Oz Dialogue Corpus

System: Have you found what you are looking for?

DF09: Yes, thank you.

System: You have finished the task.

DF09: (hey)

System: Please write down the answer.

DF09: Well I am thank you, just be quiet.

Key Findings

❖ Deciding when to intervene, how long to leave it, is tricky

❖ A human assistant would generally be preferred, but …

❖ … one doesn’t have to be polite to a computer

Open Questions from CADENCE

❖ How does a dialogue system cope with people thinking aloud?

❖ When should the system intervene?

❖ How can we make sure we „teach people to fish“?

Companions and Remembering

❖ We can’t preserve everything - humans cope because they abstract at each level of neural and cognitive information processing

❖ But computers / companions remember everything?

http://www.redorbit.com/reference/floppy_disk/

Preservation is hard.

http://www.reed.edu/reed_magazine/june2013/articles/eliot_circular/gilgamesh.html

http://www.redorbit.com/reference/floppy_disk/

The ForgetIT Approach

❖ Preservation Value: how much effort should we put into archiving?

❖ Memory Buoyancy: what do we need right now? what happened recently?

❖ EU FP7 IP, led by L3S (Hanover)

Case Study: Photo Preservation

❖ DFKI Semantic Desktop technology (via PIMO5) system allows rich contextualization and rich set of evidences

Open Questions❖ How do we find out what people value

and how much they want to invest, when they still insist on managing their photos using the file manager?(Source: ForgetIT Photo Preservation Survey of 1300+ people world wide)

❖ How can we support Curators? How those who prefer to „file and forget“?

❖ What is the difference between useful and creepy?

http://forums.androidcentral.com/ambassador-guides-tips-how-s/300182-guide-google-now.html

1. Wolters MK, Niven E, Runardotter M, Gallo F, Maus H, Logie RH. Personal Photo Preservation for the Smartphone Generation. Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA ’15 [Internet]. New York, New York, USA: ACM Press; 2015 [cited 2015 May 6]. p. 1549–54. Available from: http://dl.acm.org/citation.cfm?id=2702613.2732793

Conclusion

Companions need to focus on what matters, not on what they can store.

If companions are to learn what matters, they need to be good students.

maria.wolters@ed.ac.uk @mariawolters