Denis GouinValérie LavigneAlexandre Bergeron-Guyard

Innovative Interfaces and Interactions GroupIntelligence and Information SectionDRDC Valcartier

KSCO 2012February 2012

Human-Computer Interaction with an Intelligence Virtual Analyst

2

iVAC initiative

• Intelligent Software Assistant (ISA) identified by MIT’s Technology Review as one of 2009’s most promising emerging technologies

• Conversational, computer-generated characters capable of providing guidance to a user in the conduct of his tasks

• Intelligence Virtual Analyst Capability (iVAC) R&D initiative at Defence R&D Canada

• 3-year initiative started in 2011

• Knowledge System with important HCI component

3

Virtual Assistant / Virtual Advisor

IBM's Watson supercomputerDARPA’s CALO Cognitive Assistant

that Learns and Organizes

SIRI Apps on IPhone 4s

4

DARPA’s Personal Assistant that Learns (PAL)

http://www.youtube.com/watch?v=BF-KNFlOocQ&feature=youtu.be

Video Clip presentation

5

Cognitive Assistant that Learns and Organizes (CALO)

6

HCI Requirements - General

• HCI dialogue between iVAC and the analyst(s)

– How to input knowledge into the system

– How to raise questions

– How to task the IVA

– How does the IVA ask for precisions on the questions and the way of presenting information

• Optimization of the presentation of the results

• One and/or multiple virtual analysts can serve multiple users

• Virtual analysts can interact with each other within and across meeting spaces

7

Enabling HCI Technologies

• Smart Room Environments

• Multimodal Interaction

• Information Presentation and Adaptive Interfaces

• Augmented Cognition

• ISA Representation / Avatars

• Storytelling

8

Smart Room Environments

• Instrumented environments (e.g. cameras, microphones, biometry)

• Ubiquitous computing. The user can access the system from various locations

• Track users location

• Track users activities

US Navy Command Center of the Future (CCoF)DSTO LiveSpaces

9

Multimodal Interaction

• Allow the user(s) to employ multimodal interaction (e.g. voice, pointing, gesture, eye/gaze, neural/brain interfaces, emotion detection)

• Resolve ambiguities

– These are my priorities

– We should move that milestone back 6 months

Xbox Kinect

AFRL Interactive Data WallEmotion Detection, Valenti et al. 2007

10

Information Presentation

• In support of various tasks

– Present tools and information

– Answer questions

– Ask for clarification

– Remind the user of some tasks or procedures

– Propose alternative possibilities

– Give a briefing

11

Information Presentation

• Means

– Voice output and/or information display

– Highlight information elements

– Display information in a new window, organized as the user wants to see it

– Gather a set of documents

– Filter information based on user requests

– Capture information about a meeting (automatic transcription and semantic analysis of multiparty meetings )

12

Adaptive Interfaces

• Customized to the role and current tasks of the user

• Present tools and information based on the user preferences

• Suggest sequences of events based on learned tasks and preferences

• Understand the context / tasks

• Observe what the user is doing and his mental state

• Do not disturb

• The user is recognized using biometry

• The interface adapts to the distance of the user to the (large) display (text fonts and granularity of information)

13

Augmented Cognition

• Track the sensory and cognitive overload of the users

• Use of neural/brain interfaces, emotion detection

• Employ computational strategies to restore operational effectiveness– Intelligent interruption to improve limited working memory

– Attention management to improve focus during complex tasks

– Cued memory retrieval to improve situational awareness and context recovery

– Modality switching (i.e., audio, visual) to increase information throughput

– When not to disturb

DARPA (2011a), DARPA Defense Science Office, Improving Warfighter Information Intake Under Stress (AugCog), http://www.darpa.mil/dso/archives/warfighter/index.htm

14

ISA Representation / Avatars

• Various representations

– Speech output (voice)

– A simple icon

– A two- or three-dimensional representation of a character (avatar)

PAL

15

ISA Representation / Avatars

• Adaptation of the avatar to the task and socio-cultural context– Facial and voice features of the avatar (serious / smiling, tone of the voice )

should communicate emotions, danger, risk, uncertainty

– Consider socio-cultural factors (Gender / age / ethnicity / profession)

– Use different avatars to support different ISA tasks (weather prediction or course of action)

• Embedding avatars in a video conferencing

• Be careful of the Spectre of the Uncanny Valley

• Need for experiences with avatar representations

DSTO’s FOCALwww.postgenomic.com/faces

16

Storytelling

• Guide audience through a preconstructed narrative

• Use of tangible graphics and videos (TV news like)

• Richness of the storytelling approach

• Convey situation awareness about complex events

Logo – VisWeek -Telling Stories with Data

17

ISA for Coalition Operations

• Improved interoperability between coalition forces in terms of disseminating information

• Translate information between languages

• Share differences in tactics, techniques and procedures (TTPs)

• Synchronizing coalition activities

• Improve cognitive assistance by providing shared coalition awareness and task support

• Enable coalition organizations to learn by managing a knowledge base

• Provide better and faster decision making through access to comprehensive knowledge developed through time

18

Conclusion

• ISA, a knowledge system component

• Solve the human cognitive overload

• Conducting a wide variety of tasks

– Search and organize information

– Learn procedures and preferences

– Manage schedules

– Track people

• HCI is Key

– Multimodal interaction

– Avatar is not mandatory

– Assign tasks– Summarize documents– Mediate interactions– Guide and remind the user

19