Emotive Driver Advisory System

(EDAS)

Oleg GusikhinFord Research & Advanced Engineering

Dearborn, Michigan, USA

Outline

• Introduction • Background and Ford SYNC• Emotive Driver Advisory System

– Emotive Spoken Dialogue System – Emotive Spoken Dialogue System – Avatar as an Advanced Automotive HMI– Cloud-based Infotainment

• Marketing & Business Impact

Outline

• Introduction • Background and Ford SYNC• Emotive Driver Advisory System

– Emotive Spoken Dialogue System – Emotive Spoken Dialogue System – Avatar as an Advanced Automotive HMI– Cloud-based Infotainment

• Marketing & Business Impact

Ford Motor Company

• Global automotive industry leader based in Dearborn, MI.

• Manufactures and distributes automobiles in 200 markets across automobiles in 200 markets across six continents.

Ford Brands 2010

Ford Technology

Ford SYNC MyFord Touch

Ford Research & Advanced EngineeringDearborn, USA

Aachen, Germany

� Information Services & Connectivity

Manufacturing, Vehicles Design & Safety Research Lab

Outline

• Introduction • Background and Ford SYNC• Emotive Driver Advisory System

– Emotive Spoken Dialogue System – Emotive Spoken Dialogue System – Avatar as an Advanced Automotive HMI– Cloud-based Infotainment

• Marketing & Business Impact

The Challenge

Emotive Driver Advisory System (EDAS)

Aibo

Emotions in Engineering

• Affective (Kansei) Engineering– a method for translating feelings and

impressions into product parameters

• Affective Computing– design of systems and devices that can

recognize and/or simulate human emotions

Emotive Driver Advisory System

• Natural Interface• Personalization• Customization• Context-Awareness• Context-Awareness• Adaptable Interaction and Intelligence• Seamless Integration of Vehicle

Infotainment and Cloud-based Services

Ford SYNC

(CAN & USB) (Bluetooth) www.syncmyride.com

CAN BUS

SYNC Services

• Standard Services

�911 Assist

�Vehicle Health Report

• Subscription Services• Subscription Services

�Traffic

�Directions

�Business Search, News, Sports, Weather

Outline

• Introduction • Background and Ford SYNC• Emotive Driver Advisory System

– Emotive Spoken Dialogue System – Emotive Spoken Dialogue System – Avatar as an Advanced Automotive HMI– Cloud-based Infotainment

• Marketing & Business Impact

EDAS Overview

Speech Recognition

Semantic Interpretation

Dialog Manager

Spoken Dialog System

Input Manager

Task ManagerApplication

3d party Web

services

Home/

Display NAV Audio

Buttons

Voice

Traffic/Weather AlertsRefueling AdviceSpeed Limit Changes Maintenance ConcernsPersonalized InfotainmentLocation-Based Notes Social Networking

Emotive Response

Emotive Response

Vehicle CAN BUS

Data ManagerDriver State & Driving Conditions Analyzer

Vehicle

Portal

User/ Vehicle Profile

Application Services Office

Server Speech Portal

Vehicle Apps Safety Performance Convenience

Video

Customization Preferences Learning Collaborative Filtering

Driving Conditions Driver State

Emotive Text-to-Speech

Response Generator

Response

Outline

• Introduction • Background and Ford SYNC• Emotive Driver Advisory System

– Emotive Spoken Dialogue System– Emotive Spoken Dialogue System– Avatar as an Advanced Automotive HMI– Cloud-based Infotainment

• Marketing & Business Impact

Spoken Dialogue System (SDS)• Conversational • Communication does not need to be

hierarchical / menu driven• Supports Open Mic / Push-to-Talk• Supports Open Mic / Push-to-Talk• Multimodal (speech and/or button based

interaction)• Context aware & emotive dialogue

Emotion Recognition of Driver Ways to detect emotions:

– Facial expressions: captured by camera– Voice: input through microphone

Emotion Vector

– Biometrics: steering wheel, heart monitor– Vehicle operating pattern: aggressiveness,

workload

Emotive Conversational DialogKey Word Dialogue Interface Conversational Dialogue Interface

Speech Recognition:

converts recognized key

word to text

keyword Natural phrase

Speech Recognition: acoustic +

language models

words

keyword

Dialogue Manager

Input manager: Semantic

interpretation

Dialogue Manager

Output Manager

Output Manager

emotions

Emotion-based dialogue strategy

If Mood = good &

Driver sends request: Monitor Fuel Economy

Using Driver’s Emotions to Modify Responses

If Mood = bad &Performance = bad

If Mood = good &Performance = bad

Check Driver’s mood

Your hard acceleration is negatively affecting

your gas mileage.

The way you’re driving right now is hurting your fuel economy.

Please ease up on the pedal.

Dialogue Interface Architecturedriver speaks into microphone

SpeechRecognition

[Nuance speech recognition engine

InputManager

[Semantic

DialogueManager

[Determines system responses, e.g. send

Words Interpretation

Spoken Dialogue System (SDS)

Speech Signal Emotion

recognition software

Driver’s emotional state

EDAS systemqueries the web

sends key words in text file Query

ResultStorage

CereProc text to speech

recognition engine +Language Model built using Regulus]

interpretation, built using Regulus]

responses, e.g. send keywords to EDAS or navigate existing query result]

Output Manager

Actions tobe performed

sends xml file in special format

sends emotional tag

Output message

Outline

• Introduction • Background and Ford SYNC• Emotive Driver Advisory System

– Emotive Spoken Dialogue System – Emotive Spoken Dialogue System – Avatar as an Advanced Automotive HMI– Cloud-based Infotainment

• Marketing & Business Impact

Motivation for Avatar• Universal Multivariate Gage (Chernoff

Face)• Reference Point of Vehicle Intelligence • Agent of Customization • Agent of Customization • Emotive Supplement to Text-to-Speech• More natural interface

EDAS Prototype 2005

Avatars

Customizable Avatars, Voices, Names, & Background Scene

Jackie

Toby Shouting

Output Responses from Avatar

Driver Emotion Task

Status

Request

Emotions Status

Facial expressions

Tone of Voice

Emotive TTS

Emotion Recognition & Emotive Feedback

Playful

Emotion Recognition

Hey Jackie

SeriousJacki

FormalHello Jacki

e

SeriousJackie

Example clinic results

Example clinic results

Outline

• Introduction • Background and Ford SYNC• Emotive Driver Advisory System

– Emotive Spoken Dialogue System – Emotive Spoken Dialogue System – Avatar as an Advanced Automotive HMI– Cloud-based Infotainment

• Marketing & Business Impact

Benefits of Integration of Vehicle Infotainment and Cloud-based Services• Enable on-demand, personalized infotainment services

• Access to real-time information

• Enable continuum of personalized information and entertainment services between on-vehicle and off-vehicle activities

• Utilize power of social network collaborative filtering in recommendation services

• Outsource Computing Power to Remote Server

• Simplify maintenance and upgrade of infotainment applications

Examples of EDAS Services

• POIs• News• Music• Refueling advice• Refueling advice

POIs & News

• POIs– Facilitates Social Network Collaborative Filtering –

based Recommendations– Allow consolidation of relevant information scattered

throughout the internet for enhanced decision analysis and supportanalysis and support

• News– Personalized Interactive News Radio– Support Continuum of Personalized Information

Services between in-vehicle and off-vehicle environment

POI & NEWS Services DEMO

Personalized Music Radio

Private Collection Broadcast

Existing In-Vehicle Music Services

Personalized Music Radio

Internet Personalized Music Services

• Creates custom stations based on artists or individual songs

• Learns preferences by analyzing the musical qualities of songs listener like and dislike

Pandora Music Radio

of songs listener like and dislike

• Plays Music that matches personal preferences using AI and song genome database for song selection

EDAS Pandora Player

Intelligent Refueling Gauge

• Given a set of routes over a time period – When to buy gas– Where to buy gas– How much gas to buy

I recommend that you stop today at the

Sunoco station because gas prices

will be going up tomorrow.

System Architecture

Current Gasoline

Prices

OPIS (US)

Experian Catalist (UK)

Historical Gasoline

Prices

Wholesale Gasoline

Prices

Optimization

GPS Speed Fuel Consumption Fuel Level

Vehicle Profile Predicted Routes & Fuel Consumption

Price ForecastGasoline Station Markup ($/gallon)

(Oaks Petro Mart, 9 mile road)

-0.5

0

0.5

1

9/9

/200

7

9/1

6/2

007

9/2

3/2

007

9/3

0/2

007

10

/7/2

007

10/1

4/2

007

10/2

1/2

007

10/2

8/2

007

Date

Mar

ku

p ($

/gal

lon

)

Markup ($/gallon)

M

Sa

Home/ Office

Outline

• Introduction • Background and Ford SYNC• Emotive Driver Advisory System

– Emotive Spoken Dialogue System – Emotive Spoken Dialogue System – Avatar as an Advanced Automotive HMI– Cloud-based Infotainment

• Marketing & Business Impact

EDAS at CES & NAIAS 2009

EDAS Media Coverage

SYNC Pandora

Music Genome Project Database

Pandora Server

User Profile

SYNC API

AcknowledgmentsY. Chen Vehicle DesignD. Filev Active SafetyS. Di Cairano P/T ControlsT.J. Giuli Vehicle DesignE. Klampfl Systems AnalyticsI. Kolmanovsky P/T ControlsP. MacNeille Systems AnalyticsP. MacNeille Systems AnalyticsJ. Michelini P/T ControlsK. Prakah-Asante Active SafetyN. Rychtyckyj ITF. Syed V&B ControlS. Szwabowski P/T ControlsC. Teslak P/T ControlsK. Theisen Systems AnalyticsB. Tonshal Vehicle Design