implementation of an affective conversational agent for controlling a hi- fi system

IMPLEMENTATION OF AN AFFECTIVE CONVERSATIONAL AGENT FOR CONTROLLING A

HI-FI SYSTEM

Justo Saavedra

CONTENTS

Máster en Sistemas Electrónicos para Entornos Inteligentes

•What´s the project about? Hi-Fi

•Objectives

Introduction

•Overview

•SOAP

System Architecture

•Needs

•Appraisals

•Emotions

•Dialogue Features

Emotional System

•Classes Implemented

Object Oriented Design

•Response Generation

•Synthesizer

Hi-Fi System

•Overview of the Supervisor

Supervisor Demonstration Video

ConclusionFuture Work

THE HI-FI SYSTEM

Front End

Speech Recognition

Dialogue ManagerDialogue Manager

Language Understanding

Module

Generation Response Module

SynthesizerSynthesizer

Execution Module

OBJECTIVES

Improve user satisfaction when interacting with dialogue module for controlling the Hi-Fi system…

The Adaptation of a psychological model of emotions to the servant agent.

Design and development of the architectural communication system to be used.

Improvement of the existing dialogue module.

Include the new Emotional Speech Synthesizer

PURPOSE

In the future emotional systems

will play an important role in the

development of intelligent/affective

systems.

Both recognition of the user emotions

(smile detection) and emotional response from the system are

highly desirable.

There is a wide field of

applications which can be benefited

from integrating an emotional system

(education, entertainment systems, social

robots, etc)

SYSTEM ARCHITECTURE

Reuse, as much as possible, the previous dialogue system.

The new servers in our dialogue system, similarly to the Galaxy-II, can be in the same machine or in different machines giving broad flexibility to operate and distribute computational processing

Must provide easiness to add/delete new modules and functionalities.

SYSTEM ARCHITECTURE

Main Program Loop (Hub)

ModuleDialogue Manager

Response Generator

Synthesiser Client

Synthesiser server

Emotional system server &

Supervisor

Opencv Client

Opencv Webcam server

Execution Module (IR,

Roomba)

Speech recognizer

SIMPLE OBJECT ACCESS PROTOCOL

SOAP is a lightweight protocol for exchange of information

Decentralized, distributed environment.

It is an XML-based portable protocol

Service RequesterEmotionClient

Service Provider EmotionServer Gets Emotions

EMOTIONAL SYSTEM

Ethics

Success

Social

Safety

Survival

Adaptation of the Maslow´s Hierarchy of Needs to a robotic system

EMOTIONAL SYSTEM

Ethics

Success

Social

Safety

Survival

Appraisals

Surprise Fear Happiness

Sadness Anger Shame

Neutral

EMOTIONAL SYSTEM

Ethics

Success

Social

Safety

Survival

Relevance• Importance of the

situation: Going critical/Overcoming

Desirability• degree of valence of

the change provoked by the event

Sadness Anger Shame

Neutral

EMOTIONAL SYSTEM

Ethics

Success

Social

Safety

Survival

Relevance

Desirability

Urgency• Estimates the time

to get to the critical level

Unexpectedness• Based on the

acceleration assuming a linear prediction model.

Sadness Anger Shame

Neutral

EMOTIONAL SYSTEM

Ethics

Success

Social

Safety

Survival

Relevance

Desirability

Urgency

Unexpectedness

Unfamiliarity• Is the state a

probable situation or not

Changeability• range of the need

variation

Sadness Anger Shame

Neutral

EMOTIONAL SYSTEM

Ethics

Success

Social

Safety

Survival

Relevance

Urgency

Unexpectedness

Unfamiliarity

Changeability

Controllability• Whether or not an

event is controllable depends on the capabilities

Priority (Weight)

Sadness Anger Shame

Neutral

EMOTIONAL SYSTEM

Ethics

Success

Social

Safety

Survival

RelevanceDesirabilityUrgencyUnexpectednessUnfamiliarityChangeabilityControllabilityPriority (Weight)

Sadness Anger Shame

Neutral

EMOTIONAL SYSTEM

Ethics

Success

Social

Safety

Survival

RelevanceDesirabilityUrgencyUnexpectednessUnfamiliarityChangeabilityControllabilityPriority (Weight)

EmotionKur urgency Ku unexpectedness Kf unfamiliarity Kagent

Kc changeability Kcont controlability

sign weight relevance Kd desirability

EmotionKur urgency Ku unexpectedness Kf unfamiliarity Kagent

Kc changeability Kcont controlability

sign weight relevance Kd desirability

SUCCESS RELATED DIALOGUE FEATURES

Speech Recognition Failures

Confidence Average

Max Confidence Recognized Words Percentage

High Confidence Understood Concepts Percentage

For each phrase recognized and processed by the HiFi, it calculates several features associated with the phrase, words, concepts and actions understood.

These features are passed to the emotional system in order to calculate the success events to be generated

OBJECT ORIENTED DESIGN

Several classes were implemented in order to provide code reusability and flexibility. Emotions and needs share attributes, and methods in order to calculate, and update its status

LevelTimeHistory

CIOProccess

CNeed CEmotion

• Statistics Methods (Mean, Standard Deviation)• Storage of History Levels

LevelTimeHistory

CIOProccess

CNeed CEmotion

•Process Running (Threads)

LevelTimeHistory

CIOProccess

CNeed CEmotion

CNeeds

Csurvival Csafety Csocial Csuccess Cethics

CEmotions

CAngry CShame Chappy Cfear Csad CSurprise

• Weight (Appraisals)• Critical Values• Need Formulas

• Time Constants• EmotionsUpdate(K) Formula

HI-FI SYSTEM->RESPONSE GENERATION

[V_CANCI][REPRO_CD]¿Qué canción del cedé [V_DISCO] quiere?.¿Qué canción del cedé [V_DISCO] quiere escuchar?.

[V_CANCI][REPRO_CD][FRIEND]Tío, ¿Qué canción del cedé [V_DISCO] quieres?.

[V_CANCI][REPRO_CD][FRIEND][SADNESS]Perdona tío, ¿Qué canción del cedé [V_DISCO] quieres?.

Allows the user to emulate the responses by providing input parameters (template, slot, status, value, relation, emotion). Its functionality was expanded to adapt emotional behaviour

HI-FI SYSTEM->RESPONSE GENERATION

Example:[V_CANCI][REPRO_CD][FRIEND][HAPPINESS]

1. Look for this template2. If not found -> Exclude the last optional parameter and look for

the new template [V_CANCI][REPRO_CD][FRIEND] 3. If not found -> Exclude the last optional parameter and look for

the new template [V_CANCI][REPRO_CD]4. If not found -> Failure

Templates:

[V_CANCI][REPRO_CD]¿Qué canción del cedé [V_DISCO] quiere?.¿Qué canción del cedé [V_DISCO] quiere escuchar?.

[V_CANCI][REPRO_CD][FRIEND]Tío, ¿Qué canción del cedé [V_DISCO] quieres?.

SYNTHESIZER

A new synthesizer developed by the GTH was used. It takes into account the emotions and the intensity value.

Surprise Fear

Happiness Sadness

Neutral Anger

NegativeEmotions

PositiveEmotions

Who wins? Emotional Phrase

SYNTHESIZER

Surprise 23.9

Fear 15.0

Happiness 25.0

Sadness 16.1

Neutral 10.0

Anger 10.0

Shame 0.0

NegativeEmotions

PositiveEmotions

Who wins? Emotional Phrase

SYNTHESIZER

Surprise 23.9

Fear 15.0

Happiness 25.0

Sadness 16.1

Neutral 10.0

Anger 10.0

Shame 0.0

Negative Emotions = 23.9+15+16.1+10 = 65

Positive Emotions = 23.9+25 = 48.9

Dominant: SadnessIntensity: 16.1/(23.9+15.0+25.0+16.1+10+10)

Wins: Negative Emotions

SUPERVISOR

Monitors/Simulate/Debug/Fine-Tune

• Monitoring of the emotions• Appraisal calculations due to certain

interaction• Decay due to time of the different

needs\emotions values

Its users interface

permits real time

monitoring

SUPERVISOR->HOW IT WORKS?

• Using SOAP, the supervisor system interacts with emotional system (both send and receive data).

• The supervisor as well as the emotional system acts as a client/server depending if they are receiving or sending data.

Emotional SystemHi-Fi Supervisor

SUPERVISOR

CONCLUSIONS

The affective system implemented and the improvement of the dialogue response generation permits the future evaluation of the system (Compare with previous non-emotional model)

The architecture and features of the affective system (independent of task) make it possible to adapt it to other applications domains (games, tutoring systems)

The communication architecture implemented give the flexibility to change/include/delete modules with easiness

FUTURE WORK

Implementing more dialog features

Fill in the necessary templates to provide better response generations

Implement more events (ethic/social)

Evaluate the system

DEMONSTRATION

QUESTIONS?

implementation of an affective conversational agent for controlling a hi- fi system

Documents

t7 embodied conversational agents and affective computing

conversational conversational spanish classes - … ·...

conversational training

conversational connectors

conversational structure

conversational -...

conversational discourse

conversational implicatures

grandchair: conversational collection of grandparents'...

conversational leadership

generalized conversational

conversational apologetics

toward affective education: a guide to developing affective

conversational theory

conversational evangelism

watson apis for conversational apps - conversational apps...

€¦ · conversational commerce conversational commerce...

conversational commerce

conversational japanese

assessment of affective outcomes -...