implementation of an affective conversational agent for controlling a hi- fi system

IMPLEMENTATION OF AN AFFECTIVE CONVERSATIONAL AGENT FOR CONTROLLING A

HI-FI SYSTEM

Justo Saavedra

CONTENTS

Máster en Sistemas Electrónicos para Entornos Inteligentes

•What´s the project about? Hi-Fi

•Objectives

Introduction

•Overview

•SOAP

System Architecture

•Needs

•Appraisals

•Emotions

•Dialogue Features

Emotional System

•Classes Implemented

Object Oriented Design

•Response Generation

•Synthesizer

Hi-Fi System

•Overview of the Supervisor

Supervisor Demonstration Video

ConclusionFuture Work


THE HI-FI SYSTEM

Front End

Front End

Speech Recognition

Speech Recognition

Dialogue ManagerDialogue Manager

Language Understanding

Module


Module

Generation Response Module

Generation Response Module

SynthesizerSynthesizer

Execution Module

Execution Module

OBJECTIVES


Improve user satisfaction when interacting with dialogue module for controlling the Hi-Fi system…

The Adaptation of a psychological model of emotions to the servant agent.

Design and development of the architectural communication system to be used.

Improvement of the existing dialogue module.

Include the new Emotional Speech Synthesizer


PURPOSE

Why?

In the future emotional systems

will play an important role in the

development of intelligent/affective

systems.

Both recognition of the user emotions

(smile detection) and emotional response from the system are

highly desirable.

There is a wide field of

applications which can be benefited

from integrating an emotional system

(education, entertainment systems, social

robots, etc)

SYSTEM ARCHITECTURE


Reuse, as much as possible, the previous dialogue system.

The new servers in our dialogue system, similarly to the Galaxy-II, can be in the same machine or in different machines giving broad flexibility to operate and distribute computational processing

Must provide easiness to add/delete new modules and functionalities.

SYSTEM ARCHITECTURE


Main Program Loop (Hub)


ModuleDialogue Manager

Response Generator

Synthesiser Client

Synthesiser server

Emotional system server &

Supervisor

Opencv Client

Opencv Webcam server

Execution Module (IR,

Roomba)

Speech recognizer

SIMPLE OBJECT ACCESS PROTOCOL


SOAP is a lightweight protocol for exchange of information

Decentralized, distributed environment.

It is an XML-based portable protocol

Service RequesterEmotionClient

Service Provider EmotionServer Gets Emotions

EMOTIONAL SYSTEM


Ethics

Success

Social

Safety

Survival

Adaptation of the Maslow´s Hierarchy of Needs to a robotic system

EMOTIONAL SYSTEM


Ethics

Success

Social

Safety

Survival

Appraisals

Surprise Fear Happiness

Sadness Anger Shame

Neutral

Task1

Task2

Task3

EMOTIONAL SYSTEM


Ethics

Success

Social

Safety

Survival

Relevance• Importance of the

situation: Going critical/Overcoming

Desirability• degree of valence of

the change provoked by the event


Sadness Anger Shame

Neutral

Task1

Task2

Task3

EMOTIONAL SYSTEM


Ethics

Success

Social

Safety

Survival

Relevance

Desirability

Urgency• Estimates the time

to get to the critical level

Unexpectedness• Based on the

acceleration assuming a linear prediction model.


Sadness Anger Shame

Neutral

Task1

Task2

Task3

EMOTIONAL SYSTEM


Ethics

Success

Social

Safety

Survival

Relevance

Desirability

Urgency

Unexpectedness

Unfamiliarity• Is the state a

probable situation or not

Changeability• range of the need

variation


Sadness Anger Shame

Neutral

Task1

Task2

Task3

EMOTIONAL SYSTEM


Ethics

Success

Social

Safety

Survival

Relevance

Urgency

Unexpectedness

Unfamiliarity

Changeability

Controllability• Whether or not an

event is controllable depends on the capabilities

Priority (Weight)


Sadness Anger Shame

Neutral

Task1

Task2

Task3

EMOTIONAL SYSTEM


Ethics

Success

Social

Safety

Survival

RelevanceDesirabilityUrgencyUnexpectednessUnfamiliarityChangeabilityControllabilityPriority (Weight)


Sadness Anger Shame

Neutral

Task1

Task2

Task3

EMOTIONAL SYSTEM


Ethics

Success

Social

Safety

Survival

RelevanceDesirabilityUrgencyUnexpectednessUnfamiliarityChangeabilityControllabilityPriority (Weight)

Task1

Task2

Task3

EmotionKur urgency Ku unexpectedness Kf unfamiliarity Kagent

Kc changeability Kcont controlability

sign weight relevance Kd desirability

100

EmotionKur urgency Ku unexpectedness Kf unfamiliarity Kagent

Kc changeability Kcont controlability

sign weight relevance Kd desirability

100

SUCCESS RELATED DIALOGUE FEATURES


Speech Recognition Failures

Confidence Average

Max Confidence Recognized Words Percentage

High Confidence Understood Concepts Percentage

For each phrase recognized and processed by the HiFi, it calculates several features associated with the phrase, words, concepts and actions understood.

These features are passed to the emotional system in order to calculate the success events to be generated

OBJECT ORIENTED DESIGN


Several classes were implemented in order to provide code reusability and flexibility. Emotions and needs share attributes, and methods in order to calculate, and update its status

LevelTimeHistory

CIOProccess

CNeed CEmotion



• Statistics Methods (Mean, Standard Deviation)• Storage of History Levels

LevelTimeHistory

CIOProccess

CNeed CEmotion



•Process Running (Threads)

LevelTimeHistory

CIOProccess

CNeed CEmotion



CNeeds

Csurvival Csafety Csocial Csuccess Cethics

CEmotions

CAngry CShame Chappy Cfear Csad CSurprise

• Weight (Appraisals)• Critical Values• Need Formulas

• Time Constants• EmotionsUpdate(K) Formula

HI-FI SYSTEM->RESPONSE GENERATION


[V_CANCI][REPRO_CD]¿Qué canción del cedé [V_DISCO] quiere?.¿Qué canción del cedé [V_DISCO] quiere escuchar?.

[V_CANCI][REPRO_CD][FRIEND]Tío, ¿Qué canción del cedé [V_DISCO] quieres?.

[V_CANCI][REPRO_CD][FRIEND][SADNESS]Perdona tío, ¿Qué canción del cedé [V_DISCO] quieres?.

Allows the user to emulate the responses by providing input parameters (template, slot, status, value, relation, emotion). Its functionality was expanded to adapt emotional behaviour

HI-FI SYSTEM->RESPONSE GENERATION


Example:[V_CANCI][REPRO_CD][FRIEND][HAPPINESS]

1. Look for this template2. If not found -> Exclude the last optional parameter and look for

the new template [V_CANCI][REPRO_CD][FRIEND] 3. If not found -> Exclude the last optional parameter and look for

the new template [V_CANCI][REPRO_CD]4. If not found -> Failure

Templates:

[V_CANCI][REPRO_CD]¿Qué canción del cedé [V_DISCO] quiere?.¿Qué canción del cedé [V_DISCO] quiere escuchar?.

[V_CANCI][REPRO_CD][FRIEND]Tío, ¿Qué canción del cedé [V_DISCO] quieres?.

SYNTHESIZER


A new synthesizer developed by the GTH was used. It takes into account the emotions and the intensity value.

Surprise Fear

Happiness Sadness

Neutral Anger

Shame

NegativeEmotions

PositiveEmotions

Who wins? Emotional Phrase

SYNTHESIZER



Surprise 23.9

Fear 15.0

Happiness 25.0

Sadness 16.1

Neutral 10.0

Anger 10.0

Shame 0.0

NegativeEmotions

PositiveEmotions

Who wins? Emotional Phrase

SYNTHESIZER



Surprise 23.9

Fear 15.0

Happiness 25.0

Sadness 16.1

Neutral 10.0

Anger 10.0

Shame 0.0

Negative Emotions = 23.9+15+16.1+10 = 65

Positive Emotions = 23.9+25 = 48.9

Dominant: SadnessIntensity: 16.1/(23.9+15.0+25.0+16.1+10+10)

Wins: Negative Emotions

SUPERVISOR


Monitors/Simulate/Debug/Fine-Tune

• Monitoring of the emotions• Appraisal calculations due to certain

interaction• Decay due to time of the different

needs\emotions values

Its users interface

permits real time

monitoring

SUPERVISOR->HOW IT WORKS?


• Using SOAP, the supervisor system interacts with emotional system (both send and receive data).

• The supervisor as well as the emotional system acts as a client/server depending if they are receiving or sending data.

Emotional SystemHi-Fi Supervisor

SUPERVISOR


CONCLUSIONS


The affective system implemented and the improvement of the dialogue response generation permits the future evaluation of the system (Compare with previous non-emotional model)

The architecture and features of the affective system (independent of task) make it possible to adapt it to other applications domains (games, tutoring systems)

The communication architecture implemented give the flexibility to change/include/delete modules with easiness

FUTURE WORK


Implementing more dialog features

Fill in the necessary templates to provide better response generations

Implement more events (ethic/social)

Evaluate the system

DEMONSTRATION


QUESTIONS?


implementation of an affective conversational agent for controlling a hi- fi system

Documents