1 chap. 13 expressive behaviors for virtual worlds stacy marsella, jonathan gratch, and jeff rickel...
TRANSCRIPT
1
Chap. 13Chap. 13Expressive Behaviors for Virtual WorldsExpressive Behaviors for Virtual Worlds
Stacy Marsella, Jonathan Gratch, and Jeff Rickel
November 22, 2004
2
OutlineOutline
• Introduction
• Steve• Jack and Steve• Carmen’s Bright IDEAS (CBI)
• Mission Rehearsal Exercise (MRE)
• Conclusion
3
IntroductionIntroduction• Goal of this chapter
– To create virtual humans that convey much information to humans while interacting with them in virtual worlds
– Information includes non-verbal behaviors concerning emotional state
• Requirements for virtual human design– Believable: they must provide human-like behaviors– Responsive: they must respond to the events surrounding them– Interpretable: the user must be able to interpret their response to
situation
• This chapter describes– The progress toward a model of outward manifestations of an
agent’s cognitive and emotional state– The review of three prior systems that influenced model of this
chapter– The model of this chapter is an integration of prior systems
4
ContributionContribution
• Previous systems provides impressive capabilities in its area of research focus, but they had some limitations
• Limitation of previous systems– Agents are modeled only for conversation between two of them– Collaboration of users and agents on one task is not allowed– Agent’s presence was partially limited to a 2D– Agent’s movement and relationships are limited
• Steve’s contribution– Only Steve can interleave task-related behaviors and face-to-face
dialogue with humans and virtual humans in dynamic virtual worlds
– Interleaving 은 성능을 높이기 위해 데이터가 서로 인접하지 않도록 배열하는 방법
Steve
5
The ArchitectureThe Architecture
Steve
• Steve consists of three modules: perception, cognition and motor control
SpeechRecognition
Graphics Simulator Speech Synthesis
Steve
Domain knowledge
General capabilities
Motor commands
Cognition
MotorControl Perception
Translate intomovements, speech
Broadcast toenvironment
Filter, assembleinto coherent view
Monitor events
Current state
Commands toenvironment
Eventnotifications
6
PrimitivesPrimitives
• The cognition module generates Steve’s communicative behavior by dynamically selecting next action from a list of primitives
• Primitives– Speak– Move to an object– Manipulate an object– Visually check an object– Point at an object– Give tutorial feedback– Offer turn– Listen to student– Wait for someone
Steve
– Acknowledge an utterance– Drop hands– Attend to Action
7
React to Interruption & Context (1)React to Interruption & Context (1)• To react to interruption
– In dynamic virtual worlds, to react to the unexpected events is important for task-oriented collaboration
– It is important to maintain coherence of an agent’s behavior
• Steve considers 2 separate but complementary types of context
1. Task context– Continually monitors the virtual world– Uses the task model to plan to complete the task – Uses variant of partial-order planning techniques
2. Dialogue context– Represents the state of interaction between a student and Steve– Uses focus stack: represents the hierarchy of tasks, subtasks, and
currently engaged actions– Maintains a record, Steve’s answer etc.
Steve
8
React to Interruption & Context (2)React to Interruption & Context (2)
• Based on the current task and dialogue contexts, Steve can choose next action
• Steve can choose an action of following three roles– Steve can respond to the student to know a student’s request– Steve can choose for himself how to advance the collaborative
dialogue– Steve can choose a turn-taking or grounding act that helps
regulate the dialogue without advancing the task
• In case several actions are appropriate at one moment, – Steve choose next action considering priorities– Actions with low priority will be performed later, not be deferred
• Summary– Steve does not include an emotion model, but– It serves a valuable foundation for MRE model
Steve
9
Jack and Steve vs. SteveJack and Steve vs. Steve• The Jack and Steve system was predicated on two basic claims:
– Plans and plan reasoning can mediate expressive behavior– Small biases in how plans are evaluated and generated can result
in large systematic differences in agent behavior
• Jack and Steve vs. Steve
Jack and Steve
Jack and Steve Steve
To support biases in plan evaluation, J-S include a richer plan representation
To translate small biases into large external variations, J-S focused on plan generation
Steve focused more on plan execution and repair
J-S explored how biases in the plan generation process could support a variety of non-collaborative interactions as well, but as the focus was exploring systematic differences in joint behavior, agents only interact with other computational agents and not with human users
Steve focused on collaborative interactions between agents and human users
10
Motivating ExampleMotivating Example
• This example shows two separate runs of system where the only difference is a change in the personality of Steve (rude / fair)
– Jack: I want to make some big money– Steve: I want to catch some waves
Jack and Steve
Mental state of each agent at specific point in the interaction
Mental state shows individual action, plan, emotion etc.
11
Plan-Based Social Appraisal (1)Plan-Based Social Appraisal (1)
• Jack and Steve supports expressive and flexible interactions by implementing social reasoning as a layer atop a general-purpose planning system
• Planning system provides – Domain-independent representations of world actions– General reasoning mechanisms that construct partial plans, repair
interactions among them, and oversee plan execution• The social layer
– Manages communication – Biases plan generation and execution according to social
context
Jack and Steve
General purpose planning system
Social reasoning layer
12
Plan-Based Social Appraisal (2)Plan-Based Social Appraisal (2)
• To support various social interactions, the social reasoning layer must provide a rich model of the social context
• The social situation is described in terms of many static and dynamic features
• Static features – include innate properties of the character– social role and small set of personality variables
• Dynamic features – Are derived from inference procedure that operate on the current
mental state of agent– Include - current communication obligation
- various relations between plans in memory (e.g. your plans threaten my plans) - a model of the emotional state of the agent
Jack and Steve
13
Plan-Based Social Appraisal (3)Plan-Based Social Appraisal (3)
• One novel aspects of this system is that the social layer alters the planning process fundamentally
• In terms of planning, the social layer can bias planning to be more or less considerate to the goals of other participants
• In terms of communication, agents can vary from bossy agents that try to tell others what to do to passive agents that avoid interactions or social conflicts
Jack and Steve
14
Social Context, Operator, RuleSocial Context, Operator, Rule
• Jack and Steve system maintains a rich representation of the social context
• The social context of Jack and Steve is divided into 3 categories of plan context, emotional context, and communicative context
• Social operators are actions that occur at the social level• Social operators are subdivided into meta-planning operators
and communicative operators.
• Distinct personalities are implemented via a set of social rules• Social rules execute sequences of social operators based on
appraised features of the social context• Jack and Steve system include 30 social rules
Jack and Steve
15
IntroductionIntroduction
• Carmen’s Bright IDEAS (CBI)– An agent-based system designed to realize an Interactive
Pedagogical Drama– The pedagogical goal is to help mothers of pediatric cancer
patients– The drama mirrors the mother’s own problems– It allows the learner to interactively influence how Carmen
copes with problems
• In CBI– to model the causes of emotions is important– agents needed effective ways to convey the impact of emotion
on both the agent’s dialogue and physical behavior
Carmen’s Bright IDEAS
16
The Drama of Carmen’s Bright IDEASThe Drama of Carmen’s Bright IDEAS
• CBI is an interactive pedagogical drama• Carmen discusses her problems with a clinical counselor, Gina• Gina suggests a problem solving technique called Bright IDEAS• Bright IDEAS means
– Bright: positive attitude– I: Identify a solvable problem– D: Develop possible solution– E: Evaluate your options– A: Act on your plan– S: See if it worked
• User, the human mother, interacts with the drama by making choice for Carmen
• Both Gina’s dialogue and the user’s choices influence the cognitive and emotional state of Carmen (in conflicting ways)
Carmen’s Bright IDEAS
17
Interaction ModelInteraction Model
Carmen’s Bright IDEAS
• In this interaction model, the user gets a vivid demonstration
• Learner influences Carmen by selecting thought balloons
18
CBI Agent ModelCBI Agent Model
Carmen’s Bright IDEAS
• IPD have 5 main components, but the discussion will focus on the on-screen character agents
• Each on-screen character is realized by an agent architecture that has modules for problem solving, dialogue, emotional appraisal, and behavior generation
19
Emotional ModelEmotional Model• The pedagogical goal was for the mothers to learn how to choose
and carry out the right coping strategy for a given situation
• Cognitive appraisal theory by Richard Lazarus influences CBI– This organizes human behavior around appraisal and coping– Appraisal leads to emotion by assessing the person-environment
relationship– Coping and appraisal interact and unfold over time, supporting the
temporal character of emotion evident in human behavior
• Coping is the process of dealing with emotion– Problem-focused coping– Emotion-focused coping
Carmen’s Bright IDEAS
20
RemarksRemarks
• The design of dialogue, emotions, and expressive behavior systems was driven by a need to convey deep inner conflicts and how those conflicts play out expressively
Carmen’s Bright IDEAS
21
IntroductionIntroduction• The Mission Rehearsal Exercise (MRE) system
– Brings ideas from each of the preceding systems– Creates a broader and more flexible array of expressive behaviors– The goal is to teach leadership skills in high-stakes social situation
• To model such dramatic and interactive scenario, the MRE – Combines Steve’s ability to flexibly interact with a human user– Augment it with the richer social and emotional behaviors of CBI
and Jack and Steve
• The MRE combines a variety of capabilities in service of realistic and natural collaboration with virtual humans
• Example
Mission Rehearsal Exercise
22
Cognition and EmotionsCognition and Emotions
• Our virtual humans must– Provide realistic expressive behaviors– Require cognitive machinery to recognize which behaviors are
appropriate in the course of an unscripted interaction with a human user
• The Steve system is selected as a starting point since it supports flexible face-to-face interactions with a human user
• Integrate plan-based appraisal model of the Jack and Steve system into the Steve system
• Integrate CBI’s coping model– Coping strategies were recast explicitly into procedures that
updated Steve’s task representations and reasoning processes
Mission Rehearsal Exercise
23
Cognition and EmotionsCognition and Emotions
Mission Rehearsal Exercise
Steve
Plan-based appraisal modelof Jack and Steve
Coping model of Carmen’s Bright IDEAS
Steve supports flexible face-to-face interaction with human users
This allows agents to negotiate over tasks and express emotions, i.e. task
model is extended in a number of ways to represent the socio-emotional
context
Integrating CBI’s coping model results in a tight integration between
appraisal, coping and task reasoning that closely follows the cognitive
appraisal theory of Richard Lazarus
+
+
24
Physical BehaviorPhysical Behavior• Virtual humans in the MRE attempt to manifest the rich
dynamics of this cognitive and emotional inner state through each character’s external behavior
• The key challenge is the range of behaviors that must be integrated: Each character’s body movements must reflect – Its awareness of events in the virtual world– Its physical actions– The number of non-verbal signals that accompany speech
during social interaction– Its emotional reactions
• Expressive physical behavior in the MRE agents integrates the task-related non-verbal behaviors of the Steve system and the coping behaviors of CBI
Mission Rehearsal Exercise
25
ConclusionConclusion• Steve serves as the foundation for the virtual humans in MRE
system as his ability to interleave task-related behaviors and face-to-face dialogue in dynamic virtual worlds
• The Jack and Steve system contributed a model of task-oriented emotional appraisal and a model of socially situated planning
• The CBI system contributed a complementary model of emotional appraisal focusing on social relationships
• MRE virtual humans integrate many of the ideas from these three prior systems