Agents and Avatars

Ruth Aylett

Overview

!  Agents and Avatars !  Believability v naturalism !  Building the body !  H-anim !  Moving and sensing

IVAS

!  Intelligent Virtual Agents (IVAs) –  Also:

•  Synthetic characters •  Embodied conversational characters (ECAs) •  Virtual humans •  BUT do not have to be humanoid…

–  Embodied and autonomous •  Require a control architecture - or ‘agent mind’ •  Insides v outsides: combining AI with graphics

–  ‘Inhabit’ a virtual environment

Why IVAs?

!  Adding life to a VE –  Animals, birds, insects –  Crowds

•  E.g. students in the virtual campus –  Increase sense of presence

!  As a guide or teacher –  Front-end to embedded knowledge in a VE

!  As a character in a story –  Computer games

!  Interface agents –  On web pages to make them more human –  Personal representative –  Sales rep

Why IVAs? - 2

!  As virtual actors –  Instead of extras –  Immersive Education Media Stage

!  As part of a simulation –  Hostage release training –  Battlefield medical training

!  Scientific investigation –  Buildings evacuation –  Testing for disability friendliness –  Ecology and animal behaviour

Avatars

!  Hindu Sanskrit Term - “Representation of a Deity in visible form” –  Snow Crash (Neil Stephenson 1992) - “A

graphical representation of yourself”

!  VE representation of the user –  Embodiment need not be humanoid

!  Driven by the user –  So NOT autonomous –  A mapping rather than a control problem

Video avatars

! “magic video mirror” with back-projection

! video image mixed with computer generated overlay

Creating a video avatar (ALIVE, MIT)

• unencumbered interaction between human visitor and virtual character based on position, postures, and gestures

• main focus on “virtual presence” http://www.ai.univie.ac.at/oefai/agents/

Believability

!  Term introduced by Joe Bates of the OZ group at CMU in the 1990s –  Combined art and technology

!  Very hard to define –  A willing suspension of disbelief? –  Seem like ‘real’ characters? –  The ‘illusion of life’? –  Willingness to attribute an internal state

•  Ascribe intentionality

Believability and Naturalism

!  Are they the same thing? !  Graphics people seem to think so

–  Is Mickey a real mouse? –  Is he a believable character?

The uncanny valley

!  The ‘uncanny valley’ –  Work by a Japanese

researcher, Mori –  Acceptability v

naturalism

!  Goes -ve as becomes ‘nearly human’

SEE: http://en.wikipedia.org/wiki/Uncanny_Valley

The problem of expectations

!  Humans have hard-wired expectations –  Used to interpret inter-personal behaviour –  Fundamental social skill

!  Need to invoke this very carefully –  Acceptability of movement

•  Lip sync a key problem

–  Interactive responsiveness •  Memory of interaction a key issue •  Games create a real problem with instant rewind

Building a body

!  Use of 3D modelling package –  3ds Character Studio; Poser etc

!  Creation of skeleton –  Required for animation later

!  Polygonal body –  But overall count must be low for real-time

interaction, low 000s or less if possible

!  Texture to cover body –  One body, many textures?

Texture Images

X 4

Silhouette Images

X 4

Stage 1: Image Capture

• Generic Avatar

• Polygonal Structure

• 1500 Polygons

Change Data: Deforms the Mesh

Stage 2: Deform the Mesh

Apply Textures to Deformed

Mesh

Stage 3: Apply the Textures

The touch up process

What sort of motion?

!  Walking around –  Jumping, running, swimming –  Human and non-human

!  Picking things up –  Agent-agent interaction

!  Gesture !  Talking heads

–  Facial muscles, lip synch

!  Group Movement –  Crowds, flocks, herds

Moving the body

!  Animation – Drawing on cartoon animation

!  Motion capture – Drawing on gait analysis and then film

!  PBM – Drawing on robotics

• Analytically calculated •  Learned

Animation

!  Time consuming for good results – Requires artistic skills

!  The main character, Woody, in Toy Story had: – 700 degrees of freedom (200 for face and

fifty for the mouth) – 150 people (at Pixar) would generate 3

minutes of animation a week !  Can it be standardised?

Self-animation

!  Poses new problems: – Parametrisable animation very desirable

• E.g a walk that could be reused with different stride length or foot height

– Melding and combining of animations ‘on the fly’ • Not just a morphing problem •  ‘starting position’?

– Extra actions inserted by character

Standardising animation?

!  Animation depends on the structure of skeleton that is being animated

!  Can we standardise a humanoid skeleton? – H-anim just such an attempt – Originally in the context of VRML – See www.h-anim.org

H-anim

!  A set of standard components –  Humanoid: root of a figure –  Joint: attached using transform specifying

current state of articulation plus geometry associated with attached body part

–  Segment: specifies attributes of physical links between joints

–  Site: where can add semantics –  Displacer:range of movement allowed for object

in which embedded

H-anim

And in X3d..

<HAnimJoint DEF='hanim_l_hip' center='0.0961 0.9124 -0.0001' name='l_hip'>

<HAnimSegment DEF='hanim_l_thigh' name='l_thigh'/> <HAnimJoint DEF='hanim_l_knee' center='0.1040 0.4867 0.0308' name='l_knee'> <HAnimSegment DEF='hanim_l_calf' name='l_calf'/> <HAnimJoint DEF='hanim_l_ankle' center='0.1101 0.0656 - 0.0736' name='l_ankle'> <HAnimSegment DEF='hanim_l_hindfoot’ name='l_hindfoot'/> </HAnimJoint> </HAnimJoint>

</HAnimJoint>

Motion capture

!  Electro-magnetic or by camera !  Produces the most ‘natural’ results

–  Extensively used in film for graphical extras –  Can be used on avatars very successfully to transmit user

movement to their graphical representation

!  Even more problems for self-animation –  Harder to parametrise –  Worse combining/melding problems

Mocap process

!  Calibration !  Capture !  3D Position Reconstruction !  Fitting to the Skeleton !  Post Processing

Calibration

!  Triangulate camera position and set origin

!  Wand –  Calibrate and triangulate

cameras and capture volume

!  L-frame –  Define ground plane, up

vector and origin

Multiple Hypothesis Tracking

!  For small number of markers: Size !  Occlusions are a problem !  Multiple Hypothesis Tracking

Ringer, et al., 2002

C1

C2

Fitting to the Skeleton

!  Utopian approach –  10 – 20% length

changes

!  Markers on both sides

!  Joint Displacement !  Use Rotation Angles

Only

Post Processing

!  Motion Editing – Cut, Copy, Paste

!  Motion Warping – Speed up or Slow Down – Rotate, Scale or Translate

!  Motion Signal Processing – Smoother Motions

Physically-based modelling

!  Extending robot motor control !  Forward kinematics

–  If you move joints so much –  Where does the end effector go?

!  Inverse kinematics –  If you want the end effector at x,y,z… –  How much should which joints move?

!  Computationally demanding and often not very naturalistic –  But the most flexible option –  Animation blending often added…

Ragdoll models

!  Multiple rigid body simulation – Each tied to bone in skeleton – Constraints for joint movement

!  Extended into procedural animation of whole body – Natural Motion: Euphoria

Fish example - spring-mass model

Learning to move

!  Use of AI learning algorithms – Move – Evaluate and score – Keep movements which work well

!  Karl Sims ‘Blockies’ – Early 1990s

Terzoplolous - Fish Learning

http://www.csri.utoronto.ca/~dt/

Virtual sensors

!  Local v global interaction

– Global - read from world data-structures– Local - ‘sense’ the environment

!  Global approach very common

–  In most computer games– Efficient, easy to test

Advantages of local sensing

!  Scales well–  Not affected by global size of environment

!  Agent has independence of environment

–  Up to a point!  Makes for believability

–  Agent perceives what it should–  Can’t see you round corners–  Emergent complexity

Sensing by message passing

!  Requires an architecture that distributes events as messages – Concept of locale

• What is local for local sensing?

– Semantically determined: e.g a room

!  Scenegraphs not set up for this style of message passing – Think of routing in VRML for example

Credits

!  Edward Tse, University of Calgary