Self organized character animation based on learned synergies fromfull-body motion capture data

Aee-Ni Park, Albert Mukovskiy, Lars Omlor, Martin A. Giese

Abstract— We present a learning-based method for the real-time synthesis of trajectories for character animation. Themethod is based on the approximation of complex move-ments by a small set of spatial movement primitives, orsynergies. Applying kernel methods, such learned primitivesare associated with nonlinear dynamical systems that, similarto central pattern generators in biological systems, producehighly complex trajectories in real-time by self-organization ofbehaviour. It is demonstrated that the novel method is suitablefor the simulation of highly realistic human movements ininteractive character animation. Different simple applicationsare discussed, including crowd-animation, navigation combinedwith the dynamic variation of movements styles, and theanimation of a folk dance.

I. INTRODUCTION

Generating realistic complex body movements is a coreproblem in computer graphics and robotics. This problemis particularly difficult for systems with many degrees offreedom, like for full-body animation and in humanoidrobots. In addition, many applications in robotics and com-puter games require methods that are real-time capable andpermit an online synthesis of character behaviour. In thecomputer graphics, motion capture has become the stan-dard approach for the modeling of naturalistic movements.Usually, recorded trajectories are retargeted to kinematicor physical models [1], typically requiring tedious editingof recorded data in order to adapt it to the constraints ofthe animation. Different approaches have been developed toautomatically select segments of motion capture data fromlarge data bases and to concatenate them to longer sequencesthat fulfil specific constraints that are specified by the an-imator [2], [3], [4]. Other approaches, based on physicalor dynamical models [5], [6], focused on the simulation ofscenes with many interactive agents or crowds, navigatingautonomously or showing collective behaviours, where theunderlying character models are often simplified, resultingin a manageable complexity of the system dynamics anddynamics simulation, but lacking subtle details of realistichuman body movements. Some approaches have managedto simulate highly realistic human behaviour using dynamicmodels combined with sophisticated hierarchical controlarchitectures [7]. However, the design of such systems iscomplex and the adjustment of their parameters requiresmuch expertise of the animator. In this context, it seems

All authors are with the Laboratory for Action Representationand Learning, Dept. for Cognitive Neurology, Hertie Institutefor Clinical Brain Research, Tubingen, Germany. M.A. Giese isalso with the School of Psychology, University of Bangor, UK.{aee-ni.park,albert.mukovskiy,lars.omlor}@medizin.uni-tuebingen.de,martin.giese@uni-tuebingen.de

an interesting challenge to develop simple dynamical ar-chitectures for generating complex human movements thatintegrate the information learned from motion capture data,and which avoid a detailed simulation of all details ofthe human body dynamics, however resulting in real-timeanimation with a high degree of realism.

The proposed novel method combines a compactparameterization of movements recorded by motion capturewith a dynamical systems approach for the self-organizationof interactive behaviour. Using unsupervised learning meth-ods, we approximate classes of full-body movements bycombinations of a very small number of hidden sourcesignals. The proposed method provides significantly morecompact representations than standard approaches for dimen-sion reduction, like PCA or ICA. In a second step, applyingsupervised learning, these source signals are associated withthe stable solutions of low-dimensional nonlinear dynamicalsystems whose stability properties can be systematicallydesigned. By coupling of these dynamical systems the coor-dination between the degrees of freedom within individualcharacters is maintained, and coordinated behaviour betweenmultiple characters can be simulated.

The proposed approach is biologically inspired and ex-ploits the concept of synergies, which plays a central rolein the field of motor control. Synergies specify lower-dimensional control units that typically encompass only asubset of the available degrees of freedom. A decompositionin such low-dimensional sub-units has been proposed as wayto solve the ’degrees-of-freedom problem’, which arises inthe synthesis and control of movements in effector systemswith many degrees of freedom [8], [9]. Recent studiesin motor control have successfully applied unsupervisedlearning methods to extract low-dimensional spatio-temporalcomponents from trajectories or EMG signals, which havebeen interpreted as correlates of synergies [10], [11].

However, for an accurate resynthesis of the trajectoriesstandard methods for dimension reduction, like PCA or ICA,require typically 8-12 components for an accurate approxi-mations of human full-body movements [12]. It is shown inthe following that by choice of an appropriate mixture modelmuch more compact representations of human movementscan be learned. These representations can be associated withrelatively simple dynamical systems for real-time animationwith well-defined stability properties. Like ’central patterngenerators’ in biological systems, these dynamical systemsconsist of simple basic units that are dynamically coupledand that generate complex coordinated behaviour by self-organization.

The contributions of this paper are as follows: (1) Novelmethod for the learning of highly compact models for theaccurate approximations of human full-body movements;(2) method for the mapping of the learned components(synergies) onto dynamical systems with controlled stabilityproperties that are easy to design; (3) demonstration of thepotential of this method for a few selected problems fromcharacter animation.

In the following we first describe the novel method forthe learning of trajectory models (Section II). We thendiscuss the mapping onto the stable solutions of nonlineardynamical systems (Section III), and how the model can beaugmented by a dynamic model for navigation (Section IV).Some exemplary applications of the method are presentedin Section V. Implications of the framework and futureextensions are discussed in Section VI.

II. SYNERGY-BASED TRAJECTORY MODEL

A. Motion Capture

Motion data were recorded using a VICON 612 MotionCapture System with 8 cameras and 41 reflecting markerswith a sample frequency of 120 Hz. The animations pre-sented in this paper are based on a data set of gaits thatincludes neutral and emotional straight walking (sad, happy,angry, and fearful), and neutral walking in a circle (witha rotation of 45 deg left and right per double step). Therecorded position trajectories were fitted with a hierarchicalkinematical model (skeleton) with 17 joints. The rotationsbetween adjacent segments were parameterized as quater-nions. The angles given in an axis angle representation servedas basis for the modeling by unsupervised learning.

B. Extraction of learned Synergies

Joint angle trajectories, after substracting the means, wereapproximated by a weighted mixture of source signals. Asshown elsewhere [13], an especially compact model for gaittrajectories can be obtained by fitting an anechoic mixturemodel given by the equation:

xi(t) =∑

j

wijsj (t− τij) (1)

The functions sj denote hidden source signals and the param-eters wij are the mixing weights. Opposed to common blindsource separation techniques, e.g., PCA or ICA, this mixingmodel allows for time shifts τij of the sources before thelinear superposition. Time shifts (delays), source signals andmixing weights are determined by a blind source separationalgorithm described in the following.

Classical applications in acoustics for anechoic demixingassume frequently an underdetermined mixing model of theform (1). These are models where the sources outnumber thesignals. A new algorithm for the solution of overdeterminedproblems, i.e., if the number of sources is smaller than thenumber of signals, can be derived based on the Wigner-Ville transform [14]. The Wigner-Ville spectrum (WVS) of a

random process x is defined by the partial Fourier transformof the symmetric autocorrelation function of x:

Wx(t, ω) :=∫

E

{x(t +

τ

2) x(t− τ

2)}

e−2πiωτdτ (2)

Applying this integral transform to equation (1) results in theequation

Wx(t, ω) :=∑

j

wij2 Wsj (t− τij , ω) (3)

under the assumption that the sources are statistically inde-pendent. As two dimensional representation of one dimen-sional signals, this equation is redundant and can be solvedby computing a set of projections onto lower dimensionalspaces that specify the same information as the originalproblem. In short, all parameters in the model (1) can beestimated using the following two steps:

1) Solve:

|xi(ω)|2 =X

j

|wij |2 |sj(ω)|2 (4)

where xi and sj specify the normal Fourier transformof xi and sj . This equation can be solved usingnonnegative matrix factorization (NMF) or a positiveICA algorithm to obtain the power spectra of thesources.

2) Use the results form the previous step to solve thefollowing equation numerically in order to obtain thephase spectra and delays of the sources:

|xi(ω)|2 ∂

∂ωarg {xi(ω)} =X

n

|wij |2|sj(ω)|2[ ∂

∂ωarg{si(ω)} + τij ]

(5)

Detailed comparisons for periodic and non-periodic trajec-tory data show that the model (1) results in more compacttrajectory approximations for human movement data, requir-ing less source terms than models based on standard PCAand ICA for the same level of accuracy. This is illustrated inFig. that shows the approximation quality as function of thenumber of sources. Approximation quality was defined bythe expression 1 − (‖D−F‖)

‖D‖ , where D signifies the originaldata matrix, F its approximation by the source model, wherethe norm is the Frobenius norm. For the approximation ofthe gait data basis the new method requires only three sourcesignals to achieve an approximation quality of 97 % whilewhile PCA and ICA require more than 6 sources to achievethe same level of accuracy. See [15] for further details.

This time-frequency approach allows a very accurate esti-mation of the delays in Fourier domain without the needfor discretization. A detailed analysis shows additionallythat the mixing weights vary with the emotional style ofthe movement, while source functions and delays are verysimilar between different emotions. This makes it possibleto morph between different emotions by interpolation of themixing weights.

Fig. 1. Comparison of different blind source separation algorithms.Approximation quality is shown as a function of the number of sourcesfor traditional blind source separation algorithms (PCA/ICA) and our newalgorithm.

III. DYNAMICAL SYSTEM FOR TRAJECTORY GENERATION

The previous model for the compact approximation oftrajectories based on synergies is not suitable for real-timeanimation, e.g. for computer games, since the trajectoryhas to be synthesized by superposition and delaying of thecomplete source signals. A real-time capable algorithm canbe devised by specifying a dynamical system that producesthe same trajectories as solution. We design such a dynamicalsystem, again exploiting the fact that the movement can beapproximated by a superposition of a few basic components.For this purpose, we establish a mapping between thesolutions of simple dynamical systems and the sinosoidalsource signals of the trajectory representation. The completetrajectory is then generated by a set of such simple dynamicalsystems, which are coupled to ensure temporal coordinationbetween the different sources. In an abstract sense, theresulting system is similar to a set of coupled ’centralpattern generators’ in a biological system. Such architectureshave been proposed as model for the generation of gaitsand other motor patterns [16], [17]. In the following, wefirst introduce the attractor dynamics and discuss how themapping between its solutions and the source signals of thetrajectory representation is learned. Finally, we demonstratehow dynamic couplings can be introduced that stabilize thecoordination within single characters, and which are suitablefor the simulation of coordinated behaviour of multipleavatars.

1) Attractor Dynamics: Often, behaviour can be mappedonto stable solutions of dynamical systems [18]. Variousapplications in the field of robotics (e.g. [16], [17]) suc-cessfully demonstrate this procedure. In the experimentsof this paper we simulate only periodic gaits, while ourapproach generalizes for non-periodic movements. Limitcycle oscillators seem to be a natural choice as dynamicsfor the generation of such periodic behaviours (e.g. [16],[17]). As basis elements of our architecture we use a vander Pol oscillator. This oscillator can be understood as aharmonic oscillator with an amplitude-dependent damping.

Its dynamics is given by the differential equation:

y (t) + λ(y (t)2 − k

)y (t) + ω2

0 y (t) = 0 (6)

The parameter ω0 determines the eigenfrequency of theoscillator, and the parameter k the amplitude of the stablelimit cycle. After perturbations the state will thus return tothis attractor. In addition, the structure of the dynamics doesnot fundamentally change for moderate couplings with othersystem components (structural stability). For more details see[19].

2) Mapping of phase space onto source signals: In orderto generate the source signals in real-time, we constructnonlinear mappings between the attractor solutions of thevan der Pol oscillators and the values of the source signals.These mappings are learned by Support Vector Regression(SVR) [20]. For each of the three sinosoidal source signalsof the model (1) a separate oscillator is introduced. Inorder to model the delays for joint angles (17 joints ×3 angles resulting in 51 joint angles) we introduce newauxiliary signals for the individual sources with differentdelays according to the relationship:

sij(t) = sj(t− τij) (7)

For each of these modified source signals we construct aseparate mapping from the phase space of the van der Poloscillator that corresponds to the source sj , defined by thevariable pair yj(t) = [yj , yj(t)] and the values of the sourcefunction. The mapping is defined by the nonlinear functions:

sij(t) = fij

(yj(t)

)(8)

The nonlinear functions fij are learned by SVR from Ttraining data pairs

{yj(tk), sij(tk)

}1≤k≤T

that are derivedby sampling one cycle of the stationary solution of theoscillator and the modified source signal equidistantly overtime. After the functions fij have been learned, the dynamicscorresponds to three limit cycle oscillators with nonlinearinstantaneous observers that map the state of each oscillatoronto the corresponding set of delayed source signals. Thesesignals then are linearly combined according to (9). The com-plete reconstruction of the joint angle trajectories requiresthe addition of the average joint angles mi, which havesubtracted from the data before the blind source separation(Figure 2):

xi(t) = mi +∑

j

wij sij(t) (9)

3) Dynamic couplings: In order to stabilize the timing-relationship between the different sources (’synergies’) weintroduced dynamic couplings between the three oscillatorsthat drive the source signals of the same avatar. It has beenshown applying contraction theory that stable behaviourof such oscillator networks can be accomplished byintroduction velocity couplings [21].

Fig. 2. Illustration of the dynamics for real-time animation. The phase spaceof the limit cycle oscillator is mapped onto the time-shifted source signalsusing Support Vector Regression. Joint angles are synthesized by combiningthe signals linearly according to the learned mixture model (1). A kinematicmodel converts the joint angles into 3-D positions for animation.

The couplings within the same avatar were given in the form(cf. Figure 3):

y1 + λ(y21 − k

)y1 + ω2

0 y1 =α (y2 − y1) + α (y3 − y1)

y2 + λ(y22 − k

)y2 + ω2

0 y2 =α (y1 − y2) + α (y3 − y2)

y3 + λ(y23 − k

)y3 + ω2

0 y3 =α (y1 − y3) + α (y2 − y3)

(10)

Such dynamic couplings can also be exploited for simulatingcoordinated behaviour of multiple characters and crowds, forexample, to enforce that multiple avatars walk in synchrony(e.g. soldiers in lock-step). To implement such couplings weonly connected the oscillators assigned to the source withthe lowest frequencies (oc1). By introducing unidirectionalcouplings it is possible to make multiple characters followingone, who acts as a leader [21].

4) Dynamic variation of movement style: The proposedmodel for the real-time generation of trajectories permitsstyle morphing in a straight-forward way. To interpolatebetween two movement styles (a) and (b), e.g. neutral andemotional walking, the mixing weights and the mean jointangles are linearly interpolated. For the weights wij thisresults in the equation:

wij (t) = λ (t)waij + (1− λ (t))wb

ij (11)

The time-dependent morphing parameter λ(t) specifies themovement style. Additionally, the gait speed is adjusted byinterpolating the eigenfrequencies of the oscillators:

ω0(t) = λ (t) ωa0 + (1− λ (t))ωb

0 (12)

The same type of morphing was also applied to implementdirection changes of the avatars for navigation, morphing

Fig. 3. Coupling of multiple avatars, each of them comprising three coupledoscillators.

between straight and curved gait steps. The change of themorphing parameter can be made dependent on the be-haviour of other avatars in the scene, e.g. to influence theemotional style dependent on the distance d of the avatarfrom another ’dangerous’ colleague, introducing a distance-dependent morphing weight λ (t) = g (d (t)). Similarly, theeigenfrequency parameter ω0 can be made dependent on thedistance from other agents. In this way, the walking speedof the characters can be made dependent on the behaviourof the others.

IV. NAVIGATION

For the simulation of realistic interactive character be-haviour, navigation and obstacle avoidance have to be in-cluded in the system for real-time animation. From thegenerated articulated motion of the avatars, first, the trans-latory motion and the rotation of the hip in the horizontalplane is computed by enforcing the floor contact constraintsof the feet. In this way, we generate the propagation andheading direction of the avatars indirectly from the jointangle movements. By interpolating between straight andcurved walking we were able to simulate walking withdifferent curved walking paths. The addition of a navigationdynamic makes it possible to simulate avatars that movetowards specific targets in space, or to avoid obstacles andother avatars.

A. Computation of translation and heading

The pelvis forms the root of the kinematic chain of ouravatar model. The core of the algorithm for computingthe avatar’s propagation is to adjust the horizontal pelvistranslation and rotation in order to fulfil the constraint thatthe feet that make contact to the ground do not translate. Forthis purpose, we first detect whether the feet make groundcontact using a simple threshold criterion for the verticalposition of the foot centers. This criterion works well whenthe original motion capture data did not contain foot slipping.We add a differential translation and horizontal rotation to thepelvis coordinate system in order to minimize foot slipping.

In addition, the vertical position of the pelvis is adjusted byassuming that the foot forms the lowest point of the figure.

B. Navigation dynamics

We use a simplified version of a dynamic navigation modelwhich has been applied successfully in robotics before [22],[23]. The temporal change of the heading direction ϕi ofavatar i is given by the differential equation:

dϕi/dt = hgoal(ϕi, pi, pgoal

i

)+∑

j

havoid(ϕi, pi, pj

)+∑

j

hpcoll(ϕi, ϕj , pi, pj

) (13)

The right-hand side of this equation has three differentcomponents, where pi denotes the position of character i.The first term

hgoal(ϕi, pi, pgoal

i

)=

sin

(arctan

(pi − pgoali )2

(pi − pgoali )1

− ϕi

)(14)

models a tendency to move in the direction of the goal, wherethe goal of avatar i is specified by the 2D position vectorpgoal

i .The second term implements obstacle avoidance, where

the obstacles are defined by the other avatars. This term isgiven by:

havoid(ϕi, pi, pj

)=

sin (∆ϕij) · exp

(−

∆ϕ2ij

2σ2ϕ

)· exp

(−

d2ij

2σ2d

)(15)

with ∆ϕij = ϕi − arctan((pi − pj)2/(pi − pj)1

)and

dij = |pi − pj |.Much more realistic collision avoidance is accomplished

by inclusion of a third term in the dynamics that controlsthe heading direction. This term has the same form as theprevious term.

hpcoll(ϕi, ϕj , pi, pj

)= sin (∆ϕij) ·

exp

(−

(∆ϕpcij )2

2σ2ϕ

)· exp

(−(dpc

ij

)22σ2

d

).

(16)

However, dpcij and ∆ϕpc

ij are computed like dij and ∆ϕij

replacing the point pj by the predicted collision point of theavatar i with avatar j.

The walking direction of the characters is changed byinterpolation between straight walking and walking alongcurved paths to the left for dϕi > 0, and to the right fordϕi ≤ 0 using (11). The morph parameter λ(t) was takenproportional to |dϕi|, normalizing in a way that ensuresλ = 1 for the maximum possible value of this derivative.The heading direction ϕi (t) generated by the navigationdynamics was low-pass filtered with a time constant equal toone step cycle to improve the smoothness of the navigationbehaviour.

V. RESULTS

The proposed technique for motion capture-based self-organized simulation of character behaviour has a broadapplication spectrum, which can only be partially be exploredin this paper. In the following, we present a number of exam-ple applications that highlight possibilities of the approach.

A. Approximation quality

In general, the blind source separation method discussedin Section II provides accurate approximations of the jointtrajectories by using less components than other standardmethods for dimension reduction. Beyond the quantitativeanalysis in Fig. 1, the influence of approximation quality onanimations is illustrated in by an attached Demo1.

As further evaluation of the method, we compared thereconstruction errors of the model for different conditions,including on-line and off-line models. For this purpose, weextracted 3 shift invariant source signals from the 7 motion-captured emotional walking gaits (including left and rightturning walks). We compared the reconstructed angles tra-jectories of neutral straight walking of synthesized gaits withthe original motion-captured trajectories of the same style(corresponding to avatar 1). The second case was the off-linegeneration of the trajectory by the blind source separationmodel (1), used to animate avatar 2. The movements of avatar3 were generated by the simplest possible on-line model,mapping the phase spaces of 3 coupled oscillators by SupportVector Regression onto the source signals. The propagationspeed of this avatar deviates slightly (a few percent) fromthe avatar 1 due to approximation errors.

Finally, we created a more complex online situation whereanother avatar (avatar 4) follows avatar 1 implementing adistance-to-frequency coupling in order to keep the dis-tance between the avatars constant (cf. Section III-.4). Thedistance-frequency coupling was implemented by a sig-moidal function:

λ(t) = g (d(t)) =1

1 + exp(−γd(t))(17)

with the positive constant γ. The parameters ωb0 and ωb

0 in(12) were appropriately adjusted.

The normalized errors (unexplained variance) betweenthe trajectories in joint angle space of avatars 2, 3 and 4compared with avatar 1 (original trajectory) were: 6.89%(2-1), 8.62% (3-1), and 7.51% (4-1). This implies that themajor error is introduced by the source approximation, whilethe support vector regression has a much smaller influence.

B. Coordination of a crowd

A first example is illustrated in Figure 4. A crowd ofcharacters is first locomoting individually without fixed phaserelationship. An instruction to move synchronously (in lock-step) is introduced, modeled by introducing couplings be-tween the oscillator triples of the individual avatars, taking

1www.uni-tuebingen.de/uni/knv/arl/avi/compareN.aviwww.uni-tuebingen.de/uni/knv/arl/avi/compareH.avi

Fig. 4. Synchronization of the gaits within a crowd. (a) Avatars start withself-paced walks that are out of phase. After a transitory period (b), thegaits of the characters become completely synchronized (c).

one oscillator as a leader (Section 3). For appropriate cou-pling, the transition between uncoordinated and coordinatedcrowd behaviour is quite short (less than three steps) andlooks quite natural. After the transition, characters adaptwalking behaviour that looks highly realistic [Demo2].

The same example is useful to illustrate the benefit of morecompact generative trajectory models for on-line animation.We compared the synchronisation behaviour of two avatars(see [Demo3]). One of them (avatar 1) was driven by threeoscillators corresponding to the source signals extracted bythe discussed blind source separation algorithm. The otheravatar (avatar 2) was driven by 5 oscillators that weremapped onto PCA components of the trajectories, in thestationary state resulting in a similar approximation quality.Both avatars were coupled to a third avatar (leader) with thesame coupling strength. Starting the avatars 1 and 2 withequal initial phases which differ from the phase of avatar 3,it shows that the model with three oscillators synchronizesfaster and produces more natural-looking behaviour.

The same example also nicely demonstrates that themodel, even though it has been trained in the attractor ofthe dynamics, generalizes in a meaningful way to transientstates. This is illustrated by the phase diagram in Fig. 5 thatillustrates the trajectories of the first leading oscillators ofthe two avatars. First, this figure shows that the oscillatorsduring the synchronization period are not near the attractor,requiring the system to generalize to trajectories that were notused during the raining of the system. Second, it is obviousthat the system with less oscillators (red trajectory) returnsfaster to the attractor than the system with 5 oscillators (greentrajectory). More detailed investigations testing dependenceon coupling strength and coupling structure are underway.Since the linear weights of the PCA components are notsparse the mixing of the many oscillator inputs results injerky motion in the starting phase of the synchronizationperiod.

2www.uni-tuebingen.de/uni/knv/arl/avi/synchronization.avi3www.uni-tuebingen.de/uni/knv/arl/avi/transient.avi

Fig. 5. Phase plot of the first (principal) oscillators of two avatars that arecoupled to a leader. The diagram shows the state space of the oscillator inthe [y(t), y(t)] plane. The green line corresponds to the avatar driven by5 PCA components, and the red line to the avatar animated with 3 sourcesextracted with our algorithm. The coupling is switched on after short startingperiod.

C. Navigation and style change

The second example [Demo4] is illustrated in Figure 6. Agroup of avatars that meets in the center of the scene changestheir affect upon the contact with the other characters. Thisbehaviour was implemented by making the affect of eachavatar dependent on the distance from the others. In addition,the avatars avoid each other, due to the navigation dynamicsdescribed in section IV. In this simulation, navigation andchanges of emotional styles were combined, based on onlythree prototypical gaits: neutral walking with rotation rightor left, and emotional walking. In order to produce theemotional gait for blending with left and right neutral paces,we first created an intermediate balanced mixture by inter-polating the mixing weights. Corresponding with section IV,gait morphing was based on a piece-wise linear interpolationdependent on the sign of the change of the heading directiondϕ/dt.

D. Animation of a ’folk dance’

The next example is a self-organized scenario, where eightpairs of avatars execute a type of ’folk dance’ that requiresthem to walk in synchrony to the music along a straight line,forming a corridor. Once the couples have reached the endof the corridor, they are required to walk quickly to the otherend and re-enter the corridor from the other side. At the pointof reentrance the individual avatars need to synchronize withtheir partner and the other avatars within the corridor. In spiteof this relatively complex scenario quite natural behaviour ofthe whole group of 16 avatars can be self-organized [Demo5].

To simulate this complex behaviour, we divided this sce-nario into four main sectors that are described separately inthe following (see Figure 7):

1) At the entrance of the corridor the characters gatherand wait for the corresponding partner to move syn-chronously. To simulate the synchronized walking of

4www.uni-tuebingen.de/uni/knv/arl/avi/avoidance.avi5www.uni-tuebingen.de/uni/knv/arl/avi/dance.wmv

Fig. 6. Avoidance behaviour and change of emotional style. a) At themeeting point their emotional styles change from sad to happy. In addition,the characters avoid each other. b) Avatars heading towards their goals (redcircles) with a happy emotion.

all couples within the corridor the corresponding oscil-lators are coupled similar to equation (10), introducingcouplings not only between the avatars of one couplebut also between the subsequent couples within thecorridor.

2) At the end of the corridor reaching the second zone thetwo avatars of each couple separate and the couplingbetween their oscillators is removed. Their movementsfrom this point are asynchroneous and determined bythe the navigation dynamics according to Section IV.In addition, within this zone the emotional walkingstyle of the characters changes from happy to neutral.The curved walking paths are generated by definingappropriate goal points.

3) Along the straight paths outside the corridor the avatarsaccelerate to catch up with their partner at the begin-ning of the corridor in time, simulated by a temporaryincrease of the eigenfrequency of the correspondingoscillators.

4) In the last zone the characters decelerate, modeled by

Fig. 7. Simulation of a ’folk dance’ behaviour is organized within fourzones: 1) Avatars within the corridor move in synchrony, simulated bycoupling the corresponding oscillators. 2) Leaving the corridor the avatarsof the individual dancing couple become separated and start to move alongcurved paths. Their emotional style changes from happy to neutral. 3)Avatars walk asynchronously and ’hurry up’ to reach the entrance of thecorridor, simulated by increasing the eigenfrequency of the oscillators. 4)When the avatars approach the entrance of the corridor their walking speeddecreases, and they need to get in synchrony again with the appropriateleg. Both can be simulated by appropriate adjustment of the oscillatorfrequencies.

decreasing the frequency of the oscillators. A difficultproblem is the re-synchronization with the correct footat the entrance of the corridor. This is accomplished byslightly adjusting the oscillator frequencies to ensurere-synchronization with the appropriate leg. Again, thecurved paths are generated by defining appropriate goalpoint exploiting the navigation dynamics (13).

VI. CONCLUSION AND FUTURE WORK

We presented a new framework that links the synthesisof realistic human movements based on motion capturedata with the self-organization of behaviour using nonlineardynamical systems. The proposed method exploits biolog-ically inspired concepts and is based on the learning ofhighly compact models for human movement trajectories bya superposition of learned ’synergies’, which are controlledby nonlinear attractors dynamics. By introducing of appropri-ate dynamic couplings complex realistic looking behaviour,reproducing the fine structure of human movements, couldbe self-organized, even for complex scenarios. Presently,we extend this approach also for non-periodic movements.In addition, we work on scenarios with contact betweenmultiple avatars inducing additional kinematic constraints.

Future work will try to exploit more the synergy concept,learning sparse components that encompass only limitedsets of degrees of freedom. This will potentially result inmore flexible control of motion styles. In addition, suchcomponents offer the possibility to make individual synergiesreactive and dependent on external constraints, potentiallyproviding a basis for a much more fine-grained adaptationof the generated behaviour to external constraints. In thiscontext, possible applications in robotics will be considered.

VII. ACKNOWLEDGMENTS

Supported by DFG Forschergruppe ’Perceptual Graphics’and EC FP7 project COBOL. We thank C. Roether forhelp with the acquisition of the motion capture data.Additional support was provided by the Hermann undLilly-Schilling-Stiftung. We are grateful to T. Flash and J.J.Slotine for many interesting discussions and W. Strasser forhis support.

REFERENCES

[1] M. Gleicher, ”Retargetting Motion to New Characters”, Proc. ACMSIGGRAPH, Int. Conf. on Comp. Graph. and Interactive Techniques,1998, pp. 33-42.

[2] M. Gleicher, H.J. Shin, L. Kovar and A. Jepsen, ”Snap-togethermotion: Assembling run-time animation”, ACM Trans. on Graph.SIGGRAPH, 2003, pp. 702-702.

[3] O. Arikan, D.A. Forsyth and J.F. O’Brien, ”Motion synthesis fromannotations”, ACM Trans. on Graph. SIGGRAPH, 2003, pp. 402-408.

[4] J. Chai and J.K. Hodgins, ”Performance Animation from Low-dimensional Control Signals”, ACM Trans. on Graph. SIGGRAPH,2003, pp. 702-702.

[5] R. Grzeszczuk, D. Terzopoulos and G. Hinton, ”NeuroAnimator: Fastneural network emulation and control of physics based models”,Proc. ACM SIGGRAPH, Int. Conf. on Comp. Graph. and InteractiveTechniques, 1998, pp. 9-20.

[6] W. Shao and D. Terzopoulos, ”Autonomous pedestrians”, Proc. ACMSIGGRAPH, 2005, pp. 19-28.

[7] J.K. Hodgins, W.L. Wooten, D.C. Brogan and J.F. O’Brien, ”Animat-ing human athletics”, Proc. ACM SIGGRAPH, 1995, pp. 71-78.

[8] N.A. Bernstein, The coordination and regulation of movements, Perg-amon Press, Oxford, 1967.

[9] T. Flash and B. Hochner, Motor primitives in vertebrates and inverte-brates, Curr. Opin. Neurobiol., vol.15, no.6, 2005, pp. 660-666.

[10] A. d’Avella and E. Bizzi, Shared and specific Muscle synergies inneural motor behaviours, Proc. Natl Acad Sci USA, vol.102, no.8 2005,pp. 3076-3081.

[11] Y. Ivanenko, R. Poppele and F. Lacquaniti, Five basic muscle activa-tion patterns account for muscle activity during human locomotion,J Physiolol, vol.556, 2004, pp. 267-282.

[12] A. Safanova and J.K. Hodgins and N.S. Pollard, ”Synthesizing phys-ically realistic human motion in low-dimensional, behaviour-specificspaces”, ACM Trans. on Graphics, Proc. SIGGRAPH, 2004, pp. 514-521.

[13] L. Omlor and M. Giese, ”Extraction of spatio-temporal primitives ofemotional body expressions”, Neurocomputing, 2007, pp. 10-12.

[14] G. Matzand and F. Hlawatsch, Wigner distributions (nearly) every-where: Timefrequency analysis of signals, systems, random processes,signal spaces, and frames, Signal Processing, vol.83, no.7, 2003, pp.1355-1378.

[15] L. Omlor and M.A. Giese, ”Blind source separation for over-determined delayed mixtures”, Advances in Neural Information Pro-cessing Systems, 2006, pp. 1049-1056.

[16] A. Ijspeert, J. Nakanishi and S. Schaal, ”Learning Attractor Land-scapes for Learning Motor Primitives”, NIPS, 2002, pp. 1547-1554

[17] S. Schaal, A. Ijspeert and A. Billard, Computational approaches tomotor learning by imitation, Philos Trans R Soc Lond B Biol Sci.,vol.358, no.1431, 2003, pp. 537-547.

[18] J.A.S. Kelso, Dynamic Patterns: The Self-Organization of Brain andBehaviour, MIT Press, Cambridge, 1995.

[19] A.A. Andronov, A.A. Vitt and S.E. Khaikin, Theory of oscillators,Dover Publ. Inc., New York, 1987.

[20] V. Vapnik, Statistical Learning Theory, Wiley-Interscience, New York,1998.

[21] W. Wang and J.J.E. Slotine, On Partial Contraction Analysis forCoupled Nonlinear Oscillators, Biological Cybernetics, vol.92, no.1,2005, pp. 38-53.

[22] G. Schoner, M. Dose and C. Engels, Dynamics of behaviour: Theoryand applications for autonomous robot architectures, Robotics andAutonomous Systems, vol.16, no.2-4, 1995, pp. 213-245.

[23] W.H. Warren, The Dynamics of Perception and Action, PsychologicalRev., vol.113, no.2, 2006, pp. 213-245.