style-based robotic motion in contemporary dance …style-based robotic motion in contemporary dance...

Style-based Robotic Motion in ContemporaryDance Performance

Amy LaViers, Lori Teague, and Magnus Egerstedt

Department of Information and Systems Engineering,University of Virginia

Charlottesville, VA 22904, [email protected]

Department of Dance,Emory University

Atlanta, GA 30322, [email protected]

School of Electrical and Computer Engineering,Georgia Institute of Technology

Atlanta, GA 30332, [email protected]

Abstract. This chapter reviews a framework for generating robotic mo-tion and describes its application in a performance at Georgia Instituteof Technology. In particular, the movement model stems from the viewthat cannons of basic warm-up exercises in formalized movement genres,such as classical ballet, seed more complex phrases. This model employs aseparation of basic movement ordering and execution. Basic movementsare sequenced, and their individual execution modulated via a notion ofquality from dance theory. The sequencing framework is then employed inperformance both on a humanoid robot and real dancers. Results from ahuman study, questionnaires given to audience members after the show,are also presented. The generation framework also lends itself to move-ment interpretation and that extension will be briefly presented as well.

Keywords: Robotic motion generation, human-centered design

The topic of this chapter might, most simply, be considered “dancing robots.”Indeed, to provide an artistic outlet for the concepts presented in [21,19,20,22,23],an original work of contemporary dance was choreographed and performed April6, 2013 and April 13, 2013 in Clough Undergraduate Learning Commons at Geor-gia Institute of Technology. The four professional dancers and NAO Aldebaranhumanoid robot pictured in Fig. 1 were members of the cast. What follows isa discussion of how principles from technical publications were included in thechoreography of a contemporary modern dance showing entitled Automaton.

Thus, a method for taking human movement and animating a version of iton a robot will be provided. This is accomplished not from a direct joint angle

2 Control Applications in the Arts

Fig. 1. The four dancers in Automaton along with the humanoid robot used in theshow. Photo by Christian Moreno.

mapping from one highly articulated skeleton (the human) to a deficient one(the robot). Instead, the method provides a way to extract key stylistic featuresof human movement (recovered via motion capture) and use them to instantiatea general movement model. With this approach, novel sequences, which are inthe same style, can be produced rather than simple imitation of the originalsequence. However, such a narrative, casting the work as simply a method formimicry, short changes the diverse applications this work can have. The aim isto use engineering and art to tease out an understanding of human behavior,which is key to applications in robotics, human-centered technology, and theperforming arts.

1 The Robot as an Onstage Character

The robot used in Automaton is immediately engaging to audiences. The roboticfigure calls to mind experiences of humanoids in pop culture, film, and fantasy.This presents unique challenges and opportunities from a choreographic perspec-tive. Because the robot conjures up the audience member’s impressions aboutrobots, one of the opportunities is to challenge these preconceived ideas. Peoplecan feel quite differently about a piece of technology that is human-shaped (forexample, see Fig. 2). While few people would personify their ice maker or iPhone,visitors to our lab immediately ask what the robot’s name is (and sometimesgive it one themselves).

Thus, one of the choreographic goals of Automaton is to blur this boundarybetween human-shaped and non-human-shaped technology as well as the bound-ary between humans and programmed devices. At the interface where humans

Style-based Robotic Motion 3

Fig. 2. Two dancers depict the robot being aggressively confronted. The extent towhich this image conjures up empathy for a plastic container for motors, circuit boards,and a processor is surprising until its human shape is taken into account. Photo byChristian Moreno.

interact with technology, their own movements are induced by the automatedaspects associated with a given piece of technology [13]. In this sense, humansoften mimic robotic qualities in their movements. The confluence of the discretedescription of movement presented in following sections, a robot to implementit on, and trained dancers who could also execute that description with a highlevel of accuracy made Automaton a perfect setting to explore the interaction ofnatural human movement with the movement induced by a programmed device.

In the third movement of the piece, two dancers perform with the NAOhumanoid robot. This trio explores the idea that there is some level of expectedbehavior and reaction from any being or device with which a person interacts. Forexample, the dancers engage in a game of patty cake. This is a set of movements,a series of alternating hand claps between two people, that are essentially pre-programmed in many children (at least in Western culture). In the trio onedancer breaks the pattern of the movement – in a sense malfunctioning – andcauses his partner to have a moment of extreme confusion. The confused partnerthen turns to the robot and makes it his new patty cake partner. But the robotalso breaks the traditional pattern of patty cake. Here both the human andthe robot “malfunctioned” – seemingly on purpose – or started playing a newgame, causing the human partner, in a twist of irony, to experience a situationhe seemed not to know how to navigate, a plight that typically affects roboticsystems – particularly in dynamic environments.


This vignette displays a reversal of roles: instead of humans evolving underunknown patterns where next movements are not predictable, they engaged ina child’s game with preset movements. Because the robot can easily mimic suchpreprogrammed movements, it can play along too. But what happens to a hu-man when the rules of this game are slightly different? In this case (and in manyothers), the human malfunctions just like a robot presented with an unpredictedsituation. This is analogous to an autonomous car that was designed for oper-ation in the U.S., where cars drive on the right side of the road, that has justbeen dropped in the U.K., where cars drive on the left side of the road – it wouldcrash instantly. This metaphor mirrors aspects of human behavior as well.

Beyond the theatrical episode described here, how can we make a robot moveautomatically in a given style of movement? This is the subject of this chapter,which will proceed as follows: Sec. 2 presents a discussion of movement andmovement style from the point of view of a certified Laban movement analyst(CMA), Sec. 3 gives an overview of how stylistic features might be extractedfrom human movement, Sec. 4 describes how we build style “knobs” in order tocapture key aspects of movement style from real data, Sec. 5 describes how thismethod for stylistic movement generation was applied during the performanceof Automaton, Sec. 6 presents a human study conducted at the showings ofAutomaton, and finally conclusions are presented in Sec. 7.

2 A Description of Movement Style

Style becomes recognizable through a gestalt of integrated elements. An aspectof style is static, determined by a core group of features that repeat, like thecompositional pattering of a quilt or a dialect. However, we are continually mov-ing from this recognizable source, while we invite new combinations into ourrepertoire. How does our core influence our creativity, and vice versa? We areinstinctual. We adapt. We invent. We control. We play. The possibilities thatunfold expand our capacity to communicate, inviting us to be bold, focused,flexible, whimsical, fierce, generous, etc... The psyche does have some control,and it gives us insight into aspects of our culture and our identity.

At our core, in our most natural state, movement is authentic–expressing therelationship between the kinesthetic, creative and psychological dimensions ofthe body. Our movement vocabulary emerges, including the attitudes and emo-tions that are embedded in our psyche. There is a continuous interplay betweenthe space inside and outside of the body that embraces our survival mechanismsand of course, our need to connect. As we are becoming fully embodied humanbeings, our personal movement can already be noted in our unique walk, the toneof our personality, conversational gestures, the body’s carriage, and the space weinhabit. This is what defines our personal movement style. Whether we investin this natural state, or transform, control, and manipulate the body’s languageto express something specific, we are still synchronizing, sometimes balancing,the layers of movement itself.


More broadly, if a viewer recognizes the style of dance as a waltz, it is notjust because it is in 3

4 time. It is a particular dance within a prescribed genre–composed of its vocabulary, qualitative use of energy, composition in space, andpredominant shapes. A waltz maintains a formal carriage in the upper torso inpartnership with another dancer; the whole body sweeps and arcs through thespace, creating under curves and sustaining peaks with a patterned timing ofsteps. And break dancing moves are architectural puzzles, like a Rubik’s cube.The improvisational structure allows the mover to play with resiliency, momen-tum and balance. Body surfaces leverage and spin off the floor; weight is shiftedquickly from the hands to the feet, to the head, inverting the center of gravityand discovering tripods. Personal movement styles can potentially live inside adance genre or speak loudly on their own.

Choreographers and movement artists develop distinctive movement vocabu-laries and sequences of movements that become phrase material or dances. Theirmovement language is an autobiographical manifestation of skeletal and muscu-lar range, creativity, cultural values, and geographic resources. Ultimately theway in which any mover webs the layers, and invests in each of them, is whattheir style becomes. The dance is both a patterning of movement and a represen-tation of aesthetic choices. A choreographer’s style will become more imprintedas they discover new relationships and shape the nexus.

The movement style of Martha Graham, a modern dance pioneer, is iconic.Her shape, standing on one leg with the other leg arcing out into space, footflexed, is supported by a contraction in the torso. Her arms balance with seri-ousness and emotion. The shape of her body, one arm reaching up in the verticaland one reaching forward in the saggital, as her back pulls backwards, molds thespace. Graham’s direct, bold, architectural movement vocabulary defined herwomanhood in the 1940s. Her choreography swept powerfully left and right,pierced the diagonal, and dropped to the earth, like a shaft of light.

The description in the previous paragraph can be credited to the work ofmovement theorist Rudoph Laban who provided a framework for movement anal-ysis. When observing movement, Laban defined four areas: Body, Effort, Shape,and Space. Each area contains numerous layers. For example, when observingmovement with a body lens, you can see what part of the body is initiating;how the movement is sequencing throughout the body; how the body developsconnections through the spine and out to the limbs; what part of the body isgrounded to allow for mobility; how breath supports movement range; and howthe senses and intellect inform the qualitative nature of the movement.

Flow, weight, space and time are the motion factors of Effort, Laban’s termto describe our inner attitude towards movement. These qualitative choices,often in combination with each other, are an essential component of a personalmovement style. While conscious of flow, a dancer’s movement fluctuates betweenbound and free. The dancer can use their center of gravity or levity to produce aquality of lightness, limpness, force, or resiliency. The dancer can approach theenvironment with directness or interact with its volume. And lastly, the timing


of the dancer’s movement can be impulsive, impactful, sustained, or increase anddecrease in response to body rhythms or the mover’s surroundings.

3 Interpreting Human Motion for Robotics

How can we organize the multidimensional life of movement, including its emo-tional and cultural context? This section reviews the work of others and outlinesour approach for directly interpreting measurement of human movement, whichis not the subject of the remainder of the chapter but is an important contextwithin which to consider it.

In [3] Bregler coins the term movemes – in analogy to the linguistic conceptof a phoneme – which is cited in many following publications, often offering analternate technical definition to this same general concept. A moveme is a motionprimitive, and some combination of several movemes can desribe real humanmovement. The term may be viewed as synonomous with the goal to developan appropriate mathematical definition that can segment these primitives fromreal data – in either two (i.e., video) or three (i.e., motion capture) dimensions.The concept comes up in many arenas: general pattern recognition [6,35,8,9,33],robotics [15,29,7,28,18], and gait analysis [24,34]. The basic search that pervadeseven between these academic arenas is for atomic nuggets of movements whichcan recombine, in the same way phonemes are the sounds that create words, tosuccessfully recreate and form novel full-fledged movement sequences.

A large body of research has also aimed at understanding motion patterns forapplication to vision and animation with the hopes of implementing character toon screen avatars. Some work [2,31,27] has a more statistical bent while others[1,17,14,11,26,16] aim more directly at generation of novel motion sequencesaccording to various constraints and character-driven rules. Similarly, attemptsat “stylizing” robotic motion such as [10], aim to facilitate better human-robotinteraction through a similar, character-driven philosophy.

The remainder of this section describes our approach for interpreting high-level aspects of human movement – such as style. Subsequent describe the im-plementation of the extractions outlined here on a robot – particularly as inAutomaton. Imagine the scenario depicted in Fig. 3. On the left side, envisiona robot endowed with style “knobs.” As the values set by these parameters arechanged, the style of the robot’s motion changes accordingly; this motion maybe viewed as a trajectory of joint angles over time. On the right side, considerhuman movement recorded by motion capture; also a quantity in the form ofjoint angles over time. Optimizing a measure of similarity between these twosignals – which can be improved by adjusting the setting of the style “knobs” –would provide the setting for the style parameters that best describes the humanmotion. In this way, a measure of style can be achieved.

For example, to solve for parameters which best describe the shape of atrajectory between given start and end points over a specified interval, consider


Fig. 3. The figure shows a conceptual overview of how the stylistic parameters mightbe used to measure high level aspects of human motion. On the left imagine a signal(perhaps implementable on a robot) that we are able to generate by varying the valueof different stylistic parameters; on the right real data of human movement, such as amotion capture recording. The circle indicates we are comparing these two signals inorder to measure an aspect of the real data.

the optimal control problem

minuJu =

∫ T

0

F (x, σ, π)dt+ ψ(x(T ), σ(T ), π) (1)

s.t.

{x = f(x, u) x(0) = x0

y = h(x)(2)

where x ∈ Rn is the state, u ∈ Rm is the input, y ∈ Rl is the output, σ isa reference, and π is a vector of so-called “stylistic” parameters (which will beenumerated in subsequent sections).

The maximum principle states that the optimizer, u∗, can be expressed as afunction of x, ξ, σ, and π, where ξ is the costate satisfying

ξ = −∂F∂x

T

− ∂f

∂x

T

ξ

ξ(T ) =∂ψ

∂x(x(T ), σ(T ), π). (3)

Plugging in the optimal u∗ into the equations for x and ξ gives the expressionfor the Hamiltonian dynamics:

x = fx(x, u∗(x, ξ, σ, π)) (4)

ξ = fξ(ξ, u∗(x, ξ, σ, π)), (5)


which we denote with

x = fx(x, ξ, σ, π) (6)

ξ = fξ(x, ξ, σ, π). (7)

Now define a second cost function which we wish to minimize with respect tothe weighting parameters π under the constraints imposed by the problem justoutlined. That is,

minπJπ =

∫ T

0

L(x, ρ)dt+ Ψ(x(T ), ρ(T )) (8)

s.t.

{x = fx(x, ξ, σ, π) x(0) = x0

ξ = fξ(x, ξ, σ, π) ξ(T ) = ∂ψ∂x (x(T ), σ(T ), π)

(9)

where ρ is the empirical data we wish to mimic and classify.

The analystical solutions to this problem, and a similar segmentation prob-lem, are presented in [23]1 and give the optimal value of π needed to best recre-ate ρ according to an L2-norm. The next section (specifically Subsection 4.3)describes our formulation for what the stylistic parameters in π should be inorder to capture a relevant aspect of style.

4 Generating Stylized Motion

This section2 presents a quantitative definition of style – a choice for the style“knobs” proposed in the previous section. The definition is comprised of threedistinct mathematical objects, or stylistic parameters, which drive a two partgeneration scheme. The first, an automaton, lists basic, primary movements,which are represented abstractly by the transitions of the automaton; the statesin the automaton are preselected body poses. These primary movements arethen recombined and sequenced according to knobs on a supervisory controller,the second stylistic parameter, which are scripted either as sets or LTL specifica-tions. The primary movements are, in parallel, modulated via an optimal controlproblem where weights in a cost function, the third stylistic parameter, deter-mine the quality of each movement; this scheme generates specific trajectorieswhich are not explicitly part of the automaton. The overall scheme is illustratedin Fig. 4.

The definition of “style” of movement presented in this section is precise suchthat it can be implemented on robots, which is what Sec. 5 will describe – inthe setting of a contemporary dance performance.

1 The dissertation [23] also provides an implementation of these solutions to extractthe quality from real motion capture data.

2 Subsections 4.1, 4.2, and 4.3 previously appeared in [21,22].


Fig. 4. The figure shows a conceptual overview of how the three stylistic parametersand the two-part movement scheme fit together.

4.1 A Quantitative Movement Model

We define basic movements by enumerating them in the first element of ourmodel: a discrete event description of primary motions. These pre-defined mo-tions are initially abstracted as events in a transition system – defined only bytheir starting and ending poses. Later, we will use these motions as referencesignals for an optimal control problem that can modulate exactly how each mo-tion is executed. Thus, from the simple system first enumerated, we can producemuch more complex movement phrases that are structured combinations (seeSec. 4.2) and modulations (see Sec. 4.3) of these primary movements.

This stance is directly inspired by the method of training used in classicalballet, the barre, in which dancers repeatedly perform small movements over andover. This canonical combination is cerntral to the practice of ballet and serves asa warm-up for dancers. These basic movement snippets are then recombined laterin class and during performances to create more complex, full-fledged movementsand movement phrases – thus, we think of them as primary. For example, the skillrequired to execute a grand jete, a majestic split-legged leap common in balletchoreography, is honed through grand battements, straight-legged high kicks, atthe barre. Performing repeated kicks forward, backward, and to the side with oneleg, while the other, standing leg provides stability allows the dancer to developthese muscles. Then, away from the barre, the dancer kicks one leg forward andone leg backward simultaneously to perform the grand jete, a combination ofthese primary motions that invokes both practiced muscle groups at the sametime.

Thus, a rational choice for the primary motions, and state transitions, ina model describing free flowing motion in the style of classical ballet are themovements from the barre exercises as described in [36]. Additionally, we dis-


tinguish two transitions for each movement listed in the table using a subscriptto indicate an in and out variant. The variants stem from the fact that eachprimary movement should be unique. Thus, a movement is either going towardsor away from each of the two states (body positions) it connects, and the nextstate depends on which of these is the case.

Fig. 5. A transition system which models the working leg (right leg) of a dancer duringa ballet barre exercise. This figure illustrates the basic element of our movement model:a discrete event description of primary motions.

In order to demonstrate how a sample path through the system works, con-sider, for example, a developpe; this movement is found both in barre exercisesand more complex ballet movement phrases. A developpe is the action when theworking leg’s foot is moved from the knee and then extends from the body. Atthe barre this motion is practiced by incremental flexion of the leg; beginning flaton the floor next to the standing leg, it passes the ankle and knee and extendsout into space to full extension. Lifting the foot to the ankle or knee (without,for example, any extension to follow) are allowable movements called coupe andpasse, respectively. Thus, our uniquely defined transitions (and trajectories) arethree separate events for the working leg: coupo, passo, deveo. Next, the dancerperforms a closing movement where the foot remains extended from the bodyand the leg is lowered till the foot is returned to the starting stance. This ismodeled as the event batti – the transition from pose 8 directly to pose 2 with alabel that corresponds to the in-trajectory of a battement (a simpler movement


that looks like a high straight-legged kick). The events coupi, passi, and deveiare also defined, that is, the reverse pose transitions are allowed and used formore complex movements. And, we will enumerate a special event to describethe relative inaction of the standing leg.

Formally, we will model this system as a finite state machine

G = (X,E,O, f, Γ, o, x0, Xm, ε, ω), (10)

where X is the finite state space, E is the event set, O is an output set, f :X × E → X is the state transition function, Γ : X → 2E is the set of feasibleevents (at a given state), o : X × E → O is the output map, x0 ∈ X is theinitial condition, and Xm ⊆ X is a set of marked states. In order to allowfor both synchronous and asynchronous transitions once we take the Cartesiancomposition of two such systems, we use “empty” transitions, which are definedby the symbol ε. The interpretation is that for our finite state machine, weinsist on ε ∈ E, with the result that ε ∈ Γ (x) as well as f(x, ε) = x, ∀ x ∈X. Moreover, we associate ω ∈ O with the outputs from “empty” events, i.e.,o(x, ε) = ω, ∀ x ∈ X.

We will use the output map as a means of preventing physically infeasibleand stylistically inappropriate motions from happening when composing twoautomata in order to emulate more complex routines. For this, we consider thecontinuous set X which spans the entire physical space of configurations - namely,the ones describing the primary motions that are abstracted away by the statemachine. We may define this additional state space as

X = {(θ1, ..., θn) | θi ∈ [θimin, θimax

]} . (11)

For example, this set may describe body shapes, defined in terms of the jointangles which are limited by general kinematic constraints of humanoid geometry.

We can now think of transitions between the discrete states of the automatonas tracing paths3 through X . And, as our formulation associates every eventwith a unique state, we can associate the output map with the path that thecorresponding pose transition sweeps in X .

Sometimes a physically infeasible (i.e. trajectories which intersect) or aes-thetically undesirable trajectory pair will exist. These trajectories are kept trackof in O as follows: For e ∈ E, x ∈ X, we let

o(x, e) ∈ 2X = O. (12)

Thus, in the next section when we consider G × G, the set O will correspond totrajectory pairs. These will be trimmed systematically according to our stylisticparameters.

3 A relaxed version of this may take into account that the angles corresponding to thestates in the automaton are, in practice, approximate. In this case we may think ofthese paths as “tubes.”


4.2 Movement Sequencing

In this section we will outline a general method for combining the elementalunit of our movement model presented in the previous section such as to impartrules – which may be style based – to sequences in the combined state space.In order to capture the fact that these primary motions may be performed atthe same time and thus produce more complex movement sequences, we want tobe able to compose two state machines, of the form in Eq. 10, while transitionsthat are deemed impossible or incorrect are removed from the composed system.For example, we may aspire to go from a one-legged automaton describing theprimary movements of that single leg to a system involving two legs withoutviolating the laws of physics or rules that ensure we stay faithful to a given styleof movement.

To accommodate this, we introduce some compositional operations.Given two finite automata

Gi = (Xi, Ei, Oi, fi, Γi, oi, xi,0, Xi,m, εi, ωi), i = 1, 2,

we let the Cartesian composition of these two systems be given by

G = G1 × G2 = (X,E,O, f, Γ, o, x0, Xm, ε, ω),

whereX = X1 ×X2

E = E1 × E2

O = O1 ×O2

f((x1, x2), (e1, e2)) = (f1(x1, e1), f2(x2, e2))

Γ ((x1, x2)) = Γ1(x1)× Γ2(x2)

o((x1, x2), (e1, e2)) = (o1(x1, e1), o2(x2, e2))

x0 = (x1,0, x2,0)

Xm = X1,m ×X2,m

ε = (ε1, ε2)

ω = (ω1, ω2).

(13)

Note that this is a synchronous composition in the sense that events have tohappen to both of the two systems in order for a transition to happen. However,through the introduction of the empty word, we can produce effectively asyn-chronous transitions directly through the use of the events (e1, ε2) or (ε1, e2) ina straightforward manner.

If we perform this operation on two automata that describe the primarymovements at the ballet barre, one for the right leg and one for the left, asG = Gbarre1 × Gbarre2 , we obtain a system that no longer is a one-legged warm-up routine, but rather a two-legged dance-like model. For example, pas de chat4

is a jump in which the dancer draws both feet up quickly, flexing at the knee,

4 Literal translation from French: “step of the cat.”


in a quick canon sequence. It is modeled for either leg in this automaton as theevent sequence

coupo, passo, passi, plieo, pliei ∈ E?barre,

where ? denotes the Kleene closure. The corresponding event string for the com-posite system would be

(ε1, coupo), (coupo, passo), (passo, passi), (passi, plieo), (plieo, pliei), (pliei, ε2).

Note that both of these joint event strings belong to the composite set (Ebarre×Ebarre)

? and that these strings correspond to sequences of our primary move-ments.

We now introduce two operations, trans(·) and pose(·), which will whittleaway undesirable sequences and narrow our style to something more specificthan strings in (Ebarre × Ebarre)?. Let

G = (X,E,O, f, Γ, o, x0, Xm, ε, ω)

be a finite state machine as per Eq. 10 and let Oinfeas ⊂ O and Ounaesth ⊂ O besubsets of the output set that do not contain ω. The trans(·) operation will takeG and Oinfeas ∪Ounaesth as arguments and return a new finite state machine,

trans(G, Oinfeas ∪Ounaesth) = (X,E,O, f, Γ , o, x0, Xm, ε, ω),

where we only have changed the definition of Γ i.e., the set of events that areallowed to happen at a given state. The new such set is given by

e ∈ Γ (x) ⇔ e ∈ Γ (x) and o(x, e) 6∈ Oinfeas ∪Ounaesth. (14)

Thus, trans(·) removes transitions from our state machine that represent jointmotions that are physically impossible or not of our defined aesthetic.

Secondly, certain poses when composed with one another may not be phys-ically feasible or aesthetically desired. Thus we limit the system from enteringsuch poses through the use of a supervisory controller and two sets of statesXinfeas and Xunaesth which restrict each type of joint state, respectively. For-mally speaking, given a finite state machine together with the sets Xinfeas ⊂ Xand Xunaesth ⊂ X, with x0 6∈ Xinfeas ∪Xunaesth, we let the operation

pose(G, Xinfeas ∪Xunaesth)

be the supervised system G\S, where S is the maximally permissive, non-blocking supervisor that ensures that the set Xinfeas∪Xunaesth is never reached.Note that such supervisors can be automatically generated, e.g., [4].

The final system is then given by

pose(trans(G1 × G2, Oinfeas ∪Ounaesth), Xinfeas ∪Xunaesth). (15)

Returning to our example, the set Oballetinfeas, defined for the ballet model,allows us to whittle away excess transitions in order to produce system behaviors


consistent with the physical capabilities of a bipedal geometry. For example,lifting a flat-footed leg off the ground, in a manner which indicates a jump (or,equivalently, raising the leg when the other is already in the air), without abend in the knees to provide spring for the jump, is physically impossible. Thiscorresponds to state 2 (foot flat on the ground with a straightened knee), and theinfeasible event is an extension of the leg from the ground to state 8 (extendedaway from the body, parallel to the floor). Similar such rules used to govern aleg providing critical support (a leg in state 1, 2, or 3 when the other is in anyof the others, states 44 and 4-8) such as “no coronal extension of the supportingleg from states 1 and 2” can easily be translated into regions of 2X × 2X .

Thus, pairs of regions of continuous space define Oballetinfeas and correspond

to disallowed synchronous leg paths. Likewise, we may enumerate Oballetunaesth todescribe any undesired synchronous motions that, while possible to execute, arenot in the desired style of movement. Since we began this example from such anarrow set of primary motions which are, by design, intended to be performedsynchronously, we leave this set empty.

Each joint state in the two-legged barre automaton is physically feasible;thus, Xballet

infeas is an empty set. The definition of Xballetunaesth in the ballet model

is a straightforward list of two-legged states which are perhaps considered uglyas judged by the metric of ballet; often, these are asymmetrical poses or poseswhich cannot be seen from the audience’s distant perspective.

Finally, with both of these components in place, Eq. 15 reduces to

pose(trans(Gbarre1 × Gbarre2 , Oballetinfeas), Xballetunaesth).

This describes an automaton that accepts strings of body poses that follow ourballet-inspired rule set.

It is worth asking the question of how easy this method is to implement– for example, is naming the sets Oinfeas, Ounaesth, Xinfeas, and Xunaesth insituations where the desired motion style is less formalized a manageable task? Apossible resolution to this question is offered in the notable and exciting extensionpresented in [19] and [20] where the discrete sequencing rules are assembledthrough statements scripted in linear temporal logic (LTL), which resemblesnatural language as it uses Boolean and temporal operators to operate on basicstatements about the system.

4.3 Movement Modulation

As outlined in Sec. 2, motions may be performed with different dynamic quality.Thus, the basic primary motions – which are abstracted away in the discretemotion sequencing framework from the previous section – tell a limiting story ofstylized motion. In this section, we use a linear quadratic optimal control frame-work to find time-varying trajectories between static poses through a mappingbetween Rudolf Laban’s Effort system and weights in a cost function that wasfirst proposed in [21].


Laban names four categories of Effort or motion factors: space, weight, time,and flow. Space, weight, and time deal with individual movements while flowdescribes the quality of a succession of movements; each are described in detailin [25,32].

TIME

WEIGHT

SPACE

flexible direct

strong

light

sudden

sustained Dabbing

Floating Gliding

Flicking

Thrusting Slashing

Wringing Pressing

Fig. 6. The dynamosphere. Laban’s arrangement of eight basic Efforts according tothe axes of space, weight, and time. In bold font are the three Laban motion factorswhich deal with single movements; in italics are the two qualities Laban associates witheach factor; and in plain font are the eight basic Efforts which result from the pairwisecombination of each quality. [32]

The relationship of space, weight, and time can be seen in Laban’s dynamo-sphere as in Fig. 6. The extremes of these three motion factors combine pairwiseto form the eight basic Efforts: dabbing, gliding, floating, flicking, thrusting,pressing, wringing, and slashing. These movements embody the extreme notionsof each motion factor, and our framework will generalize this binary scale to oneof continuously variable weights. Each of the basic Efforts corresponds to a fa-miliar pedestrian action, highlighting, even to a lay audience, the nature of thedynamosphere arrangement: changing the quality of one motion factor movesaround the cube to a different basic Effort. These three motion factors, andthe fourth factor, flow, which describes the quality of the connection betweenmovements, are described – along with some intuition behind our mathematicalinterpretation – in the next four paragraphs.

The space axis describes how the dancer’s attitude toward space is perceived.Flexible movements seem more carefree, meandering, and indirect; direct motionsappear more matter of fact and judicious with their use of space. We pair thisconcept with a system’s notion of reference tracking; direct movements will tracktheir path more aggressively than flexible ones. Thus, we will make use of nominaltrajectories away from which our solutions may deviate or adhere closely.

The axis of weight deals with the emanated sense of weight in the dancer’sbody during the movement. Light movements look as though they are less in-fluenced by gravity – as if they are effortless – whereas strong movements aremuscular and seem taxing on the body to perform. We interpret this as a specifi-cation for how much control effort is used to perform the movement, or, in terms


of a cost function, how “cheap” it is to increase the magnitude of the controlsignal.

Thirdly, Laban prescribes a time axis on which the movement may either besudden or sustained. This describes a quality which, while it adheres closely tothe colloquial notion of these terms, is more subtle than just the duration ofthe movement; that is, a movement which lasts five seconds may be executedwith a sudden or sustained quality. We interpret this as a metric over how muchthe state of the system is allowed to change: during a sustained movement itshould change less while in a sudden movement it may deviate wildly to producea trajectory that appears to the audience as frantic.

Finally, Laban describes the quality of a series of movements with flow; thesemay either be free or bound. In free flow, dancers seems to move through move-ments with great ease and continuity; while in bound flow the dancer feels controland rigidity in the muscles. We interpret this from a systems perspective as alesser (or greater) desire for the dancer to hit poses between movements ex-actly. In the sequencing framework we will employ, this translates to varying theweights on a terminal pose.

To make these concepts mathematically rigorous, consider a linear systemwith an input u ∈ Rm, a state x ∈ Rn, and an output y ∈ Rl which tracks areference signal r ∈ Rl. We establish a quadratic cost function

J =1

2

∫ Tf

0

[(y − r)TQ(y − r) + uTRu+ xTPx

]dt+

1

2(y − r)TS(y − r)

∣∣∣∣Tf

(16)

in order to find an input u principled on the weight matrices Q ∈ Rl×l, R ∈Rm×m, P ∈ Rn×n, and S ∈ Rl×l. By construction, each of these matrices arepositive definite and symmetric. Furthermore, their entries create a continuous,quantitative version of Laban’s Effort system and will determine which move-ment qualities are exhibited by the optimal trajectory, i.e. the trajectory maybe bound, direct, sudden, and strong.

Based on the previous discussion, we associate the weight Q to the Laban’smotion factor space, R with the factor weight, P with time, and S with flow.These weights correlate with the quality of each factor as follows:

Q ∼ direct (17)

R ∼ light (18)

P ∼ sustained (19)

S ∼ bound (20)

where the opposite of the qualities listed, flexible, strong, sudden, and free, areachieved when these weights are relatively small, respectively.


Using these weights as the style-based parameters for varying the resultingtrajectory, we solve the optimal control problem

minuJ

s.t.

{x = Ax+Bu

y = Cx(21)

where A ∈ Rn×n, B ∈ Rn×m, and C ∈ Rl×n.Differentiating the Hamiltonian

H =1

2[(y − r)TQ(y − r) + uTRu+ xTPx] + λf

with respect to u and x gives the first order necessary condition for optimalityand the dynamics of the costate λ = [λ1, λ2, ...λn]:

∂H

∂u= uT (R+BTPB) + xTATPB + λB = 0

from which it follows that

u = −(R+BTPB)−1(BTPAx−BTλT ) (22)

and

∂H

∂x= −λ = xT (CTQC +ATPA) + uTBTPA+ λA− rTQC

from which it follows that

ξ = λT = (ATPB(R+BTPB)−1BTPA− CTQC −ATPA)x

+(ATPB(R+BTPB)−1BT −AT )ξ + CTQr. (23)

Applying the transversality condition we obtain:

ξ(Tf ) = CTSCx(Tf )− CTSr(Tf ). (24)

To solve this system for an optimal x(t) we need to find ξ0. Thus, we assemblea new state z = [x, ξ]T . Now

z = Mz +Nr (25)

where the entries of M and N are determined from Eqs. 21 - 23 and are givenbelow:

M11 = A−B(R+BTPB)−1BTPA (26)

M12 = −B(R+BTPB)−1BT (27)

M21 = ATPB(R+BTPB)−1BTPA− CTQC −ATPA (28)

M22 = ATPB(R+BTPB)−1BT −AT (29)

N1 = [0]n×l (30)

N2 = CTQ. (31)


We know that in general

z(Tf ) = eMTf z0 +

∫ Tf

0

eM(Tf−t)Nr(t)dt, (32)

which we denote byz(Tf ) = Φz0 + q, (33)

where

Φ =

[Φ11 Φ12

Φ21 Φ22

]and q =

[q1

q2

]. (34)

Combining Eq. 24 with Eqs. 33, 34, and 34 we get

ξ0 = (CTSCΦ12 − Φ22)−1[(Φ21 − CTSCΦ11)x0

+CTSCq1 + q2 + CTSr(Tf )]. (35)

Thus, we have found our initial condition z0 = [x0, ξ0]T . When combined withEq. 25, this gives the optimal x(t) – and thus our output y(t) – as nominatedby the weights in Eq. 16.

5 Rule-based Movement Generation

Choreographers are constantly seeking sources of inspiration and techniques togenerate new movement. Often this involves using measures external to theirbodies to generate new constraints for their creative problems – a la Merce Cun-ningham and his dice. Cunningham [5] famously used dice to introduce chanceand randomness into his work; a roll of the dice would determine the next move-ment in a sequence or the order of a show on a given night. These techniquesproduced new and exciting movements. Later in his career, he also developedtechnology, the software suite LifeForms, which allowed for movement generationon a computer, to facilitate his choreographic process. The tools in this chapterafford a similar opportunity: style of motion may be programmed through theautomata-theoretic description of style presented in this chapter.

Developing movement “styles” using automata inherits the same advantagethat automata afford to the technological objects which they are traditionallyused to describe. Namely, they allow for discrete changes in state. In the case ofa remote control, this means that when the highest channel is reached, the TVinterprets the “up” signal from the remote as a command to go down to the firstor lowest channel. This is to some degree inconsistent and would be extremelyinconvenient to capture this desired behavior using continuous mathematics. Inthe case of dancing, the programmatic advantage is that moving between bodyposes does not have to be governed by the natural movement patterns of thechoreographer, but instead offers an opportunity to produce unusual sequencesas drawing arrows between states is a completely disembodied task – just asCunningham’s technique of rolling the dice.


Fig. 7. The underlying poses for the five styles of movement performed during Au-tomaton. Photos snapped by Jessica Portugal.

Fig. 8. The rule set for the dancer Helen and the humanoid robot.

Fig. 9. The rule set for the dancer Camille.

Fig. 10. The rule set for the dancer Alex.


Fig. 11. The rule set for the dancer Erik.

Figures 7-11 show the rule sets for each dancer. Just as the movement pat-terns look different to watch, these automata have different structures5. For ex-ample, one automaton is quite linear and represents the same sequence of move-ment repeated over and over; whereas others present more options and allow forrepeated movement. Moreover, these descriptions are amenable to robotics, andthe styles of movement may be automated. Thus, the rule-set pictured in Fig.8, is also animated on the NAO humanoid robot. By employing different rulesets on each dancer and the robot, the audience is able to see different automataanimated, live. A snapshot from this section of the piece is shown in Fig. 12.

Fig. 12. Five styles of movement are performed during Automaton. Photo by Rob Felt.

6 What did the audience see?: A Human Study

This section will provide an initial step toward validating the goal of endowing arobot with a consistent style of movement. The sequencing framework described

5 Audience members were given an experiential tutorial in automata via a printedautomaton that instructed them on how to fold their programs


in this chapter was used in Automaton as described in the previous section.Audience members were polled after the performances (held on April 6 and 13,2013) for their impressions of the robot’s movement.

Between the two showings, about 100 people saw Automaton, and 28 audi-ence members volunteered to fill out a survey collecting their impressions of therobot’s movement. Figures 13-14 show the results of the responses and the av-eraged values for each question. (The complete responses can be found in [23].)The questions aimed to capture the viewer’s impression of a colloquial notionof “style” as applied to the overall movement of the robot in the show. Thestudy subjects were asked if their impressions of the movement were that it was“specific,” “stylized,” “virtuosic,” “enjoyable,” and whether they “felt” anythingat all about the movement. These questions were phrased broadly and were notmeant to define a subjective or psychological notion of “style.” Instead, the ques-tions simply probed whether the audience believed that something consistent wasunderlying the robot’s movement.

The responses indicate that the viewers polled did experience the robot’smovement as something other than random motions set to music. In particular,the responses to the first two questions support this claim. Question 1 polledthe viewer’s sense that the robot’s movement was very “specific.” The averageof almost 8 on a scale of 1 to 10, with 1 corresponding to random motion and 10corresponding to very specific motion, indicates the audience did feel the robot’smovement was “specific.” Question 2 was similar except phrased using the term“style”; the average rating of about 7 out of 10 indicates that the viewers polledfound the robot’s motion to be more “stylized” than not. The responses to theother questions also indicated the audience sensed some high-level design of therobot’s movement. Thus, this initial study indicates we have achieved roboticmovement that is perceived to be of a certain style.

7 Conclusions

Functional tasks have a straightforward interpretation for metrics of success,efficiency, and completion and are, thus, implementable on robots in a varietyof shapes. However, robots that are shaped like humans have a greater vocab-ulary of movement to aspire to as their movements are compared to the highlycapable and varied natural movements of human beings. To cite two popular ex-amples of how a fully autonomous humanoid robot might first appear in complexroles: in-home hospice robots should move as though they are gentle and caringand museum tour guide robots should have a brisk air of authority about theirmovements. Thus, a quantitative parameterization of high-level aspects of hu-man movement is a necessary component for human-shaped robots that interactwith humans.

Futher, technology of any shape relies more and more on an interpretation ofhuman movement. Human-centered technology has moved from vending machinestyle interfaces where users need to figure out the function of various buttons tooperate a machine to iPhones where much of the interaction takes place based


Fig. 13. A histogram of survey responses collected between both showings of Automa-ton completed by volunteers from the audience. The question shown here was stated asfollows: Please rate your impression of the overall consistency of the robots movement.A rating of 1 indicates that it seemed to be moving completely randomly. A rating of10 indicates that it was moving in a very specific way. Each column counts how manysubjects selected 1, 2, 3, ..., 10, respectively, for this question.

Fig. 14. A histogram of survey responses collected between both showings of Automa-ton completed by volunteers from the audience. The question shown here was statedas follows: Please rate your impression of the style of the robots movement. A rating of1 indicates that it seemed to be moving without style. A rating of 10 indicates that itwas moving in a very stylized way. Each column counts how many subjects selected 1,2, 3, ..., 10, respectively, for this question.


on the machine’s interpretation of the motion of the human fingers. Followingthis trend, Microsoft’s Kinect extracts a human skeleton using computer visiontechniques and aims to incorporate full-body motions into the Xbox entertain-ment system. Being able to interpret the style and other high-level aspects of auser’s movement can enhance these efforts.

Finally, the work presented in this chapter can interface with art. The anal-ysis and creation of art can benefit from quantitative tools. The focus of thischapter has been on the creation of art, but on the part of analysis, being able togenerate – and even measure – stylized movement from three parameters allowsfor movement from different dancers, choreographers, genres, and sections of thesame piece to be compared quantitatively. Such analysis can bolster the bodyof academic work that qualitatively discusses the evolution of and connectionbetween various genres of dance and the differences between the work of twochoreographers and provide insight into the magical effect stylized movementcan have on an audience.

Acknowledgment

This work was supported by the US National Science Foundation through grantnumber 0757317. The performance of “Automaton” was funded by a generousgrant from GA Tech’s School of Electrical and Computer Engineering.

References

1. E. Bradley and J. Stuart. Using chaos to generate variations on movement se-quences. Chaos: An Interdisciplinary Journal of Nonlinear Science, 8(4):800–807,1998.

2. M. Brand and A. Hertzmann. Style machines. In Proceedings of the 27th annualconference on Computer graphics and interactive techniques, pages 183–192. ACMPress/Addison-Wesley Publishing Co., 2000.

3. C. Bregler. Learning and recognizing human dynamics in video sequences. In Com-puter Vision and Pattern Recognition, 1997. Proceedings., 1997 IEEE ComputerSociety Conference on, pages 568–574. IEEE, 2002.

4. C.G. Cassandras and S. Lafortune. Introduction to discrete event systems. Springer,2008.

5. Roger Copeland. Merce Cunningham: The modernizing of modern dance. Rout-ledge, 2003.

6. D. Del Vecchio, R.M. Murray, and P. Perona. Decomposition of human motioninto dynamics-based primitives with application to drawing tasks* 1. Automatica,39(12):2085–2098, 2003.

7. M. Do, J. Romero, H. Kjellstrom, P. Azad, T. Asfour, D. Kragic, and R. Dill-mann. Grasp recognition and mapping on humanoid robots. In Humanoid Robots,2009. Humanoids 2009. 9th IEEE-RAS International Conference on, pages 465–471. IEEE, 2009.

8. M. Egerstedt, T. Balch, F. Dellaert, F. Delmotte, and Z. Khan. What are the antsdoing? vision-based tracking and reconstruction of control programs. In Proceedingsof the IEEE International Conference on Robotics and Automation (ICRA 2005),pages 18–22, 2005.


9. C. Fanti, L. Zelnik-Manor, and P. Perona. Hybrid models for human motion recog-nition. In Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEEComputer Society Conference on, volume 1, pages 1166–1173. IEEE, 2005.

10. M.J. Gielniak, C.K. Liu, and A.L. Thomaz. Stylized motion generalization throughadaptation of velocity profiles. In RO-MAN, IEEE, pages 304–309. IEEE, 2010.

11. M. Gillies. Learning finite-state machine controllers from motion capture data.Computational Intelligence and AI in Games, IEEE Transactions on, 1(1):63–72,2009.

12. P. Hackney. Making connections: Total body integration through Bartenieff funda-mentals. Routledge, 1998.

13. Donna Haraway. A cyborg manifesto: Science, technology, and socialist-feminismin the late twentieth century. Simians, Cyborgs and Women: The Reinvention ofNature, pages pp.149–181., 1991.

14. E. Hsu, K. Pulli, and J. Popovic. Style translation for human motion. In ACMTransactions on Graphics (TOG), volume 24:3, pages 1082–1089. ACM, 2005.

15. O.C. Jenkins and M.J. Mataric. Automated derivation of behavior vocabulariesfor autonomous humanoid motion. In Proceedings of the second international jointconference on Autonomous agents and multiagent systems, pages 225–232. ACM,2003.

16. P. Kingston and M. Egerstedt. Time and output warping of control systems:Comparing and imitating motions. Automatica, 2011.

17. L. Kovar, M. Gleicher, and F. Pighin. Motion graphs. In ACM Transactions onGraphics (TOG), volume 21:3, pages 473–482. ACM, 2002.

18. D. Kulic, C. Ott, D. Lee, J. Ishikawa, and Y. Nakamura. Incremental learning of fullbody motion primitives and their sequencing through human motion observation.The International Journal of Robotics Research, 31(3):330–345, 2012.

19. A. LaViers, Y. Chen, C. Belta, and M. Egerstedt. Automatic generation of balleticmotions. ACM/IEEE Second International Conference on Cyber-Physical Systems,pages 13–21, 2011.

20. A. LaViers, Y. Chen, C. Belta, and M. Egerstedt. Automatic sequencing of balletposes. Robotics and Automation Magazine, IEEE, 18(3):87–95, 2011.

21. A. LaViers and M. Egerstedt. The ballet automaton: A formal model for humanmotion. American Control Conference, Proceedings of the 2011, 2011.

22. A. LaViers and M. Egerstedt. Style based robotic motion. American ControlConference, Proceedings of the 2012, 2012.

23. A. LaViers and M. Egerstedt. Style Based Robotic Motion. PhD thesis, 2012.24. L. Lee and WEL Grimson. Gait analysis for recognition and classification. In Auto-

matic Face and Gesture Recognition, 2002. Proceedings. Fifth IEEE InternationalConference on, pages 148–155. Ieee, 2002.

25. V. Maletic. Body, space, expression. Walter de Gruyter & Co., Berlin, 1987.26. P. Martin and M.B. Egerstedt. Timing control of switched systems with appli-

cations to robotic marionettes. Discrete Event Dynamic Systems, 20(2):233–248,2010.

27. T. Matsubara, S. Hyon, and J. Morimoto. Learning stylistic dynamic move-ment primitives from multiple demonstrations. In Intelligent Robots and Systems(IROS), 2010 IEEE/RSJ International Conference on, pages 1277–1283. IEEE,2010.

28. S. Miller, J. Van Den Berg, M. Fritz, T. Darrell, K. Goldberg, and P. Abbeel.A geometric approach to robotic laundry folding. The International Journal ofRobotics Research, 31(2):249–267, 2012.


29. S. Nakaoka, A. Nakazawa, K. Yokoi, and K. Ikeuchi. Leg motion primitivesfor a dancing humanoid robot. In Robotics and Automation, 2004. Proceedings.ICRA’04. 2004 IEEE International Conference on, volume 1, pages 610–615. IEEE,2004.

30. T. Nakata, T. Mori, and T. Sato. Analysis of impression of robot bodily expression.Journal of Robotics and Mechatronics, 14(1):27–36, 2002.

31. A. Nakazawa, S. Nakaoka, and K. Ikeuchi. Synthesize stylistic human motionfrom examples. In Robotics and Automation, 2003. Proceedings. ICRA’03. IEEEInternational Conference on, volume 3, pages 3899–3904. IEEE, 2003.

32. J. Newlove and J. Dalby. Laban for All. Nick Hern Books, 2004.33. X. Ren. Finding people in archive films through tracking. In Computer Vision and

Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pages 1–8. IEEE,2008.

34. E. Surer and A. Kose. Methods and technologies for gait analysis. ComputerAnalysis of Human Behavior, pages 105–123, 2011.

35. L. Wang, W. Hu, and T. Tan. Recent developments in human motion analysis.Pattern recognition, 36(3):585–601, 2003.

36. G.W. Warren and S. Cook. Classical ballet technique. University of South FloridaPress, Gainesville, FL, 1989.

style-based robotic motion in contemporary dance …style-based robotic motion in contemporary dance...

Documents