On the parameterization of clapping

Herwin van WelbergenZsófia Ruttkay

Human Media Interaction, University of Twente

Content

Context and goals Related work Experiment setup Results Conclusion Questions

Context: Reactive Virtual Trainer An ECA acting out exercises a user is

supposed to do Perceives the movement of the user Reactive

Gives feedback Using speech, gestures and motion

(Re)schedules and adapts exercises Tempo changes

Subtle timing and lifelikeness of motion is important

Goal The generation of believable,

adaptable exercise motion in real time How can we parameterize motion?

What parameters? Tempo, amplitude (for accentuation?), …

How do the parameters relate? How do they affect movement?

How is speech synchronized with exercise motion?

Related work: biomechanics Typical biomechanical research setup

Very obtrusive Measuring one movement characteristic Gaining ‘deep’ knowledge

Our setup Unobtrusive Measuring a wider range of characteristics Less depth, measure on an abstraction level

that gains us parameters for movement generation

Related work: finding animation parameters Statistical methods (Egges et al), machine

learning (Brand et al) Finds independed parameters Not intuitive Highly depended on analyzed data set

Laban Movement Analysis Effort can automatically be found from

movement data (Zhao et al) Shape? Our parameters (tempo, amplitude) can be

mapped to LMA parameters

Related work: parameterized animation Rule based (EMOTE, Neff et al, Hartmann et

al, ..) Uses movement models Typically does not deal with dependence

between parameters Lack of detail

Example based (Wiley et al, Kovar et al) By blending examples Nr of examples needed grows exponentially

with nr of parameters

Related work: model based gesture synthesis Kopp et al: Uses biomechanical rules of

thumb to generate movement Real time Domain: speech accompanying gestures

We plan to extend on this work Use in rhythmic movement domain Providing parameterization Providing whole body movement Introducing movement variability

Focus Analysis of a clapping exercise Analyzed aspects:

Synchronization of speech and motion How does a change of tempo affect

movement? Time distribution Movement path Amplitude Left-right hand symmetry Whole body involvement

Clapping experiment: setup

Mocap analysis of two subjects

Instructions: ‘Free clap’:

Clap and count from 21 to 31

‘Metronome driven clap’:

Clap and count to the metronome

Synchronization of clap and speech

Phases from movement in gestures The phonological synchrony rule

holds for clapping

Time distribution in phases Free clap was executed consistently

at ≈ 60 bpm One subject made use of a pre-

stroke hold at 30 bpm The relative duration of the phases

does not change with tempo The standard deviation of the

relative duration decreased with one subject

Movement path of the hands

Amplitude: how to measure

Maximum distance between hands Path is curved

Max distance between hands alone does not display the amount of motion

Distance along path

Amplitude: observations Path distance and max. hand distance

decrease with tempo Average speed is constant at different

tempos There is a linear relation between period

and path distance Pathdistance = a + b ▪ period

Amplitude of free clap is higher Average speed of free clap is higher

Amplitude

Period vs path length

Left-right hand symmetry:How to measure Model: self oscillating

systems Closed orbit between position

(x) and speed (v) x is the normalized angle x^ θ is the phase angle Relative phase angle:

Φ = θleft-θright

Negative Φ means right hand leads

Left-right hand symmetry at 90 bpm

Left-right hand symmetry:Theory

Right handed subjects lead a rhythmic task with their right hand (Treffner et al) But such asymmetry can disappear when the task

is metronome driven Stability of Φ depends on the tempo and

mass imbalance (Treffner et al, Fitzpatrick et al) Higher tempo => higher |Φ| Higher tempo => higher variability in Φ

Left-right hand symmetry:Findings Mean Φ is consistently negative for our

right-handed subjects No difference between metronome

driven and free clap in mean Φ The standard deviation of Φ

increases with tempo No significant relation between mean Φ

and tempo was found

Whole body involvement

By annotating if markers move in the same tempo as the hands

Movement was found on the head and torso for all tempos

For low tempos movement was even observed up to the thighs and knees

Conclusions The phonological synchrony rule was validated

for clapping Clapping can be sped up by making the path

distance smaller A pre-stroke hold can be used to slow down Clapping is clearly a whole body motion At a faster tempo, fewer body parts are

perceivably involved Left-right hand movement variability increases

with tempo For both right-handed subjects, the right hand

was leading The metronome did not diminish this lead

Further work Ultimately: generate clapping motion given

tempo + personal characteristics More recordings

Free clapping without counting Tempo transitions How do personal characteristics affect movement?

Deeper analysis How does variability affect the movement path?

Generation Can movement on the rest of the body be generated

given movement on the arms (as in Egges, Pullen)? Blending clap animation at different tempos to gain

animation at a new tempo (as in Kovar)?

Questions

Easter eggs

Why use gesture-like phases for clapping? The stroke of a speech accompanying gestures (SAG) is

at an energy peak in the movement and expresses meaning (McNeill)

Claps have such a clear peak But this peak does not express meaning

Why compare SAG and clapping? The form of clap movement and SAG is similar

Excursions: start in rest, end in rest Peak structure Well bounded But not symmetric

May find information on the nature of the phonological synchrony rule

Does it depend on form or meaning?

Precision

Precision No significant correlation between

metronome period and avg clap ‘error’ or variability of clap tempo was found

Measured both absolute and relative to the metronome period

Left-right hand position at 90 bpm

3D hand & elbow positions

3D hand positions at different tempos

Free clap amplitude vs metronome drive clap amplitude