motioncomposer - the search for coherence in movement and sound

Robert Wechsler

Workshop/Roundtable

McGill University, Feb.29-Mar.2, 2012

MotionComposer- the search for coherence in movement and sound

www.motioncomposer.de this power point may be downloaded from: www.palindrome.de/ppt

Mercredi

1 – 6 workshop

Jeudi

1 – 6 workshop

Vendredi

2 – 3 presentation

3 – 5 roundtable

Überblick


introductions

demonstration

3 talks - 40 min ea.

discussion after each 20 – 40 min.

project design and execution

project presentation, de-briefing

Überblick - Workshop (both days)


who am i?

interactive performance

motion tracking

mapping and synaesthesia

the motioncomposer: an intuitive expression device for persons with disabilities (Bauhaus Universität)

Überblick – 3 talks over 2 days.


dancer, choreographer, developer

- New York, Nürnberg, Stuttgart, Weimar

- Palindrome Dance Company

- interactive since 1975

- developed EyeCon software with Frieder Weiss 1995-2005

- teach motion tracking

- MotionComposer project director, Bauhaus University

me

www.palindrome.de/ppt Part 1: me

influences

New York City in the 1980's

www.palindrome.de/ppt Part 1: me

John Cage Merce Cunningham

Interactive what

activity response

activity /response activity /response

www.palindrome.de/ppt Part 2: Interactive Performance

interactive performance is nothing new

Cave drawings, Spain ca. 4000 BC

Dionysis Thr. Athens - 300 BC

mechanical technology was used to empower performers

Louis XIV - sun king and megalomanic

Louis loved mechanical devices, toys, remote

acting devices (automation)

He also loved cannons, the ultimate amplifier of gesture.

Interactive devices were used by New York performance artists in the 60's and 70's

Robert Rauschenberg - "Pelican" 1963

Laurie Anderson - "Walking Solo for Tape Bow Violin" 1979

Interactive Performance today

Interaction Design

Gesture Sonification

Motion Capture

Virtual Environments

Immersive Environments

Haptic responders

Diversified Performance (using internet)

and and and

www.palindrome.de/ppt Part 2: Interactive Performance

Part 3: Motion Tracking

motion tracking vs. motion capture

motion capture– highly accurate, expensive (ca. 1m €)– generally not realtime– used for data collection (research), making realistic human or animal animations

(films, games, etc.)

motion tracking– less data, less equipment, less cost (ca. 1k-2k €)– concerned with motion qualities like dynamic, direction of motion– real time

www.palindrome.de/ppt Part 3: Motion Tracking

motion capture

– tracks location of fixed positions on body with reflective markers– 12-24 cameras, each lens is ringed with infrared lights


• Vicon is a leading company in motion capture

motion capture

motion capture

– performances with interactive animated characters: "No DNA" (http://www.nodna.com)

– dance pieces on stage

– Motione (2003-5) Arizona State U. (http://ame.asu.edu/motione)

(Trisha Brown, Kaiser, et.al.)

"22" (Bill T. Jones, Kaiser , et.al.)– Luc Vanier et al. at the interactive performance facility U of Wisconsin at

Milwaukee (http://www.isl.uiuc.edu/Publications/final%20dance1.pdf)

real-time mo-capture


http://www.nodna.com/

http://www.nodna.com/

http://ame.asu.edu/motione

http://ame.asu.edu/motione

http://www.isl.uiuc.edu/Publications/final%20dance1.pdf

motion tracking sensors

• body-oriented

• environment-oriented


motion tracking sensors

• body-oriented

– body-worn electrodes

• EEG

• ECG

• EMG

• Touching

– joint bend-sensors

– accelerometers, other body-worn devices

• environment-oriented

– video camera-based (eg. eyecon)

– infrared camera-based

– floor sensors, wall sensors, etc


Part 4: Mapping and Synaesthesia

Synaesthesia

Kandinsky

Dostoyovsky

www.palindrome.de/ppt Part 4: Mapping and Synaesthesia


Dance + Music = ?

Where , how do they connect?

How can we build up this connection?

What is Mapping?

movement sound


movement sound


What possibilities does the technology allow?

analogue Digital

sensor device

d-to-aconversion

a-to-dconversion

processing output device

e.g. movemente.g. sound

analogue

What is Mapping?

movement soundmakes


What is Mapping?

dance musicmakes


What is Mapping?

dance music


What is Mapping?

dance music

correlation?

What is Mapping?

turns off

correlation

turns on

iterate

toggle

lower

raise

complexcontinuous variables

dance music

What is Mapping?

turns off

iterate

toggle

lower

raise

correlation

turns on

dance music

complexcontinuous variables

What is Mapping?

correlationdance music

What is Mapping?

Number of mappings (M) = number of Input parameters (I) x number of Relationships (R) x number of Output parameters (O)

M = I x R x O


There are thousands of possibilities.

Most are intelligible.

few are palpable.


Coherent Mapping -- movement parameters

Coherent Mapping -- movement parameters

spatial / temporal

Coherent Mapping -- movement parameters / spatial

Coherent Mapping -- music parameters / time and timing

1 discrete events

1 discrete events

Accents Stops

Starts Other instaneous changes



2 Tags



3 Modulation



4 Repetition, sequences and timing patterns

A, B, A, B, ... (bi-modal)

A, B, C, D, E, F, ... (scales and melodies)

A, A, A, A, A, A,A, ... (rhythms and randomness)



5 Multi-dimensional „qualitative“ assocations

many evocative qualities arise through combinations of aural and kinesthetic experience.


Coherent Mapping – time and space

dynamic activity, stillness, flow (direction) sensing

background substraction, „3d cams“ position, shape

intelligemt systems body part and gesture recognition

Motion Tracking technology - approaches

Motion Tracking - Software

MT:EyeCon, Eyesweb, MotionComposer, MAX/Jitter (et al.

Cyclops), Isadora, VNS, Icarus .. Vicon (etc.)

Sound:EyeCon, Kontakt, Ableton (Max live), ...

DSP:Csound, Max/msp, Pure Data, Supercollider

Hardware

Computers

Cameras

Framegrabbers

Motion Tracking - Hardware

thank you.

questions?

Join MotionComposer...

www.motioncomposer.com

[email protected]

www.palindrome.de this power point may be downloaded from: www.palindrome.de/ppt

thank you.

questions? ideas?

[email protected]

www.palindrome.de this power point may be downloaded from: www.palindrome.de/ppt

motioncomposer - the search for coherence in movement and sound

Documents

interactive devices

motion trackingmotion

interactive performancepart

motion qualities

power point

motion trackingless

new york performance

mechanical devices