1 roberto bresin – humaine workshop s2s 2 – 2004.09.19_21 real-time visualization of musical...

1Roberto Bresin – Humaine Workshop S2S2 – 2004.09.19_21

Real-time Visualizationof

Musical Expression

Roberto BresinSpeech, Music and Hearing Dep. - KTH

http://www.speech.kth.se/~robertohttp://www.speech.kth.se/music


Outlook

• Expressive music performance

• Acoustical cues of importance for communicating expressivity

• How acoustical cues can influence the performance style/intended emotion

• Real-time Visualization of Musical Expression

• Examples and demonstration


The MusicianThe musicianMusical communicationModelling music performanceEmotional colouringApplications

The Listener Recognition of emotionVisualisation of musical expression


The Score and the Performance

How important is the performance?

• Dead-pan by computer and sampler• Schumann’s Träumerei by Alfred Brendel

10

20

30

40

50

60

70

80 ²DR [%] Horowitz 65

-10

-20

-30

1 2 3 4 5 6 7 8 9

10

20

30

40

50

60

70

80

-10

-20

-30

10

10

10

20

30 ²DR [%] Schnabel

-10

-20

1 2 3 4 5 6 7 8 9

30

-10

-20

10

20

30

40

50 ²DR [%] Brendel

-10

-20

-30

1 2 3 4 5 6 7 8 9

20

30

40

50

-10

-20

-30

IOI (%) BrendelIOI (%) Brendel

Tim

e d

evia

tio

n f

rom

sco

re

The musicianMusical communicationModelling music performanceEmotional colouringApplications



The Score and the Performance

10

20

30

40

50

60

70

80 ²DR [%] Horowitz 65

-10

-20

-30

1 2 3 4 5 6 7 8 9

10

20

30

40

50

60

70

80

-10

-20

-30

10

10

10

20

30 ²DR [%] Schnabel

-10

-20

1 2 3 4 5 6 7 8 9

30

-10

-20

10

20

30

40

50 ²DR [%] Brendel

-10

-20

-30

1 2 3 4 5 6 7 8 9

20

30

40

50

-10

-20

-30

IOI (%) BrendelIOI (%) BrendelTim

e d

evia

tio

n f

rom

sco

re

IOI (%) IOI (%) SchnabelSchnabel

IOI (%) IOI (%) Horowitz 65Horowitz 65




Design of Performance Rules

Performance rules obtained mainly with 2 methods:

• analysis-by-synthesis• analysis-by-measurement

Generative grammar for automatic music performance




Dead-pan, K=0

Exaggerated, K = 4.4

Moderate, K = 2.2

Inverted, K = -2.2

Duration contrast ruleIn

ter o

nset

I nte

rval d

evia

t ion

s (

%)


Phrase Arch Rule

Dead-pan

Exaggerated

ΔIO

I (%

)


KTH Performance Rules

• Descriptions of different performance principles used by

musicians

• General applicability

• K values change the overall quantity of each rule

• Context dependency

• ~ 30 rules

• 30 years of research at KTH

ScoreScore RulesRules PerformancePerformance

K valuesK values




KTH Performance Rules

Phrasing Phrase archFinal

ritardandoPunctuationHigh loud

Harmonic/melodic tension Harmonic/melodic charge

Repetitive patterns and grooves Swing

Articulation PunctuationStaccato/legato

Accents Accent rule

Ensemble timing Ensemble swingMelodic sync




Director MusicesA program for modelling music performancehttp://www.speech.kth.se/music/performance




Mapping from Emotional Expression to Rule Parameters

For each emotion:

Select a palette of rule parameters according to previous findings

Mapping

Emotional expression

Rule parameters




Cues for the simulation of

emotions in music

performance

(by A.Gabrielsson and P.Juslin,

Psychology of Music, 1996, vol. 24)

• No expression

• Tenderness

• Solemnity

• Happiness

• Sadness

• Anger

• Fear

Synthesis of Emotional ExpressionThe musicianMusical communicationModelling music performanceEmotional colouringApplications



• TENDERNESSslow mean tempo (Ga96)slow tone attacks (Ga96)low sound level (Ga96)small sound level variability (Ga96)legato articulation (Ga96)soft timbre (Ga96)large timing variations (Ga96)accents on stable notes (Li99)soft duration contrasts (Ga96)final ritardando (Ga96)

• HAPPINESSfast mean tempo (Ga95)small tempo variability (Ju99)staccato articulation (Ju99)large articulation variability (Ju99)high sound level (Ju00)little sound level variability (Ju99)bright timbre (Ga96)fast tone attacks (Ko76)small timing variations (Ju/La00)sharp duration contrasts (Ga96)rising micro-intonation (Ra96)

• ANGERhigh sound level (Ju00)sharp timbre (Ju00)spectral noise (Ga96)fast mean tempo (Ju97a)small tempo variability (Ju99)staccato articulation (Ju99)abrupt tone attacks (Ko76)sharp duration contrasts (Ga96)accents on unstable notes (Li99)large vibrato extent (Oh96b)no ritardando (Ga96)

• SADNESSslow mean tempo (Ga95)legato articulation (Ju97a)small articulation variability (Ju99)low sound level (Ju00)dull timbre (Ju00)large timing variations (Ga96)soft duration contrasts (Ga96)slow tone attacks (Ko76)flat micro-intonation (Ba97)slow vibrato (Ko00)final ritardando (Ga96)

• FEARstaccato articulation (Ju97a)very low sound level (Ju00)large sound level variability (Ju99)fast mean tempo (Ju99)large tempo variability (Ju99)large timing variations (Ga96)soft spectrum (Ju00)sharp micro-intonation (Oh96b)fast, shallow, irregular vibrato (Ko00)

Positive Valence

Negative Valence

High ActivityLow Activity

From Juslin (2001)


Example: SADNESS

Expressive Cue Analysis Synthesis (Director Musices)

Tempo Slow Tone IOI is lengthened by 30%

Sound level Moderate or low Sound level is decreased by 6 dB

Articulation Legato Tone duration = Tone IOI

Time deviations Moderate Duration Contrast Rule (k = -2)

Phrase Arch Rule applied on phrase level (k = 1.5)

Phrase Arch Rule applied on sub-phrase level (k = 1.5)

Final ritardando Yes Obtained from the Phrase Arch Rule




IOI deviations

dB deviations

articulation

Example: SADNESSModel Score


ConclusionsConclusions

Emotional expressioncan be derived directlyfrom the music score,

simplyby enhancing music structure




Better Monophonic Ringtones!

Today (nom.)

Natural

:-) Happy

>:-( Angry

:-( Sad

=|:-| Solemn

Mozart G minor

0 2 4 6 8 10 12 -1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

0 2 4 6 8 10 12 -1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8 0 2 4 6 8 10 12 -1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

0 2 4 6 8 10 12 -1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

0 2 4 6 8 10 12 -1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

0 2 4 6 8 10 12 -1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

Dead-pan

Happy

Angry

Natural

Sad

Solemn

www.notesenses.com




Usher, Burn

MechanicalMusicalHappy

Jennifer Ellison, Bye Bye Boy

MechanicalMusicalRomantic

Better Polyphonic Ringtones!

www.notesenses.com




pDM –performance rules in real-timeAnders Friberg + MEGA project (IST EU)




Real-time Visualization of

Musical Expression


Background

Feel-Me project

Design a computer program for teaching students to play expressively

The system includes a tool for automatic extraction of acoustic cues (CUEX):

pitch, duration, sound level, articulation, vibrato, attack velocity, spectrum




Tempo

Loudness

Timbre

Encoding Decoding

Cue Utilization Cue Utilization

The Listener

Articula.

The Performance The Performer intention expressive cues judgment

Accuracy

others

Anger Anger

rPerformer rListener

Matching

.87

.26 .47 .63

-.26

.22 .55 .61

-.39

.92

Lens model: quantifies the expressive communication between performer and listener


Aim

Design a tool for real-time visual feedback to expressive performance

Mapping of acoustic cues:

– Non-verbal– Intuitive– Informative (including emotional expression)

Previous studies: cross-modality speeds stimuli discrimination




Cue analysis

Expression mapper

Audio Emotion

TempoSound level

Articulation...

Implementations

CUEX

Simplified real-timeversion

Mult. Regression

Fuzzy inspired

Recognition of EmotionThe musicianMusical communicationModelling music performanceEmotional colouringApplications

The ListenerRecognition of emotionVisualisation of musical expression


From audio to emotion recognition

Standardisation

Audio input

Cue extraction

Fuzzy mapping

- +0

∑ / 3

Happiness 0-1Happiness 0-1

temposoundlevel articulation

∑ / 3

Sadness 0-1

∑ / 3

Anger 0-1

Fuzzy mapping

- +0

Fuzzy mapping

- +0

The musicianIntroductionModelling music performanceEmotional colouringApplications

The ListenerRecognition of emotionBody ExpressionGhost in the Cave


Experiment

2 melodies, Brahms (minor) & Haydn (Major)3 instruments (piano, guitar, saxophone)12 performances per instrument (12 emotional intentions)

24 colour nuances8 levels of hue 2 levels of brightness 2 levels of saturation

2 groups of 11 subjects each(1 group per melody)




Experiment: main resultsHUE

Happiness YellowFear BlueSadness Violet & BlueAnger RedLove Blue & Violet

BRIGHTNESS

Observed tendency:Minor tonality Low brightness (Dark colours)Major tonality High brightness (Light colours)

Interaction involving sadness:Even for major tonality low brightness is preferred for all instruments




Experiment: main resultsBox & Whis ker Plo t: happ ines s

Brigh tnes s

ha

pp

ine

ss

Mean ±SE ±1 ,96*SE

Inst

rum

en

t: P

ian

o

-0 , 5

0 ,0

0 ,5

1 ,0

1 ,5

2 ,0

2 ,5

3 ,0

Inst

rum

en

t: G

uita

r

-0 , 5

0 ,0

0 ,5

1 ,0

1 ,5

2 ,0

2 ,5

3 ,0

P iec e : H A Y D N

Inst

rum

en

t: S

axo

ph

on

e

1 . 0 0 . 5-0 ,5

0 ,0

0 ,5

1 ,0

1 ,5

2 ,0

2 ,5

3 ,0

P iec e : B R A H MS

1 .0 0 . 5


Experiment: main resultsBox & Whis ker Plo t: s adnes s

Brigh tnes s

sad

ne

ss

Mean ±SE ±1 ,96*SE

Inst

rum

en

t: P

ian

o

0 , 20 ,40 ,60 ,81 ,01 ,21 ,41 ,61 ,82 ,02 ,22 ,42 ,6

Inst

rum

en

t: G

uita

r

0 , 20 ,40 ,60 ,81 ,01 ,21 ,41 ,61 ,82 ,02 ,22 ,42 ,6

P iec e : H A Y D N

Inst

rum

en

t: S

axo

ph

on

e

1 . 0 0 . 50 ,20 ,40 ,60 ,81 ,01 ,21 ,41 ,61 ,82 ,02 ,22 ,42 ,6

P iec e : B R A H MS

1 .0 0 .5


Experiment: main resultsSimilar colour palettes within instruments

DISGUST (SAX)

0

0,5

1

1,5

2

2,5

3

3,5

4

red

red

_d

ark

red

_lig

ht

ora

ng

e

ora

ng

e_

da

rk

ora

ng

e_

ligh

t

yello

w

yello

w_

da

rk

yello

w_

ligh

t

gre

en

gre

en

_d

ark

gre

en

_lig

ht

cya

n

cya

n_

da

rk

cya

n_

ligh

t

blu

e

blu

e_

da

rk

blu

e_

ligh

t

vio

let

vio

let_

da

rk

vio

let_

ligh

t

ma

ge

nta

ma

ge

nta

_d

ark

ma

ge

nta

_lig

ht

Haydn

Brahms




Experiment: main resultsSHAME (SAX)

0

1

2

3

4

5re

d

red

_d

ark

red

_lig

ht

ora

ng

e

ora

ng

e_

da

rk

ora

ng

e_

ligh

t

yello

w

yello

w_

da

rk

yello

w_

ligh

t

gre

en

gre

en

_d

ark

gre

en

_lig

ht

cya

n

cya

n_

da

rk

cya

n_

ligh

t

blu

e

blu

e_

da

rk

blu

e_

ligh

t

vio

let

vio

let_

da

rk

vio

let_

ligh

t

ma

ge

nta

ma

ge

nta

_d

ark

ma

ge

nta

_lig

ht

Haydn

Brahms

SHAME (GUITAR)

0

1

2

3

4

5

red

red

_d

ark

red

_lig

ht

ora

ng

e

ora

ng

e_

da

rk

ora

ng

e_

ligh

t

yello

w

yello

w_

da

rk

yello

w_

ligh

t

gre

en

gre

en

_d

ark

gre

en

_lig

ht

cya

n

cya

n_

da

rk

cya

n_

ligh

t

blu

e

blu

e_

da

rk

blu

e_

ligh

t

vio

let

vio

let_

da

rk

vio

let_

ligh

t

ma

ge

nta

ma

ge

nta

_d

ark

ma

ge

nta

_lig

ht

Haydn

Brahms

SHAME (PIANO)

0

1

2

3

4

5

red

red

_d

ark

red

_lig

ht

ora

ng

e

ora

ng

e_

da

rk

ora

ng

e_

ligh

t

yello

w

yello

w_

da

rk

yello

w_

ligh

t

gre

en

gre

en

_d

ark

gre

en

_lig

ht

cya

n

cya

n_

da

rk

cya

n_

ligh

t

blu

e

blu

e_

da

rk

blu

e_

ligh

t

vio

let

vio

let_

da

rk

vio

let_

ligh

t

ma

ge

nta

ma

ge

nta

_d

ark

ma

ge

nta

_lig

ht

Haydn

Brahms


The ExpressiBall Expressive performance space as a mapping of acoustical cues and emotions

X Tempo Color EmotionY Sound level Shape Articulation Z Attack velocity & Spectrum energy

Lo

ud

So

ft

Slow Fast

StaccatoAngryFast attackHigh energy

Lo

ud

So

ftSlow Fast

LegatoSadSlow attackLow energy


The ExpressiBall

DEMO!




ExpressiBall:Current & Future Work…

• Sonification of the ExpressiBall

• Usability test with students

• Set-up depending on instrument/spectrum• Use complex colour palettes (Pictures?

Abstract patterns/shapes?)

• Other possible applications:• ”Colour Monitor” in discotheques• Computer screen saver• …




Outlook

• Expressive music performance

• Acoustical cues of importance for communicating expressivity

• How the use of acoustical cues can influence the performance style/intended emotion

• Real-time Visualization of Musical Expression

• Examples and demonstration

http://www.speech.kth.se/music/performance


• FET-Open Coordination Action (contract no. IST-2004-03773)

• 11 partners (3 Humaine partners)

• Started: June 1, 2004

• Duration: 3 years

• Overall objective:

”To bring together the state-of-the-art research in the sound domain and in the proper combination of human sciences, technological research and neuropsychological sciences that does relate to sound and sense”.

Sound to Sense, Sense to Soundhttp://www.s2s2.org


The End

http://www.speech.kth.se/music/performance

1 roberto bresin – humaine workshop s2s 2 – 2004.09.19_21 real-time visualization of musical...

Documents

performance ioi

automatic music performance

kth score rules performance

demonstration slide

score ioi

exaggerated ioi

alfred brendel ioi

schnabel ioi