1 roberto bresin - an introduction to affective music – 2004.08.30 an introduction to affective...

1Roberto Bresin - An Introduction to Affective Music – 2004.08.30

An Introduction to Affective Music,

Theory and Some Applications

Roberto Bresin

http://www.speech.kth.se/music/performance


OutlookAim

To explain how it is possible to communicate different emotions with the same music score

Part I: The science of music performanceAnalysis & synthesis of music performance• The most important techniques for measuring and modelling a

performance• The acoustical cues of importance for communicating expressivity• How the use of acoustical cues can influence the performance style• A rule-based system for the synthesis of music performance

Part II: Emotion in music performance• Emotionally expressive music performance• Real-time Visualization of Musical Expression • Examples• Applications

– Emotional colouring of music performance– expressive ringtones in mobile phones – visual display of emotion in music performance


Part I

The Science of Music Performance


Musical Communication

composer musician instrument listener

score gestures sound

Computational modelsComputational models


The MusicianThe musicianMusical communicationModelling music performanceEmotional colouringApplications

The Listener Recognition of emotionVisualisation of musical expression


What is communicated?

• The music

• Emotions

• Imagined and real motion• …

The musicianMusical communicationModelling music performanceEmotional colouringApplications



Score - Score - Notes, harmony, melody, Notes, harmony, melody, rhythm, pitch, texture, instrumentsrhythm, pitch, texture, instruments

Performance - Performance - Tempo, phrasing, Tempo, phrasing, articulation, intonationarticulation, intonation

AuditiveAuditive

Environment - Concert, club, Environment - Concert, club, home, live/recordinghome, live/recording

AudienceAudience

Social/culturalSocial/cultural

Body movements and gesturesBody movements and gestures

People, clothes, stage lightning, etc.People, clothes, stage lightning, etc.

VisualVisual

MemoryMemory

Musical knowledgeMusical knowledge

Past experiencePast experience

Different factors in the communication


What can be studied?

• What is a musical performance?

• Emotional communication: Accuracy,

musical factors

• Emotional affect

• Couplings to motion: Musicians gestures and

the resulting sound

• Visual perception of musicians body

movements




The Score and the Performance

How important is the performance?

• Dead-pan by computer and sampler• Schumann’s Träumerei by Alfred Brendel

10

20

30

40

50

60

70

80 ²DR [%] Horowitz 65

-10

-20

-30

1 2 3 4 5 6 7 8 9

10

20

30

40

50

60

70

80

-10

-20

-30

10

10

10

20

30 ²DR [%] Schnabel

-10

-20

1 2 3 4 5 6 7 8 9

30

-10

-20

10

20

30

40

50 ²DR [%] Brendel

-10

-20

-30

1 2 3 4 5 6 7 8 9

20

30

40

50

-10

-20

-30

IOI (%) BrendelIOI (%) Brendel

Tim

e d

evia

tio

n f

rom

sco

re




The Score and the Performance

10

20

30

40

50

60

70

80 ²DR [%] Horowitz 65

-10

-20

-30

1 2 3 4 5 6 7 8 9

10

20

30

40

50

60

70

80

-10

-20

-30

10

10

10

20

30 ²DR [%] Schnabel

-10

-20

1 2 3 4 5 6 7 8 9

30

-10

-20

10

20

30

40

50 ²DR [%] Brendel

-10

-20

-30

1 2 3 4 5 6 7 8 9

20

30

40

50

-10

-20

-30

IOI (%) BrendelIOI (%) BrendelTim

e d

evia

tio

n f

rom

sco

re

IOI (%) IOI (%) SchnabelSchnabel

IOI (%) IOI (%) Horowitz 65Horowitz 65




Collecting Data of Expressive Performances

• Expert musicians (Lars Frydén for KTH)– Expertise is translated into rules

• Measurements of recorded performances– Commercial recordings (CDs)– Computer controlled acoustical instruments

(Disklavier, Böserndorfer)




Design of Performance Rules

Performance rules obtained mainly with 2 methods:

• analysis-by-synthesis• analysis-by-measurement

Generative grammar for automatic music performance




MUSICSCORE(MIDI)

PERFORMEDMUSIC(MIDI)

DIRECTOR MUSICES

(performance rules)

PROGRAMMERPROFESSIONAL

MUSICIAN

NEW / MODIFIEDRULE

K values (Rule quantity)

Analysis-by-Synthesis of Music Performance


Dead-pan, K=0

Exaggerated, K = 4.4

Moderate, K = 2.2

Inverted, K = -2.2

Duration contrast ruleIn

ter o

nset

I nte

rval d

evia

t ion

s (

%)


Analysis-by-Measurement Designing Articulation Rules

3 main classes of articulation:

• Legato (overlapping)

• Staccato (detaching)

• Repetition




Legato and Staccato Tones

IOI = Inter-onset-Interval

DR = Tone Duration

KOT = Key Overlap Time

KDT = Key Detached Time


Data

• Natural• Glittering• Dark• Heavy• Light• Hard• Soft• Passionate• Flat

5 pianists played the

same score

9 times on a

DisklavierThe first sixteen bars of the

Andante movement of W A

Mozart’s Piano Sonata in G

major, K 545

1 pianist played 13 Mozart piano sonatas

on a computer-monitored Bösendorfer


Mean Legato (KOR)

0

5

10

15

20

25

Fla

t

Da

rk

So

ft

Lig

ht

Na

tura

l

Glit

teri

ng

Pa

ssio

na

te

Ha

rd

He

avy

Adjective

Me

an

KO

R (

%)

0

5

10

15

20

25

P1 P2 P3 P4 P5Pianist

Me

an

KO

R (

%)

Legato articulation rule


Mean Staccato(Key Detached Ratio, KDR)

0

25

50

75

100

Heavy

Passio

nate

Dark

Soft

Fla

t

Hard

Natu

ral

Glit

tering

Lig

ht

Adjective

Mean

KD

R (

%)

0

25

50

75

100

Na

tura

l

Glit

teri

ng

Da

rk

He

avy

Lig

ht

Ha

rd

So

ft

Pa

ssio

na

te

Fla

t

Adjective

Mea

n K

DR

(%

)

P1 P2 P3 P4 P5

Staccato articulation rule

19


Context Influence in Staccato Production

Amount of staccato (KDR) in different contexts for the 2nd note in a three notes pattern.

(S = staccato note, N = Non–staccato note)

NSN

0

10

20

30

40

50

60

-100 -75 -50 -25 0 25 50 75 100

KDR (%)

Fre

qu

en

cy (

N)

Previous KDR

Current KDR

Next KDR

NSS

0

10

20

30

40

50

60

-100 -75 -50 -25 0 25 50 75 100

KDR (%)

Fre

qu

en

cy (

N) Previous KDR

Current KDR

Next KDR

SSS

0

10

20

30

40

50

60

-100 -75 -50 -25 0 25 50 75 100

KDR (%)

Fre

qu

en

cy (

N)

Previous KDR

Current KDR

Next KDR

SSN

0

10

20

30

40

50

60

-100 -75 -50 -25 0 25 50 75 100

KDR (%)

Fre

qu

en

cy (

N)

Previous KDR

Current KDR

Next KDR

64

66

68

70

72

74

76

SSS NSS SSN NSN

Staccato context

KD

R (

%)


Legato and walking

Staccato and running

Step and key overlap time

-50

0

50

100

150

200

250

300

350

400

0 500 1000 1500 2000

Tc/2 (ms), IOI

Td

su (

ms)

, KO

T

Normal step frequency

High step frequency

Low step frequency

Bresin & Battel

MacKenzie and Van Eerd

Repp

Step and key detached time

050

100150200250300350400450

0 200 400 600

Tc/2 (ms), IOI

Tai

r (m

s), K

DT

Running

Natural

Heavy

Adagio

Allegro

Presto

Default mezzostaccato

Control model for

step sounds

LegatoLegato and and SStaccatotaccato AAllude to llude to WWalking and alking and RRunning?unning?


WalkingWalking RunningRunning

Footsteps


Controlling Footsteps

Pd model for crumpling sounds controlled with performance rules




KTH Performance Rules

• Descriptions of different performance principles used by

musicians

• General applicability

• K values change the overall quantity of each rule

• Context dependency

• ~ 30 rules

• 30 years of research at KTH

ScoreScore RulesRules PerformancePerformance

K valuesK values




Director MusicesA program for modelling music performancehttp://www.speech.kth.se/music/performance




Performance Rules

Phrasing Phrase archFinal

ritardandoPunctuationHigh loud

Harmonic/melodic tension Harmonic/melodic charge

Repetitive patterns and grooves Swing

Articulation PunctuationStaccato/legato

Accents Accent rule

Ensemble timing Ensemble swingMelodic sync




Phrase Arch Rule

Dead-pan

Exaggerated

ΔIO

I (%

)


Moderate

Inverted

Phrase Arch Rule Δ

IOI

(%)


The DM system (~30 rules)

Differentiation RulesExample: Duration Contrast Rule

Grouping RulesExample: Phrase Articulation Rule

Synchronization/Ensemble RulesExample: Ensemble timing

Other RulesExample: Repetition Articulation Rule




Part II

Emotion in Music Performance


HappyHappy oror sadsad musicmusic??


Mapping from Emotional Expression to Rule Parameters

For each emotion:

Select a palette of rule parameters according to previous findings

Mapping

Emotional expression

Rule parameters




Cues for the simulation of

emotions in music

performance

(by A.Gabrielsson and P.Juslin,

Psychology of Music, 1996, vol. 24)

• No expression

• Tenderness

• Solemnity

• Happiness

• Sadness

• Anger

• Fear

Synthesis of Emotional ExpressionThe musicianMusical communicationModelling music performanceEmotional colouringApplications



• TENDERNESSslow mean tempo (Ga96)slow tone attacks (Ga96)low sound level (Ga96)small sound level variability (Ga96)legato articulation (Ga96)soft timbre (Ga96)large timing variations (Ga96)accents on stable notes (Li99)soft duration contrasts (Ga96)final ritardando (Ga96)

• HAPPINESSfast mean tempo (Ga95)small tempo variability (Ju99)staccato articulation (Ju99)large articulation variability (Ju99)high sound level (Ju00)little sound level variability (Ju99)bright timbre (Ga96)fast tone attacks (Ko76)small timing variations (Ju/La00)sharp duration contrasts (Ga96)rising micro-intonation (Ra96)

• ANGERhigh sound level (Ju00)sharp timbre (Ju00)spectral noise (Ga96)fast mean tempo (Ju97a)small tempo variability (Ju99)staccato articulation (Ju99)abrupt tone attacks (Ko76)sharp duration contrasts (Ga96)accents on unstable notes (Li99)large vibrato extent (Oh96b)no ritardando (Ga96)

• SADNESSslow mean tempo (Ga95)legato articulation (Ju97a)small articulation variability (Ju99)low sound level (Ju00)dull timbre (Ju00)large timing variations (Ga96)soft duration contrasts (Ga96)slow tone attacks (Ko76)flat micro-intonation (Ba97)slow vibrato (Ko00)final ritardando (Ga96)

• FEARstaccato articulation (Ju97a)very low sound level (Ju00)large sound level variability (Ju99)fast mean tempo (Ju99)large tempo variability (Ju99)large timing variations (Ga96)soft spectrum (Ju00)sharp micro-intonation (Oh96b)fast, shallow, irregular vibrato (Ko00)

Positive Valence

Negative Valence

High ActivityLow Activity

From Juslin (2001)


Tempo

Loudness

Timbre

Encoding Decoding

Cue Utilization Cue Utilization

The Listener

Articula.

The Performance The Performer intention expressive cues judgment

Accuracy

others

Anger Anger

rPerformer rListener

Matching

.87

.26 .47 .63

-.26

.22 .55 .61

-.39

.92

Lens model: quantifies the expressive communication between performer and listener


Example: SADNESS

Expressive Cue Analysis Synthesis (Director Musices)

Tempo Slow Tone IOI is lengthened by 30%

Sound level Moderate or low Sound level is decreased by 6 dB

Articulation Legato Tone duration = Tone IOI

Time deviations Moderate Duration Contrast Rule (k = -2)

Phrase Arch Rule applied on phrase level (k = 1.5)

Phrase Arch Rule applied on sub-phrase level (k = 1.5)

Final ritardando Yes Obtained from the Phrase Arch Rule




IOI deviations

dB deviations

articulation

Example: SADNESSModel Score


Synthesis of Emotion: Listening Test Results

Percentage of “right” classification (%)

0102030405060708090

100

FE

AR

AN

GE

R

HA

PP

INE

SS

SA

DN

ES

S

SO

LE

MN

ITY

TE

ND

ER

NE

SS

NO

EX

PR

ES

SIO

N

Ekorrn

Mazurka




ConclusionsConclusions

Emotional expressioncan be derived directlyfrom the music score,

simplyby enhancing music structure




Better Monophonic Ringtones!

Today (nom.)

Natural

:-) Happy

>:-( Angry

:-( Sad

=|:-| Solemn

Mozart G minor

0 2 4 6 8 10 12 -1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

0 2 4 6 8 10 12 -1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8 0 2 4 6 8 10 12 -1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

0 2 4 6 8 10 12 -1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

0 2 4 6 8 10 12 -1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

0 2 4 6 8 10 12 -1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

Dead-pan

Happy

Angry

Natural

Sad

Solemn

www.notesenses.com




Usher, Burn

MechanicalMusicalHappy

Jennifer Ellison, Bye Bye Boy

MechanicalMusicalRomantic

Better Polyphonic Ringtones!

www.notesenses.com




pDM –performance rules in real-timeAnders Friberg + MEGA project (IST EU)




Real-time Visualization of

Musical Expression


Background

Feel-Me project

Design a computer program for teaching students to play expressively

The system includes a tool for automatic extraction of acoustic cues (CUEX):

pitch, duration, sound level, articulation, vibrato, attack velocity, spectrum




Aim

Design a tool for real-time visual feedback to expressive performance

Mapping of acoustic cues:

– Non-verbal– Intuitive– Informative (including emotional expression)

Previous studies: cross-modality speeds stimuli discrimination




Cue analysis

Expression mapper

Audio Emotion

TempoSound level

Articulation...

Implementations

CUEX

Simplified real-timeversion

Mult. Regression

Fuzzy inspired

Recognition of EmotionThe musicianMusical communicationModelling music performanceEmotional colouringApplications

The ListenerRecognition of emotionVisualisation of musical expression


Experiment

2 melodies, Brahms (minor) & Haydn (Major)3 instruments (piano, guitar, saxophone)12 performances per instrument (12 emotional intentions)

24 colour nuances8 levels of hue 2 levels of brightness 2 levels of saturation

2 groups of 11 subjects each(1 group per melody)




Experiment: main resultsHUE

Happiness YellowFear BlueSadness Violet & BlueAnger RedLove Blue & Violet

BRIGHTNESS

Observed tendency:Minor tonality Low brightness (Dark colours)Major tonality High brightness (Light colours)

Interaction involving sadness:Even for major tonality low brightness is preferred for all instruments




Experiment: main resultsSimilar colour palettes within instruments

DISGUST (SAX)

0

0,5

1

1,5

2

2,5

3

3,5

4

red

red

_d

ark

red

_lig

ht

ora

ng

e

ora

ng

e_

da

rk

ora

ng

e_

ligh

t

yello

w

yello

w_

da

rk

yello

w_

ligh

t

gre

en

gre

en

_d

ark

gre

en

_lig

ht

cya

n

cya

n_

da

rk

cya

n_

ligh

t

blu

e

blu

e_

da

rk

blu

e_

ligh

t

vio

let

vio

let_

da

rk

vio

let_

ligh

t

ma

ge

nta

ma

ge

nta

_d

ark

ma

ge

nta

_lig

ht

Haydn

Brahms




Experiment: main resultsSHAME (SAX)

0

1

2

3

4

5re

d

red

_d

ark

red

_lig

ht

ora

ng

e

ora

ng

e_

da

rk

ora

ng

e_

ligh

t

yello

w

yello

w_

da

rk

yello

w_

ligh

t

gre

en

gre

en

_d

ark

gre

en

_lig

ht

cya

n

cya

n_

da

rk

cya

n_

ligh

t

blu

e

blu

e_

da

rk

blu

e_

ligh

t

vio

let

vio

let_

da

rk

vio

let_

ligh

t

ma

ge

nta

ma

ge

nta

_d

ark

ma

ge

nta

_lig

ht

Haydn

Brahms

SHAME (GUITAR)

0

1

2

3

4

5

red

red

_d

ark

red

_lig

ht

ora

ng

e

ora

ng

e_

da

rk

ora

ng

e_

ligh

t

yello

w

yello

w_

da

rk

yello

w_

ligh

t

gre

en

gre

en

_d

ark

gre

en

_lig

ht

cya

n

cya

n_

da

rk

cya

n_

ligh

t

blu

e

blu

e_

da

rk

blu

e_

ligh

t

vio

let

vio

let_

da

rk

vio

let_

ligh

t

ma

ge

nta

ma

ge

nta

_d

ark

ma

ge

nta

_lig

ht

Haydn

Brahms

SHAME (PIANO)

0

1

2

3

4

5

red

red

_d

ark

red

_lig

ht

ora

ng

e

ora

ng

e_

da

rk

ora

ng

e_

ligh

t

yello

w

yello

w_

da

rk

yello

w_

ligh

t

gre

en

gre

en

_d

ark

gre

en

_lig

ht

cya

n

cya

n_

da

rk

cya

n_

ligh

t

blu

e

blu

e_

da

rk

blu

e_

ligh

t

vio

let

vio

let_

da

rk

vio

let_

ligh

t

ma

ge

nta

ma

ge

nta

_d

ark

ma

ge

nta

_lig

ht

Haydn

Brahms


The ExpressiBall Expressive performance space as a mapping of acoustical cues and emotions

X Tempo Color EmotionY Sound level Shape Articulation Z Attack velocity & Spectrum energy

Lo

ud

So

ft

Slow Fast

StaccatoAngryFast attackHigh energy

Lo

ud

So

ftSlow Fast

LegatoSadSlow attackLow energy


The ExpressiBall

DEMO!




ExpressiBall:Current & Future Work…

Sonification of the ExpressiBall

Set-up depending on instrument/spectrumUse complex colour palettes (pictures?)

Usability test with students

Other possible applications: ”Colour Monitor” in discotheques Computer screen saver …




OutlookAim

To explain how it is possible to communicate different emotions with the same music score

Part I: The science of music performanceAnalysis & synthesis of music performance• The most important techniques for measuring and modelling a

performance• The acoustical cues of importance for communicating expressivity• How the use of acoustical cues can influence the performance style• A rule-based system for the synthesis of music performance

Part II: Emotion in music performance• Emotionally expressive music performance• Real-time Visualization of Musical Expression • Examples• Applications

– Emotional colouring of music performance– expressive ringtones in mobile phones – visual display of emotion in music performance


The End

1 roberto bresin - an introduction to affective music – 2004.08.30 an introduction to affective...

Documents

affective music

musical performance

automatic music performance

music emotions

performance ioi

music score midi

performance style

communication slide