C@MERATA at MediaEval 2014 - Extracting Answer Passages from

Classical Music Scores using Natural Language Descriptions

Richard Sutcliffe, University of Essex Tim Crawford, Goldsmiths, University of London

Chris Fox, University of Essex Eduard Hovy, Carnegie-Mellon University

Deane Root, University of Pittsburgh

2

Outline Background

Task Design

Passages and Divisions

Query Types and Music Scores

Gold Standard

Evaluation Metrics

XML Formats

Baseline System

Campaign this Year

Results

Conclusions

3

C@merata stands for: Cl@ssical Music Extraction of Relevant Aspects by Text Analysis

4

C@merata Task There is a series of questions with required answers. Provided Question:

• A short noun phrase in English referring to musical features in a score,

• A short classical music score in MusicXML. Required Answer:

• The location(s) in the score of the requested musical feature. We call each such location a passage.

5

Motivation When studying musicological analyses of works of western classical art music there are frequent references to relevant passages in the printed score. Our work if successful could enable such references to be associated with passages automatically, hence facilitating the scholarly work of musicologists and musicians, both professional and amateur. It could also enable passages to be found interactively on the basis of an input natural language phrase.

6

Organisers Richard Sutcliffe, University of Essex (has worked on CLEF QA evaluations for ten years). Tim Crawford, Goldsmiths, University of London (lutenist with the Parley of Instruments, musicologist specialising in electronic editions of works by Henry Purcell, Silvius Leopold Weiss etc). Eduard Hovy, Carnegie-Mellon University (has worked extensively on NLP and QA at TREC, MUC, TIPSTER, CLEF, Machine Reading etc). Deane Root, University of Pittsburgh (Editor-in-Chief of Grove Online - a comprehensive source of information on western classical music) Chris Fox, University of Essex (has worked on NLP for many years, also has a thorough knowledge of music theory).

7

Background Originally there was Information Retrieval along with the SIGIR conference, from 1978 onwards. Interest developed in Music Information Retrieval, leading to the ISMIR conference, from 2000 onwards. TREC IR evaluations began in 1992. CLEF multi-lingual and cross-lingual evaluations began in 2000. MIREX evaluations began in 2005. They are run by the University of Illinois at Urbana-Champaign. Focus on low-level tasks like melodic similarity, mostly with audio data.

8

Background cont. Question Answering started at CLEF in 2003. In 2011, QA4MRE task started. 12 documents on 3 topics (Aids, Climate Change, Music and Society), ten MCQ questions on each. Used TED film transcripts. In 2012, QA4MRE had same topics and used various sources for the music topic, including 1911 Encyclopaedia Britanica and Wikipedia. In 2013, QA4MRE used articles from Grove Online for the music topic: ‘Johann Baptist Cramer’, ‘Electronic Dance Music’, ‘Film Music - Hollywood’, ‘Disciplines of Musicology - Analytic Traditions’.

9

Task Design There are 200 queries. We saw earlier that we have: Provided Question:

• A short noun phrase in English referring to musical features in a score,

• A short classical music score in MusicXML. Required Answer:

• The location(s) in the score of the requested musical feature. We call each such location a passage.

10

Example Query

t: pitch_and_length

q: semiquaver G#

s: corelli_allegro_tr_clementi.xml

[ 4/4, 4, 2:5-2:5 ]

[ 4/4, 4, 5:15-5:15 ]

[ 4/4, 4, 7:3-7:3 ]

...

11

Specifying Passages in a Score We specify a passage as follows:

• a time signature, • a divisions value • a start bar and beat • an end bar and beat Example:

[4/4,1,1:1-2:4] time signature is 4/4. divisions value is 1 so we count in crotchets (quarter notes) Passage starts in bar 1 before the first crotchet (i.e. 1:1) Passage ends in bar two after the fourth crotchet (i.e. 2:4) Thus passage consists of the two complete bars (measures) numbered one and two.

12

What is a Divisions Value? The number of beats into which we divide a crotchet. A suitable value depends on what we wish to demarcate. Working in whole crotchets: Divisions = 1

Working in quavers: Divisions = 2

Working in quaver triplets: Divisions = 3

Working in semiquavers or quavers or triplet quavers: Divisions = 12 !!

With Divisions = 12:

one crotchet = 12 beats

one quaver = 6 beats

one quaver triplet = 4 beats

one semiquaver = 3 beats

Note: MusicXML uses the divisions concept and we got the idea from there.

13

Query Types In C@merata there are twelve types of query:

Type No Example

simple_pitch 30 G5 simple_length 30 dotted quarter note

pitch_and_length 30 D# crotchet perf_spec 10 D sharp trill stave_spec 20 D4 in the right hand word_spec 5 word "Se" on an A flat

followed_by 30 crotchet followed by semibreve melodic_interval 19 melodic octave

harmonic_interval 11 harmonic major sixth cadence_spec 5 perfect cadence

triad_spec 5 tonic triad texture_spec 5 polyphony

All 200

14

MusicXML Scores

Work Staves Scoring Lang

bach_cello_suite_1_bwv1007_prelude 1 vc Amer. bach_chorale_24835b3 4 SATB Eng.

bach_chorale_507b 4 SATB Amer. bach_minuet_in_g_bwv_anh114 2 hpd Amer.

carissimi_o_felix_anima 3 SAB Eng. charpentier_te_deum_preludium 5 2 vn, va, 2 vc Amer.

corelli_allegro_tr_clementi 2 hpd Eng. cutting_galliard_11 1 lute Eng.

dowland_earl_of_essex_measure 1 A Amer. lassus_psalm_50 3 SAB Eng.

lully_andante 3 2 vn, vc Amer. monteverdi_lasciatemi_morire 5 SSATB Amer. purcell_fie_nay_prithee_zd10 3 SSA Eng.

scarlatti_a_se_florindo 3 S, hpd Amer. scarlatti_k466 2 hpd Eng.

tallis_all_praise_to_thee 1 A Eng. telemann_taenzchen 2 hpd Amer.

telemann_twv33_21_tres_vite 2 hpd Eng. vivaldi_concerto_rv299_largo 3 vc, hpd Eng.

weiss_sonata_34_prelude 2 lute Amer.

15

Source of MusicXML Scores Public domain scores were obtained from sites such as musescore.com. Most of these are transcribed by amateurs. MusicXML is an interchange format. Thus, most score-writing programs e.g. Finale or Sibelius can export to MusicXML (and input from it).

16

Distribution of Scores by Number of Staves Staves Frequency

1 4 2 6 3 6 4 2 5 2

17

Example Query 1 (from earlier)

t: pitch_and_length

q: semiquaver G#

s: corelli_allegro_tr_clementi.xml

[ 4/4, 4, 2:5-2:5 ]

[ 4/4, 4, 5:15-5:15 ]

[ 4/4, 4, 7:3-7:3 ]

...

18

Example Query 2

t: followed_by

q: G sharp followed by B

s: corelli_allegro_tr_clementi.xml

[ 4/4, 4, 2:5-2:6 ]

[ 4/4, 4, 5:15-5:16 ]

[ 4/4, 4, 8:11-8:12 ]

...

19

Example Query 3

t: melodic_interval

q: rising melodic octave

s: corelli_allegro_tr_clementi.xml

[ 4/4, 4, 1:1-1:8 ]

[ 4/4, 4, 4:1-4:8 ]

[ 4/4, 4, 4:9-4:16 ]

...

20

Example Query 4

t: triad_spec

q: tonic triad

s: bach_chorale_507b.xml

[ 4/4, 1, 0:1-0:1 ]

[ 4/4, 1, 7:1-7-1 ]

21

Creation of the Gold Standard of 200 Questions Stage 1

20 Baroque scores were collected. 4 on 1 stave, 6 on 2 staves etc. 10 questions set on each score. 10 scores: English terminology, 10 scores: American terminology. Mix of question types set against each score. For each question, answers were found in the score. Gradually built up the required question-type distribution. Questions were recorded in C@merata ascii format.

22

Creation of the Gold Standard (cont.) Stage 2 Answers from Stage 1 were carefully checked. (Precision was good, Recall was sometimes lacking.) Stage 3

Questions in C@merata ascii format automatically converted into C@merata XML format. Stage 4

Test questions automatically extracted from Gold Standard (minus the answers).

23

Evaluation of a Returned Passage Passage is beat correct if it starts at exactly the correct beat (as specified by the divisions value) in the correct start bar and also ends at the correct beat in the end bar. - Useful for applications of results which are themselves automatic. Passage is measure correct if it starts in the bar where the requested feature starts and ends in the bar where the requested feature ends. - Sufficient in many cases for humans to use.

24

Example Query 1: Beat correct vs. Measure Correct

t: pitch_and_length

q: semiquaver G#

s: corelli_allegro_tr_clementi.xml

[ 4/4, 4, 2:5-2:5 ]

[ 4/4, 4, 5:15-5:15 ]

[ 4/4, 4, 7:3-7:3 ]

...

25

Beat Precision (BP) is the number of beat correct passages returned by a system divided by the number of passages (correct or incorrect) returned. Beat Recall (BR) is the number of beat correct passages returned by a system divided by the total number of answer passages known to exist. Measure Precision (MP) is the number of measure correct passages returned by a system divided by the number of passages (correct or incorrect) returned. Measure Recall (MR) is the number of measure correct passages returned by a system divided by the total number of answer passages known to exist.

26

XML Question Format <questions task="camerata" year="2014" task_no="1"

organisation="University of Essex"

group="LAC Group">

<question number="001" music_file="f01.xml" divisions="4">

<text>harmonic perfect fifth</text>

<answer>

</answer>

</question>

</questions>

27

XML Answer Format (for Runs)

<questions task="camerata" year="2014" task_no="1"

runtag="lacg01" organisation="University of Essex"

group="LAC Group">

<question number="001" music_file="f01.xml" divisions="4">

<text>harmonic perfect fifth</text>

<answer>

<passage start_beats="4" start_beat_type="4"

end_beats="4" end_beat_type="4"

start_divisions="4" end_divisions="4"

start_bar="2" start_offset="1"

end_bar="2" end_offset="4" />

<passage start_beats="4" start_beat_type="4"

end_beats="4" end_beat_type="4"

start_divisions="4" end_divisions="4"

start_bar="7" start_offset="5"

end_bar="7" end_offset="8" />

<passage start_beats="4" start_beat_type="4"

end_beats="4" end_beat_type="4"

start_divisions="4" end_divisions="4"

start_bar="9" start_offset=1"

end_bar="9" end_offset=8" />

</answer>

</question>

</questions>

28

Baseline System To encourage participation we created a baseline C@merata system Written in Python 27 Uses excellent Music21 of Michael Scott Cuthbert (Will convert MusicXML into a tractable form and manipulate it) Handles all the basic input of questions, processing and output Shows how to use Music21 Actual question-answering component is skeletal

29

Architecture of Baseline System 1. Analyse the input xml file and extract questions and score files 2. For each question:

• Parse question using Stanford Parser • Determine question type (similar to normal QA) • Parse the score file using Music21 • Based on question type, search for answer passages in score 3. Write out answers to output xml file

30

Campaign this Year Fri 24.03.14 Participant registration Fri 02.05.14 Release of training data Mon 16.06.14 - Fri 20.06.14 Systems answer test questions Fri 27.06.14 Publication of results Thu 16.10.14 - Fri 17.10.14 MediaEval Workshop in Barcelona

31

Participants Runtag Leader Affiliation Country

CLAS Stephen Wan CSIRO Australia DMUN Tom Collins De Montfort University England OMDN Donncha Ó Maidín University of Limerick Ireland TCSL Nikhil Kini Tata Consultancy Services India UNLP Kartik Asooja NUI Galway Ireland

32

Results

Run BP BR BF MP MR MF

CLAS01 0.713 0.904 0.797 0.764 0.967 0.854

DMUN01 0.372 0.712 0.489 0.409 0.784 0.538

DMUN02 0.380 0.748 0.504 0.417 0.820 0.553

DMUN03 0.440 0.868 0.584 0.462 0.910 0.613

LACG01 0.135 0.101 0.116 0.188 0.142 0.162

OMDN01 0.415 0.150 0.220 0.424 0.154 0.226

TCSL01 0.633 0.821 0.715 0.652 0.845 0.736

UNLP01 0.113 0.516 0.185 0.155 0.703 0.254

UNLP02 0.290 0.512 0.370 0.393 0.692 0.501

Maximum 0.713 0.904 0.797 0.764 0.967 0.854

Minimum 0.113 0.150 0.185 0.155 0.154 0.226

Average 0.420 0.654 0.483 0.460 0.734 0.534

33

Summary of All Results Based on Beat F-score (BF) ranking is: CLAS01 (0.797), TCSL01 -(0.715), DMUN03 (0.584), UNLP02 (0.370), OMDN01 (0.220), LACG01 (0.116) All submitted runs beat the baseline Best scores were quite high, so task was perhaps quite easy

34

Results by Question Type

Type BP BR BF MP MR MF

simple_pitch 0.645 0.736 0.677 0.685 0.787 0.720

simple_length 0.780 0.846 0.810 0.830 0.906 0.864

pitch_and_length 0.662 0.726 0.644 0.719 0.803 0.710

perf_spec 0.339 0.547 0.339 0.350 0.582 0.352

stave_spec 0.408 0.682 0.508 0.432 0.732 0.540

word_spec 0.487 0.771 0.520 0.487 0.771 0.520

followed_by 0.291 0.518 0.278 0.351 0.716 0.355

melodic_interval 0.396 0.417 0.402 0.471 0.501 0.481

harmonic_interval 0.185 0.207 0.188 0.269 0.329 0.281

cadence_spec 0.071 0.141 0.093 0.171 0.297 0.214

triad_spec 0.081 0.125 0.095 0.124 0.171 0.138

texture_spec 0.060 0.109 0.075 0.072 0.141 0.092

35

Summary of Question Type Results Ranking by question type using BF: simple_length (0.810), simple_pitch (0.677), pitch_and_length (0.644), word_spec (0.520), stave_spec (0.508), melodic_interval (0.402), perf_spec (0.339), followed_by (0.278), harmonic_interval (0.188), triad_spec (0.095), cadence_spec (0.093), texture_spec (0.072) Not all question types are equally easy First few types in the list are quite easy Last three or four types in the list are very difficult. melodic_interval easier than harmonic_interval

36

Conclusions Results are generally high So task was rather easy Some question types were easier than others Technical aspects (data formats, passages, evaluation etc) worked out well Need more complex queries for a second campaign Participants are interested in continuing

37

The End

38

39

Example Questions The following slides show some example scores and questions against them.

40

Work: J.S. Bach, Das Wohltemperierte Klavier, Book 1, Prelude No. 2 in C minor BWV 847 Extract:

41

Q: passage in common time A: [4/4,1,1:1-4:4] Q: interval of a melodic sixth A: [4/4,4,1:1-1:2], [4/4,4,1:8-1:9] [4/4,4,1:9-1:10], [4/4,4,2:1-2:2], [4/4,4,2:8-2:9], [4/4,4,2:9-2:10], [4/4,4,3:1-3:2], [4/4,4,3:8-3:9], [4/4,4,3:9-3:10] Q: second A: None, because a second is harmonic by default Q: C followed by Eb A: [4/4,4,1:5-1:6], [4/4,4,1:13-1:14], [4/4,4,4:1-4:2], [4/4,4,4:9-4:10] Q: C followed by Eb in the bass clef A: [4/4,4,4:1-4:2], [4/4,4,4:9-4:10] Q: semiquaver E natural / sixteenth note E natural A: [4/4,4,2:3-2:4]

42

Work: J.S. Bach, Suite No. 3 in C Major for Cello, BWV 1009, Sarabande Extract:

Q: harmonic interval of a minor third A: [3/4,2,208:1-208:1] Q: minor third A: [3/4,2,208:1-208:1] (thirds are harmonic by default) Q: dotted quaver / dotted eighth note A: [3/4,4,206:5-206:7], [3/4,4,207:5-207:7], [3/4,4,208:5-208:7] Q: harmonic perfect fifth A: [3/4,1,206:1-206:1], [3/4,1,207:1-207:1], [3/4,4,208:5-208:7] (in last passage, C and G should be dotted but were not in the original) Q: simultaneous harmonic perfect fifth and harmonic eleventh A: [3/4,4,208:5-208:7] (same point, C and G are assumed dotted in this example)

43

Work: J. Dowland, King of Denmark's Galliard, P 40 Extract:

Q: perfect cadence

A: [3/4,1,3:3-4:3] (We are assuming the cadence continues until the end of the bar) Q: four consecutive quavers / four consecutive quarter notes A: [3/4,2,2:3-2:6] Q: dotted minim in the bass / dotted half note in the bass A: [3/4,1,2:1-2:3] Q: harmonic fourth A: [3/4,1,1:1-1:1], [3/4,1,1:2-1:3], [3/4,1,4:1-4:3] (Note that there are two instances in bar (measure) 1 because the chord is played twice)

44

Work: J. Dowland, Pauana Dulandi, P 86 Extract:

Q: consecutive 5ths in the bass A: [4/4,1,2:1-2:4] Q: semibreve Bb in the treble clef / whole note Bb in the treble clef A: [4/4,1,1:1-1:4], [4/4,1,4:1-4:4] Q: octave followed two bars later by another octave

A: [4/4,1,1:1-3:4] (assumed harmonic) Q: Vc triad A: [4/4,1,2:3-2:3]

45

Work: G.F. Handel, Messiah, HG xlv, "And the Glory of the Lord" Extract:

Q: crotchet rest / quarter note rest A: [3/4,1,1:1-1:1] Q: dotted crotchet followed by three quavers / dotted quarter note followed by three eighth

notes A: [3/4,1,5:1-5:3], [3/4,1,6:1-6:3] Q: four hemidemisemiquavers / four sixty-fourth notes A: [3/4,4,3:9-3:9] Q: three quavers in a row / three eighth notes in a row A: [3/4,2,5:4-5:6], [3/4,2,6:4-6:6]

46

Work: G.F. Handel, Messiah, HG xlv, Overture Extract:

Q: D sharp crotchet / D sharp quarter note

A: [4/4,1,22:1-22:1], [4/4,1,25:1-25:1] Q: D natural quaver / D natural eighth note

A: [4/4,2,23:3-23:3], [4/4,2,24:4-24:4] Q: open VII triad in the first inversion A: [4/4,1,23:3-23:3] Q: melodic fourth in the bass clef A: [4/4,1,24:1-24:2]

47

Work: D. Scarlatti, Keyboard Sonata in D minor, K 1 Extract:

48

Q: eight staccato notes in succession A: [4/4,1,17:1-17:4] Q: augmented melodic fourth A: [4/4,2,19:3-19:4], [4/4,2,20:4-20:5] Q: third

A: [4/4,4,17:13-17:13], [4/4,2,19:2-19:2], [4/4,2,19:4-19:4], [4/4,4,19:9-19:9], [4/4,4,20:9-20:9] (these are assumed to be harmonic and can thus be across parts) Q: a quaver, then a major third / an eighth note, then major third

A: [4/4,2,19:1-19:2], [4/4,2,19:3-19:4] (third assumed to be harmonic and to follow the quaver immediately) Q: change from bass to treble clef A: [4/4,1,17:2p] (recall that this means the clef change is immediately after 17:2)

49

Work: D. Scarlatti, Keyboard Sonata in D major, K 430 Extract:

Q: three fourths

A: [3/8,2,56:1-56:3] (assumed consecutive) Q: treble clef F natural A: [3/8,2,62:2-62:2] Q: melodic octave

A: [3/8,4,57:1-57:2], [3/8,4,57:2-57:6] (melodic must be stated, otherwise it is harmonic) Q: harmonic 5th followed by harmonic 4th A: [3/8,2,58:1-58:2], [3/8,2,62:1-62:2]

50

Architecture of Baseline System We have built a very basic system to perform the task: 1. Analyse the input xml file and extract questions and score files 2. For each question:

• Parse question using Stanford Parser • Determine question type (similar to normal QA) • Parse the score file using Music21 • Based on question type, search for answer passages in score 3. Write out answers to output xml file

51

Linguistic Observations Queries are noun phrases Head noun group is the main feature (e.g. F#) PP modifiers qualify this (e.g. in the bass clef) A lot of terminology is used (e.g. semiquaver, soprano part, forte bar) Search based on query classification will work at least for simple examples.

52

Summary We are trying to find ways to link natural language descriptions of music to musical scores. Starting with the C@merata task, we are working with very simple tasks to develop the technology. Then we will progress to more complex tasks which are of genuine interest to musicologists. In due course, we aim to tie passages in musicological texts such as in Grove Online to the corresponding music scores.

53

Music Extracts symphony opening with a horn call over shimering strings Bruckner 4th Symphony http://www.youtube.com/watch?v=J8t1TzN0RRY Start: 0:00 End: 0:44 symphony closing with six unison chords Sibelius 5th Symphony http://www.youtube.com/watch?v=nkzrSZKA4cM Start: 9:36 End: 9:56 Nielsen 4th Symphony, Finale (shows two timpanists well) http://www.youtube.com/watch?v=yXDe1hj4HBo Start: 32:10 End: 32:48

54

Extracts Not Used symphony opening with two octave descending arpeggio Beethoven 9th Symphony string quartet opening including double stopping on all four instruments Beethoven op 127 symphony featuring a battle between two timpanists Nielsen 4th Symphony, Finale (not so good) http://www.youtube.com/watch?v=sD9I-UiYfW8 Start: 6:30 End: 7:08

55

Grove’s Dictionary of Music and Musicians • 1879-1889 George Grove, civil engineer, music administrator, writer and then Director of

Royal College of Music, wrote A Dictionary of Music and Musicians in four volumes. • 1904-910 Fuller Maitland edited the second edition - Grove's Dictionary of Music and

Musicians - in five volumes. • 1927 Henry Colles edited the third edition - in five volumes. • 1940 Henry Colles edited the fourth edition - in seven volumes. • 1954 Eric Blom edited the fifth edition - in nine volumes.

56

• 1980 Stanlie Sadie edited the sixth edition - The New Grove Dictionary of Music and

Musicians - in twenty volumes. Contained 22,500 articles and 16,500 biographies. • 2001 Stanlie Sadie edited the seventh edition - in 29 volumes. This was also available

online. • 2009 Deane Root was appointed editor of Grove Music Online. By this time it contained

more than 50,000 articles. Grove is now considered to be the most comprehensive and scholarly source of information on Western Classical Art Music which exists. It is used daily by musicians and musicologists worldwide.

57

What might our queries be like in future? We will show some examples:

• A natural language phrase describes a musical feature.

• A regular expression or other pattern could not capture it.

58

Some Examples Q: 'horn call over shimmering strings'

59

Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic"

60

Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra'

61

Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra' A: Jean Sibelius, Symphony No. 5

62

Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra' A: Jean Sibelius, Symphony No. 5 Q: 'symphony featuring a battle between two timpanists'

63

Some Examples Q: 'horn call over shimmering strings' A: Anton Bruckner, Symphony No. 4 "Romantic" Q: 'movement ending with six stacatto chords for full orchestra' A: Jean Sibelius, Symphony No. 5 Q: 'symphony featuring a battle between two timpanists' A: Carl Nielsen, Symphony No. 4 "The Inextinguishable"

64

What are the Characteristics of these Queries? Not very long Not that specific ('over', 'featuring') Use musical terms ('horn', 'strings', 'stacatto', 'chords', 'full orchestra', 'symphony', 'timpani') Also use non-musical terms, interpreted in a musical way ('shimmering', 'battle')

65

What are the Characteristics of these Queries? Not very long Not that specific ('over', 'featuring') Use musical terms ('horn', 'strings', 'stacatto', 'chords', 'full orchestra', 'symphony', 'timpani') Also use non-musical terms, interpreted in a musical way ('shimmering', 'battle') Nevertheless, experts can answer them!

66

Table 4. Results for simple_pitch Questions

Run BP BR BF MP MR MF

CLAS01 0.959 0.972 0.965 0.982 0.995 0.988

DMUN01 0.717 0.674 0.695 0.790 0.743 0.766

DMUN02 0.729 0.729 0.729 0.798 0.798 0.798

DMUN03 0.955 0.972 0.963 0.968 0.986 0.977

LACG01 0.000 0.000 0 0.200 0.028 0.049

OMDN01 0.000 0.000 0.000 0.000 0.000 0.000

TCSL01 0.959 0.963 0.961 0.982 0.986 0.984

UNLP01 0.422 0.789 0.550 0.478 0.894 0.623

UNLP02 0.422 0.789 0.550 0.478 0.894 0.623

Maximum 0.959 0.972 0.965 0.982 0.995 0.988

Minimum 0.000 0.000 0.000 0.000 0.000 0.000

Average 0.645 0.736 0.677 0.685 0.787 0.720

67

Table 5. Results for simple_length Questions

Run BP BR BF MP MR MF

CLAS01 0.904 0.988 0.944 0.915 1.000 0.956

DMUN01 0.858 0.852 0.855 0.879 0.874 0.876

DMUN02 0.863 0.889 0.876 0.884 0.911 0.897

DMUN03 0.955 0.985 0.970 0.967 0.997 0.982

LACG01 0.529 0.197 0.287 0.620 0.231 0.337

OMDN01 0.408 0.471 0.437 0.419 0.483 0.449

TCSL01 0.979 0.988 0.983 0.991 1.000 0.995

UNLP01 0.636 0.797 0.707 0.791 0.991 0.880

UNLP02 0.636 0.797 0.707 0.791 0.991 0.880

Maximum 0.979 0.988 0.983 0.991 1.000 0.995

Minimum 0.408 0.471 0.437 0.419 0.483 0.449

Average 0.780 0.846 0.810 0.830 0.906 0.864

68

Table 6. Results for pitch_and_length Questions

Run BP BR BF MP MR MF

CLAS01 0.860 0.937 0.897 0.895 0.975 0.933

DMUN01 0.653 0.785 0.713 0.721 0.867 0.787

DMUN02 0.663 0.823 0.734 0.730 0.905 0.808

DMUN03 0.760 0.943 0.842 0.770 0.956 0.853

LACG01 0.157 0.196 0.174 0.172 0.215 0.191

OMDN01 0.714 0.032 0.061 0.714 0.032 0.061

TCSL01 0.723 0.892 0.799 0.754 0.930 0.833

UNLP01 0.460 0.696 0.554 0.582 0.880 0.701

UNLP02 0.460 0.696 0.554 0.582 0.880 0.701

Maximum 0.86 0.943 0.897 0.895 0.975 0.933

Minimum 0.460 0.032 0.061 0.582 0.032 0.061

Average 0.662 0.726 0.644 0.719 0.803 0.710

69

Table 7. Results for perf_spec Questions

Run BP BR BF MP MR MF

CLAS01 1.000 0.862 0.926 1.000 0.862 0.926

DMUN01 0.407 0.379 0.393 0.444 0.414 0.428

DMUN02 0.407 0.379 0.393 0.444 0.414 0.428

DMUN03 0.741 0.690 0.715 0.741 0.690 0.715

LACG01 0.000 0.000 0.000 0.000 0.000 0.000

OMDN01 0.000 0.000 0.000 0.000 0.000 0.000

TCSL01 0.066 0.897 0.123 0.066 0.897 0.123

UNLP01 0.045 0.586 0.084 0.053 0.690 0.098

UNLP02 0.045 0.586 0.084 0.053 0.690 0.098

Maximum 1.000 0.897 0.926 1.000 0.897 0.926

Minimum 0.000 0.000 0.000 0.000 0.000 0.000

Average 0.339 0.547 0.339 0.350 0.582 0.352

70

Table 8. Results for stave_spec Questions

Run BP BR BF MP MR MF

CLAS01 0.568 1.000 0.724 0.568 1.000 0.724

DMUN01 0.534 0.840 0.653 0.568 0.893 0.694

DMUN02 0.534 0.840 0.653 0.568 0.893 0.694

DMUN03 0.619 0.973 0.757 0.619 0.973 0.757

LACG01 0.165 0.240 0.196 0.174 0.253 0.206

OMDN01 0.000 0.000 0.000 0.000 0.000 0.000

TCSL01 0.661 0.987 0.792 0.661 0.987 0.792

UNLP01 0.173 0.440 0.248 0.230 0.587 0.331

UNLP02 0.173 0.373 0.236 0.241 0.520 0.329

Maximum 0.661 1.000 0.792 0.661 1.000 0.792

Minimum 0.000 0.000 0.000 0.000 0.000 0.000

Average 0.408 0.682 0.508 0.432 0.732 0.540

71

Table 9. Results for word_spec Questions

Run BP BR BF MP MR MF

CLAS01 1.000 1.000 1.000 1.000 1.000 1.000

DMUN01 0.750 0.750 0.750 0.750 0.750 0.750

DMUN02 0.750 0.750 0.750 0.750 0.750 0.750

DMUN03 1.000 1.000 1.000 1.000 1.000 1.000

LACG01 0.044 0.250 0.075 0.058 0.333 0.099

OMDN01 0.000 0.000 0.000 0.000 0.000 0.000

TCSL01 0.261 1.000 0.414 0.261 1.000 0.414

UNLP01 0.067 0.833 0.124 0.067 0.833 0.124

UNLP02 0.067 0.833 0.124 0.067 0.833 0.124

Maximum 1.000 1.000 1.000 1.000 1.000 1.000

Minimum 0.000 0.000 0.000 0.000 0.000 0.000

Average 0.487 0.771 0.520 0.487 0.771 0.520

72

Table 10. Results for followed_by Questions

Run BP BR BF MP MR MF

CLAS01 0.748 0.859 0.800 0.830 0.953 0.887

DMUN01 0.090 0.797 0.162 0.093 0.820 0.167

DMUN02 0.092 0.820 0.165 0.094 0.844 0.169

DMUN03 0.094 0.844 0.169 0.096 0.859 0.173

LACG01 0.003 0.008 0.004 0.068 0.156 0.095

OMDN01 0.567 0.133 0.215 0.567 0.133 0.215

TCSL01 0.733 0.688 0.710 0.842 0.789 0.815

UNLP01 0.000 0.000 0.000 0.025 0.695 0.048

UNLP02 0.000 0.000 0.000 0.260 0.633 0.369

Maximum 0.748 0.859 0.800 0.842 0.953 0.887

Minimum 0.000 0.000 0.000 0.025 0.133 0.048

Average 0.291 0.518 0.278 0.351 0.716 0.355

73

Table 11. Results for melodic_interval Questions

Run BP BR BF MP MR MF

CLAS01 0.660 0.837 0.738 0.699 0.886 0.781

DMUN01 0.528 0.545 0.536 0.724 0.748 0.736

DMUN02 0.521 0.610 0.562 0.701 0.821 0.756

DMUN03 0.562 0.659 0.607 0.736 0.862 0.794

LACG01 0.000 0.000 0.000 0.158 0.024 0.042

OMDN01 0.000 0.000 0.000 0.000 0.000 0.000

TCSL01 0.894 0.683 0.774 0.904 0.691 0.783

UNLP01 0.000 0.000 0.000 0.000 0.000 0.000

UNLP02 0.000 0.000 0.000 0.000 0.000 0.000

Maximum 0.894 0.837 0.774 0.904 0.886 0.794

Minimum 0.000 0.000 0.000 0.000 0.000 0.000

Average 0.396 0.417 0.402 0.471 0.501 0.481

74

Table 12. Results for harmonic_interval Questions

Run BP BR BF MP MR MF

CLAS01 0.158 0.429 0.231 0.353 0.957 0.516

DMUN01 0.415 0.386 0.400 0.585 0.543 0.563

DMUN02 0.415 0.386 0.400 0.585 0.543 0.563

DMUN03 0.492 0.457 0.474 0.631 0.586 0.608

LACG01 0.091 0.014 0.024 0.273 0.043 0.074

OMDN01 0.000 0.000 0.000 0.000 0.000 0.000

TCSL01 0.000 0.000 0.000 0.000 0.000 0.000

UNLP01 0.000 0.000 0.000 0.000 0.000 0.000

UNLP02 0.000 0.000 0.000 0.000 0.000 0.000

Maximum 0.492 0.457 0.474 0.631 0.957 0.608

Minimum 0.000 0.000 0.000 0.000 0.000 0.000

Average 0.185 0.207 0.188 0.269 0.329 0.281

75

Table 13. Results for cadence_spec Questions

Run BP BR BF MP MR MF

CLAS01 0.238 0.625 0.345 0.286 0.750 0.414

DMUN01 0.083 0.125 0.100 0.333 0.500 0.400

DMUN02 0.083 0.125 0.100 0.333 0.500 0.400

DMUN03 0.167 0.250 0.200 0.417 0.625 0.500

LACG01 0.000 0.000 0.000 0.200 0.125 0.154

OMDN01 0.000 0.000 0.000 0.000 0.000 0.000

TCSL01 0.000 0.000 0.000 0.000 0.000 0.000

UNLP01 0.000 0.000 0.000 0.000 0.000 0.000

UNLP02 0.000 0.000 0.000 0.000 0.000 0.000

Maximum 0.238 0.625 0.345 0.417 0.75 0.500

Minimum 0.000 0.000 0.000 0.000 0.000 0.000

Average 0.071 0.141 0.093 0.171 0.297 0.214

76

Table 14. Results for triad_spec Questions

Run BP BR BF MP MR MF

CLAS01 0.348 0.727 0.471 0.391 0.818 0.529

DMUN01 0.100 0.091 0.095 0.200 0.182 0.191

DMUN02 0.100 0.091 0.095 0.200 0.182 0.191

DMUN03 0.100 0.091 0.095 0.200 0.182 0.191

LACG01 0.000 0.000 0.000 0.000 0.000 0.000

OMDN01 0.000 0.000 0.000 0.000 0.000 0.000

TCSL01 0.000 0.000 0.000 0.000 0.000 0.000

UNLP01 0.000 0.000 0.000 0.000 0.000 0.000

UNLP02 0.000 0.000 0.000 0.000 0.000 0.000

Maximum 0.348 0.727 0.471 0.391 0.818 0.529

Minimum 0.000 0.000 0.000 0.000 0.000 0.000

Average 0.081 0.125 0.095 0.124 0.171 0.138

77

Table 15. Results for texture_spec Questions

Run BP BR BF MP MR MF

CLAS01 0.182 0.500 0.267 0.273 0.750 0.400

DMUN01 0.100 0.125 0.111 0.100 0.125 0.111

DMUN02 0.100 0.125 0.111 0.100 0.125 0.111

DMUN03 0.100 0.125 0.111 0.100 0.125 0.111

LACG01 0.000 0.000 0.000 0.000 0.000 0.000

OMDN01 0.000 0.000 0.000 0.000 0.000 0.000

TCSL01 0.000 0.000 0.000 0.000 0.000 0.000

UNLP01 0.000 0.00 0.000 0.000 0.000 0.000

UNLP02 0.000 0.000 0.000 0.000 0.000 0.000

Maximum 0.182 0.500 0.267 0.273 0.750 0.400

Minimum 0.000 0.000 0.000 0.000 0.000 0.000

Average 0.060 0.109 0.075 0.072 0.141 0.092