Using String Kernels to Identify Famous

Performers from their Playing Style

Craig Saunders, David R. Hardoon, John Shawe-Taylor, and Gerhard Widmer

Presented by: Raghad Al-Awwad

OutlineOutline

1.1. IntroductionIntroduction2.2. Musical RepresentationMusical Representation

A A preformancepreformance alphabetalphabet3.3. String KernelString Kernel4.4. Partial Lease SquaresPartial Lease Squares5.5. Kernel PLS Kernel PLS 6.6. ExperimentExperiment7.7. Conclusion Conclusion

IntroductionIntroduction

ProblemProblem: : Try to identify famous pianist Try to identify famous pianist using only minimal information obtained using only minimal information obtained from audio recordings of their playingfrom audio recordings of their playing

Solution:Solution: A new way of applying String A new way of applying String Kernels. Kernels.

IntroductionIntroduction

Previous work on this data has compared a Previous work on this data has compared a varityvarity of ML techniques, while using as of ML techniques, while using as features statistical quantities obtained features statistical quantities obtained from the from the Performance wormPerformance worm

Introduction Introduction ––Technique (Technique (Step1Step1))

the the Performance WormPerformance Worm, is a technique , is a technique which plots a realwhich plots a real--time trajectory over 2D time trajectory over 2D space and is used to space and is used to analyseanalyse changes in changes in tempo and loudness at the beat level, and tempo and loudness at the beat level, and extract features for learning. extract features for learning.

Introduction Introduction ––Technique (Technique (Step2Step2))

From the worm trajectory it is possible to From the worm trajectory it is possible to obtain a set of cluster prototypes which obtain a set of cluster prototypes which capture certain characteristics over a small capture certain characteristics over a small time frametime frame……

These cluster prototypes form the These cluster prototypes form the ‘‘performance alphabetperformance alphabet’’, which captures , which captures some aspect of individual playing stylesome aspect of individual playing style

Introduction Introduction ––Technique (Technique (Step3Step3))

Once the alphabet is obtained, each Once the alphabet is obtained, each prototype can be assigned a symbol. This prototype can be assigned a symbol. This allows the audio recordings to be allows the audio recordings to be represented as strings. represented as strings.

Old

New

Musical Representation Musical Representation

The data:The data:Sonatas of W.A. Sonatas of W.A. MozartMozartplayed by 6 famous concert pianistsplayed by 6 famous concert pianistsAnalyzed across 12 different movements.Analyzed across 12 different movements.

represent a cross section of both playing represent a cross section of both playing keys, tempi and time signatureskeys, tempi and time signatures

Standard audio recordings, as oppose to Standard audio recordings, as oppose to MIDI format (which is more detailed)MIDI format (which is more detailed)

Musical RepresentationMusical Representation

•D. Barenboim K.279 1st movement C major•M. Uchida K.279 1st movement C major

Performance WormPerformance Worm

Tool developed to Tool developed to analyseanalyse audio dataaudio data

Extracts data by examining Extracts data by examining tempo tempo general loudness of the audiogeneral loudness of the audio

measured at the beatmeasured at the beat--levellevel..

Performance WormPerformance Worm

PreformancePreformance AlphabetAlphabet

from the worm, patterns can be observed from the worm, patterns can be observed which can help which can help characterisecharacterise the individual the individual playing styles of some pianistsplaying styles of some pianists

MitsukoMitsuko UchidaUchida

To capture these To capture these characterisationscharacterisations a a ““Mozart Performance AlphabetMozart Performance Alphabet’’ is is constructedconstructed

PreformancePreformance AlphabetAlphabet

1.1. The trajectories of the performance The trajectories of the performance worm are cut into short segments of a worm are cut into short segments of a fixed lengthfixed length

2.2. they are clustered into groups of similar they are clustered into groups of similar patterns to form a series of prototypespatterns to form a series of prototypes

3.3. Transcribe the performance in terms of Transcribe the performance in terms of this alphabetthis alphabet

4.4. Compare using string matching Compare using string matching techniquestechniques

Audio RecordingsAudio Recordings

String KernelsString Kernels

Uses GapUses Gap--weighted subsequences kernelweighted subsequences kernelKey idea: compare strings by means of the Key idea: compare strings by means of the subsequences they containsubsequences they containThe more subsequences they have in The more subsequences they have in common the more similar they are.common the more similar they are.

String KernelsString Kernels

String KernelsString Kernels

Partial Least SquaresPartial Least Squares

PLS is a flexible algorithm that was designed for PLS is a flexible algorithm that was designed for regression problems regression problems It is similar to PCAIt is similar to PCA

Instead of finding Instead of finding hyperplaneshyperplanes, it finds linear , it finds linear regression models.regression models.LR Models are found by projecting independent LR Models are found by projecting independent variable and dependent variables into new spacevariable and dependent variables into new space

PLS is used to find the fundamental relations PLS is used to find the fundamental relations between two Matricesbetween two Matrices

Partial Least SquaresPartial Least Squares

Offers an effective approach to solving Offers an effective approach to solving problems with training data with few points problems with training data with few points but high dimensionalitybut high dimensionality

1.1. Projecting the data into a lowerProjecting the data into a lower--dimensional space dimensional space 2.2. UtilisingUtilising a Least Squares (LS) regression modela Least Squares (LS) regression model

Resulting features are used in a Resulting features are used in a differenctdifferenctclassification or regression algorithmclassification or regression algorithm

Kernel PLSKernel PLS

Like PLS, it is a dimension reduction Like PLS, it is a dimension reduction method that is followed by SVMmethod that is followed by SVM

Uses a kernel instead of feature vectorUses a kernel instead of feature vectorIt is nonIt is non--linear transformationlinear transformation

It is faster than PLS in computation of It is faster than PLS in computation of componentscomponents

Kernel PLSKernel PLS

With KWith K--PLS the number of components PLS the number of components gradually increases until the model gradually increases until the model reaches some optimal dimension.reaches some optimal dimension.KK--PLS performs considerably well in cases PLS performs considerably well in cases of high noise level (especially compared to of high noise level (especially compared to kk--PCA)PCA)

Because generally KBecause generally K--PLS uses less PLS uses less componentscomponents

ExperimentsExperiments

For each pair of performers a leaveFor each pair of performers a leave--oneone--out out procedure was followedprocedure was followedEach run one movement played by each of the Each run one movement played by each of the pair was used for testing, the rest training.pair was used for testing, the rest training.

i.e. i.e. M.UchidaM.Uchida and and D.BarenboimD.Barenboim (MU(MU--DB) DB) .The algorithm run it 12 times, they are 12 .The algorithm run it 12 times, they are 12 mvtmvt, each , each iteration 1 iteration 1 mvtmvt (by both performers) was left out of (by both performers) was left out of the training set for testingthe training set for testing

This was repeated for all possible 15 pairsThis was repeated for all possible 15 pairsThe results are the number of The results are the number of correct correct classifications made by the algorithmclassifications made by the algorithm

Previous ResultsPrevious Results

Used feature based representationUsed feature based representationUsed WEKA for a variety of ML techniquesUsed WEKA for a variety of ML techniques––bayesianbayesian, rule, rule--based, treebased, tree--based and nearestbased and nearest--neighbourneighbour

The best result are for a classification via The best result are for a classification via regression metaregression meta--learnerlearnerFB rep used, included raw measures of tempo FB rep used, included raw measures of tempo and and loundnessloundness, variance and standard deviation, , variance and standard deviation, the correlation of tempo and loudness valuesthe correlation of tempo and loudness valuesAll extracted from the WormAll extracted from the Worm

ResultsResults

Experiments were conducted using standard string Experiments were conducted using standard string kernel and nkernel and n--gram kernelgram kernelBoth conducted using standard SVM, Kernel PLS, Kernel Both conducted using standard SVM, Kernel PLS, Kernel PCR and SVM in conjunction with projected features PCR and SVM in conjunction with projected features extracted by KPLSextracted by KPLSOne pair was chosen (RBOne pair was chosen (RB--DB) to select various DB) to select various paramatorsparamators

includes number of characters used by string kernel, decay paramincludes number of characters used by string kernel, decay parameters eters and the number of PLS features extracted where appropriate.and the number of PLS features extracted where appropriate.Substring lengths of Substring lengths of kk=1,...,10, lambda ={0.2,0.5,0.9} decay =1,...,10, lambda ={0.2,0.5,0.9} decay parameters and feature ranged from 1 to 10parameters and feature ranged from 1 to 10..

In each case the parameters that In each case the parameters that delievereddelievered the best the best performance on the RBperformance on the RB--DB data were chosen, and fixed DB data were chosen, and fixed for the remaining pairsfor the remaining pairs

ResultsResultsFB: Feature-Based

KPLS: Kernal PLS

KP-SV: SVM using KPLS features

KPCR: Kernel Principal Components regression

ResultsResults

There is an improvement of performance There is an improvement of performance over the featureover the feature--based approach.based approach.NN--gram kernelgram kernel

Projective into a lower subspace is beneficial Projective into a lower subspace is beneficial KPLS performance gain is attributed to its KPLS performance gain is attributed to its corrlecationscorrlecations between feature directions it between feature directions it selects and output variableselects and output variable

ResultsResults

String kernelString kernelPerformers significantly better than featurePerformers significantly better than feature--base and nbase and n--gram kernelgram kernel

Allows gaps in matching subsequences Allows gaps in matching subsequences Complex features are needed to obtain Complex features are needed to obtain good performancegood performanceUsing KPLS features did not Using KPLS features did not imporveimporveperformanceperformance

ConclusionConclusion

Presented a novel applications of the string kernelPresented a novel applications of the string kernelTo classify pianists by examining their playing style, rather thTo classify pianists by examining their playing style, rather than an analysinganalysingstatistical features obtained from audio recordingsstatistical features obtained from audio recordings

Their approach of examinations is using featureTheir approach of examinations is using feature--prjectionprjectionmethods in conjunction with kernels which operates on methods in conjunction with kernels which operates on texttextThis is applied to a string of characteristic tempoThis is applied to a string of characteristic tempo--loudness curves, that are obtained from the Performance loudness curves, that are obtained from the Performance WormWormString kernel outperforms nString kernel outperforms n--gram kernelsgram kernels

Contrary to text problemContrary to text problem

Determining when using KPLS to obtain features Determining when using KPLS to obtain features yeildsyeildsan improvement is still an open probleman improvement is still an open problem