deep learning penmiir - sebastian stober · 1 chim chim cheree 3/4 210 13.3 13.5 2 take me out to...

Research FocusCognitive Sciences

Learning Discriminative Features from EEGRecordings by Encoding Similarity ConstraintsSebastian Stober

MLC g Research Focus Cognitive SciencesMachine Learning in Cognitive Science Lab

MLC g

songs with / without lyrics: meter tempo length (s)1 Chim Chim Cheree 3/4 210 13.3 13.52 Take me out to the Ballgame 3/4 186 7.7 7.73 Jingle Bells 4/4 200 9.7 9.04 Mary Had a Little Lamb 4/4 160 11.6 12.2

instrumental pieces:1 Emperor Waltz 3/4 178 8.32 Harry Potter Theme 3/4 166 16.03 Imperial March (Star Wars Theme) 4/4 104 9.24 Eine Kleine Nachtmusik 4/4 140 6.9

penMIIRpublic domain dataset of EEG recordingsfor music imagery information retrieval

This research was supported by the donation of a Geforce GTX Titan X graphics card.

Classifier Features Accuracy McNemar’s test (mid-p) vs. *

SVC raw EEG 18.52% 0.0002SVC raw EEG channel mean 12.41% < 0.0001End-to-end NN raw EEG 18.15% 0.0001Neural Net (NN) SCE output 27.22% 0.8200*SVC SCE output 27.59%

Stimulus Recognition (12-class classification)• minimalistic encoder: 1 spatial convolution filter• nested cross-validation scheme:

Deep Learning for EEG Analysis?

• high-dimensional data

• limited amount of trials

• bad signal-to-noise ratio

https://github.com/sstober/openmiir

• passive listening condition• 64 EEG channels @ 512 Hz• 9 subjects x 12 stimuli x 5 trials• all trials cropped at 6.9 s

• 12 music stimuli from 8 music pieces:

540 trials (~ 62 min total) with > 225k dimensions

9-foldsubjectcross-validationtrainondatafrom8subjects(8x5x12=480trials)

testonremainingsubject(1x5x12=60trials)

Pre-Training

5-foldtrialblockcross-validation8x4x12=384trainingtrialsfrom4trialblocks

8x1x12=96validationtrialsfromremainingtrialblock

trainon50688tripletsfrom384trainingtrials

selectmodel(earlystopping)basedon21120validationtriplets(a,b,c)witha from96validationtrialsandb,c from480trainingandvalidationtrials

encoderlayer(L1)=averageoverfolds

Supe

rvise

dTraining 5-foldtrialblockcross-validation

sametraining/validationsplitsasinpre-trainingphase

trainwithCon480trainingtrials classifierlayer(L2)=meanoverfolds

LinearSupportVectorClassifier(SVC)gridsearch forCwithhighest

meanvalidationaccuracy

SimpleNeuralNetwork(NN)selectfoldmodel(earlystopping)

basedonhighestvalidationaccuracy

Chim Chim Cheree (lyrics)Take Me Out to the Ballgame (lyrics)

Jingle Bells (lyrics)Mary Had a Little Lamb (lyrics)

Chim Chim ChereeTake Me Out to the Ballgame

Jingle BellsMary Had a Little Lamb

Emperor WaltzHedwig’s Theme (Harry Potter)

Imperial March (Star Wars Theme)Eine Kleine Nachtmusik

minimizeconstraintviolations

FeatureExtraction(signalfilter)

InputTriplet

(sharedparameters)

PairwiseSimilarity(dotproduct)

Prediction(binary)

EncoderPipeline

EncoderPipeline

EncoderPipeline

Similarity

Similarity

Output

Reference

PairedTrial

OtherTrial

netw

ork

stru

ctur

e

A

B

C

• systematic errors: confusing lyrics/non-lyrics pairs• little difference between classifiers

Average Weights of the Neural Network Classifier(including pre-trained encoder filter)

Network Activation Analysis

http://www.uni-potsdam.de/mlcog

• temporal patterns in weights and activations are picking up musical events (down beats)

forward model:(regression)

• dimensionality reduction by factor 64• significant improvement of signal-to-noise ratio• more complex encoders and classifiers possible• pre-training is key to reduce over-fitting

goo.gl/4xzma3

Comparison against Baselines

Confusion Analysis

• common encoder pipeline extracts features that are representative and allow to distinguish classes

sim(A,B) > sim(A,C)

Cha

lleng

e

Dat

aset

Met

hod

Expe

rimen

t

Res

ults

for all A,B from same class and C from other class

Similarity-Constraint Encoding (SCE) Pre-Training• motivated by metric learning• find features that satisfy similarity constraints