multi-way modelling and analysis of high-dimensional eeg...

Multi-way modelling and analysis ofhigh-dimensional EEG data

Roman Rosipal

Department of Theoretical MethodsInstitute of Measurement Science, SAS

Bratislava, Slovak Republic&

Pacific Development and Technology, LLCPalo Alto, CA

FMFI UK, February 25, 2013, Bratislava

Introduction Data & Methods Results Conclusions & Beyond

Work carried out with:

Leonard J Trejo & Paul Nunez


Estimation of Cognitive Status

Reward system

Executive

control

Working memory Sensation

& perception

Autonomic system

Other Processes

Internal Processes Aroused

or Overloaded

Fatigued

Resting Engaged

Working

Other States

Behavior and Performance

Biosignals


Useful Definitions

Work- load

Mental Fatigue

Non- specific Factors

Engage- ment

General Cognitive

Status

Engagement: selection of a task as the focus of attention andeffort

Workload: significant commitment of attention and effort to taskOverload: task demands outstrip performance capacityMental Fatigue: desire to withdraw attention and effort from atask


Useful Definitions

Work- load

Mental Fatigue


Engage- ment

General Cognitive

Status

Engagement: selection of a task as the focus of attention andeffortWorkload: significant commitment of attention and effort to task

Overload: task demands outstrip performance capacityMental Fatigue: desire to withdraw attention and effort from atask


Useful Definitions

Work- load

Mental Fatigue


Engage- ment

General Cognitive

Status

Engagement: selection of a task as the focus of attention andeffortWorkload: significant commitment of attention and effort to taskOverload: task demands outstrip performance capacity

Mental Fatigue: desire to withdraw attention and effort from atask


Useful Definitions

Work- load

Mental Fatigue


Engage- ment

General Cognitive

Status

Engagement: selection of a task as the focus of attention andeffortWorkload: significant commitment of attention and effort to taskOverload: task demands outstrip performance capacityMental Fatigue: desire to withdraw attention and effort from atask


Why to monitor cognitive status?

Critical safety, high workload, stressful, etc., environments


Experiments - (A) Cognitive Workload Monitoring

Uninhabited Air Vehicle (UAV) control

Trained subjects were monitoring several UAVs as they flew apreplanned mission; processing SAR images (synthetic apertureradar), vehicle health control, etc.Different task conditions were used to control cognitive workloadlevels




Trained subjects were monitoring several UAVs as they flew apreplanned mission; processing SAR images (synthetic apertureradar), vehicle health control, etc.

Different task conditions were used to control cognitive workloadlevels




Trained subjects were monitoring several UAVs as they flew apreplanned mission; processing SAR images (synthetic apertureradar), vehicle health control, etc.Different task conditions were used to control cognitive workloadlevels


Experiments - (B) Mental Fatigue Monitoring

Problem solving≈ 4 – 15 s

Intertrial interval 1sResponse

Problem solving≈ 4 – 15 s

3+5-7+1 <=>2??

9-6+2-4 <=>2??

Continuos performance of mental arithmetic for up to three hours


Data - Electroencephalogram (EEG)

Cerebral Cortex •  the outermost layers of brain •  2-4 mm thick (human)


Data - EEG Sources


Data - EEG Sample


Spectral EEG Data Representation

Data usually segmented into epochs (e.g. 2 sec long)

Spectral representation: Estimate of the power spectrum density;that is the distribution of power per unit frequency (Thompsonmulti-taper , Welch, etc.)

Pxx (f ) = Fx (f )F ∗x (f )

where Fx (f ) is the Fourier transform of the signal x and ∗indicates the complex conjugateExample:

5 10 15 20 25 300

0.5

1

1.5

2

2.5

3

Hz

μV2 /H

z

An examle of the power spectral density estimate Subject B, electrode Cz



Data usually segmented into epochs (e.g. 2 sec long)Spectral representation: Estimate of the power spectrum density;that is the distribution of power per unit frequency (Thompsonmulti-taper , Welch, etc.)

Pxx (f ) = Fx (f )F ∗x (f )

where Fx (f ) is the Fourier transform of the signal x and ∗indicates the complex conjugate

Example:

5 10 15 20 25 300

0.5

1

1.5

2

2.5

3

Hz

μV2 /H

z




Data usually segmented into epochs (e.g. 2 sec long)Spectral representation: Estimate of the power spectrum density;that is the distribution of power per unit frequency (Thompsonmulti-taper , Welch, etc.)

Pxx (f ) = Fx (f )F ∗x (f )

where Fx (f ) is the Fourier transform of the signal x and ∗indicates the complex conjugateExample:

5 10 15 20 25 300

0.5

1

1.5

2

2.5

3

Hz

μV2 /H

z



Coherence EEG Data Representation

Coherence representation: Cross power spectra density Pxy (f ),

Pxy (f ) = Fx (f )F ∗y (f )

or magnituted squared (coherence)

Cxy (f ) =|Pxy (f )|2

Pxx (f )Pyy (f )


Data - Multi-modal Multi-Sensor

ECG - hear rate, heart rate variabilityEOG and eyes control - hEOG, vEOG movements, blinks, pupildiameterEMGSkin conductance, SCR, GCRVideotaped recordingsResponse time, Correctness of responsesSubjective responses and questionnairesetc.


Data Structure

Workload

cond+ons

(e.g., trials, +me)

EEG Frequency

Electr

odes

Data matrix construction: X(I×J×K )

I - time segmentsJ - electrodes or electrode pairsK - PSD or CSD (coherences)


Bilinear Unfolding

Dim 1 Dim 2 Dim 3

Dim

2 &

3

X X

X1 X2 X3 D

im 1

& 3

Dim

1 &

2

Representing all experimental factors in one dimension &observations (trials) in second dimensionContrast each dimension vs. pair of the other two


Bilinear Unfolding - Modelling

Factor Analysis

Principal Component Analysis (PCA)

ijjf

F

fifij ebax +=∑

=1

∑=

=F

f 1 af

bf

0=ije


Bilinear Unfolding - Regression/Classification

Partial Least Squares

Ø  Data sets: X (nobjects x Nvariables)

Y (nobkjects x Mresponses) Ø  Bilinear decomposition: X = TPT + E Y = UQT + F where: T,U matrices of score vectors (LV, components) P,Q matrices of loadings E,F matrices of residuals (errors) Ø  Criterion: max|r|=|s|=1[cov(Xr, Ys)]2 = [cov(Xw, Yc)]2

= var(Xw)[corr(Xw, Yc)]2var(Yc) = [cov(t,u)]2


NIPALS

PLS - bilinear decomposition of X and Y maximizing covariancebetween score vectors t = Xw and u = Yc

max|r|=|s|=1[cov(Xr,Ys)]2 = [cov(Xw,Yc)]2

= var(Xw)[corr(Xw,Yc)]2var(Yc)= [cov(t,u)]2

NIPALS algorithm finds the weights w,c :

1) w = XT u/(uT u) 4) c = YT t/(tT t)2) ‖w‖ → 1 5) ‖c‖ → 13) t = Xw 6) u = Yc

7) go to 1)


Eigenproblem & Deflation

instead of NIPALS we can solve an eigenproblem:

w ∝ XT u ∝ XT Yc ∝ XT YYT t ∝ XT YYT Xw

XT YYT Xw = λwt = Xw

or XXT YYT t = λtu = YYT t

p = XT t/(tT t) ; q = YT u/(uT u)

sequential extraction of {ti}mi=1

X0 = Xti = Xi−1wi , Xi = Xi−1 − tipT

i = X−∑i

j=1 tjpTj

deflation schemes define different forms of PLS


CCA, PLS, and PCA CR

PLS:

max|r|=|s|=1

[cov(Xr,Ys)]2 = max|r|=|s|=1

var(Xr)[corr(Xr,Ys)]2var(Ys)

CCA:max|r|=|s|=1

[corr(Xr,Ys)]2

PCA:max|r|=1

[var(Xr)]

Continuum Regression (Stone & Brooks’90)

max|r|=|s|=1

var(Xr)γ1 [corr(Xr,Ys)]2var(Ys)γ2

γ1, γ2 ∈ (0,∞)


Canonical Ridge Analysis - CCA PLS

(Vinod’76)

([1− γX ]XT X + γX I)−1XT Y([1− γY ]YT Y + γY I)−1YT Xw = λw

CCA: γX = 0, γY = 0PLS: γX = 1, γY = 1Orthonormalized PLS: γX = 1, γY = 0 or γX = 0, γY = 1Ridge Regression, Regularized FDA or CCA: γX ∈ (0,1), Y ∈ R


PLS Discrimination/Classification

Y =

1n1 0n1 . . . 0n1

0n2 1n2 . . . 0n2

......

. . . 1ng−1

0ng 0ng . . . 0ng

Orthonormalized PLSY = Y(YT Y)−1/2

YT Y = I


Bilinear Unfolding - PLS - Classification


Nonlinear (kernel) mapping: an example


Example: All Degree d=2 monomials

Φ : X = R2 → F = R3

x = (x1, x2)→ Φ(x) = (x21 ,√

2x1x2, x22 )

K (x,y) = 〈Φ(x),Φ(y)〉 = (x21 ,√

2x1x2, x22 )T (y2

1 ,√

2y1y2, y22 )

= x21 y2

1 + 2x1x2y1y2 + x22 y2

2= ((x1, x2)T (y1, y2))2

= 〈x,y〉2

Problem: 〈x,y〉d ,X ⊆ RN → (N+d−1)!d!(N−1)!

N = 16× 16,d = 5→ dim. 1010


Bilinear Unfolding - (Kernel) PLS - Regression

Rosipal,R & Trejo, LJ (2001). Kernel Partial Least Squares Regression inReproducing Kernel Hilbert Space. Journal of Machine Learning Research,2(Dec):97-123.


Multi-way Analysis

PARAFAC

ijkkfjf

F

fifijk ecbax +=∑

=1

∑=

=F

f 1 af

bf cf


PARAFAC model

The PARAFAC model with F factors: decomposition of the datamatrix X using three loading matrices, A, B, and C with elementsaif , bjf , and ckf

xijk =F∑

f=1

aif bjf ckf + εijk

The criterion:

minaif ,bjf ,ckf

= ‖xijk −F∑

f=1

aif bjf ckf‖2


Multi-way PLS

Multi-way PLS (n-PLS)

X

frequency

workload condi1on

X

∑=

=F

f 1 af

cf bf

∑=

=F

f 1 vf

uf

1me

1me

max. covariance

electr

odes

EEG

Labels

af – spectral atom bf – spatial atom cf – temporal atom

vf – workload atom uf – temporal atom


Mental Fatigue - PLS analysis

Black = Alert Red = Mentally Fatigued Fz Pz

Frontal Theta

Parietal Alpha



Robust EEG-Based Classification of Mental Fatigue 2300 (Day 1) vs. 1900 Hrs (Day 2)

40

50

60

70

80

90

100

Signal-to-noise Ratio (dB)

Test

Pro

porti

on

Corr

ect

21 Channels 100 100 99 94 79 56 50 12 Channels 98 98 96 88 65 51 50 4 Channels 87 88 90 88 76 54 50

0 -3 -6 -9 -12 -15 -18


Mental Fatigue - Spectrum Analysis - PARAFAC

APECS Final Report ARO Contract No. W911NF08CO121 September 15, 2009 PDT Report No. UA01BF0636B131

Page A3-2

1200 2400 3600 4800 6000 7200 8400 9600 10800-6

-4

-2

0

2

4

6

Time(s)

Load

ing

(bas

elin

e ad

just

ed z

- sco

re)

5 10 15 20-0.2

0

0.2

0.4

0.6

0.8

1

1.2

Frequency(Hz)

Load

ing

O2 O1 OZ PZ P4 CP4 P8 C4 TP8 T8 P7 P3 CP3CPZ CZ FC4FT8 TP7 C3 FCZ FZ F4 F8 T7 FT7 FC3 F3 FP2 F7 FP1-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

Electrodes

Load

ing

Subject: GSD Epoch: 2 s Overlap: 0 s Method: Welch

Figure 33. Atomic decomposition of EEG from participant GSD of the NASA-C study. EEG recordings from 30 channels were processed using PARAFAC decomposition to yield a model consisting of four atoms, each have dimensions of space (electrodes), frequency (power spectral density) and time (time on task). Graphical conventions are the same as in Figure 32. This participant performed the task for three hours, or 12 15-minute blocks. The time axis measures seconds as multiples of 2-second long EEG epochs which were not all contiguous, due to rejection of EEG segments containing movement or other artifacts. Some blocks have fewer epochs than others because the incidence of EEG artifacts increased during those blocks.


Mental Fatigue - Coherence Analysis - PARAFAC

APECS Final Report ARO Contract No. W911NF08CO121 September 15, 2009 PDT Report No. UA01BF0636B131

Page A4-3

Figure 45. Coherence analyses for participant GSD. Graphing conventions are explained in Figure 44.


Workload - UAV - PARAFAC

Subjects E,G,I, K (plotted subject E)



Subjects E - coherence

We found the similar decomposition for subjects B, G, I, K



Subjects B,E,G,I, K - coherence


Workload - UAV - Coherence Analysis

6 10 14 180

0.1

0.2

0.3

0.4

0.5

0.6

Frequency(Hz)

0 200 400 600 800-6

-4

-2

0

2

4

6

Time(s)

Subject: I Factor: 1 Perc.: 90

O2O1 OZ

PZ P4

CP4

P8

C4

TP8

T6

P7P3

CP3 CPZ

CZ

FC 4 T4

TP7

C3

FC Z

FZ F4F8

T5

T3 FC 3

F3

FP2

F7

FP1


O2O1 OZ

PZ P4

CP4

P8

C4

TP8

T6

P7P3

CP3 CPZ

CZ

FC 4 T4

TP7

C3

FC Z

FZ F4F8

T5

T3 FC 3

F3

FP2

F7

FP1


O2O1 OZ

PZ P4

CP4

P8

C4

TP8

T6

P7P3

CP3 CPZ

CZ

FC 4 T4

TP7

C3

FC Z

FZ F4F8

T5

T3 FC 3

F3

FP2

F7

FP1

Factor 1Factor 2Factor 3


Conclusions

Results show that mental workload may be tracked by EEGcomponents isolated using PARAFAC

On UAV data set, the workload related atoms was remarkablystable in 5 out of the 6 subjects

The short-and long range coherence related atoms are morestable across the subjects, provide higher discrimination of thelow and high workload levels and seem to be less susceptible tothe movement related artifacts

We observed similarly promising and remarkable results onadditional two data sets monitoring cognitive status


Future Plans

BCI, VEGA & APVV with Centre for Cognitive Science, FMFI

Resting State Networks, neurofeedback and lateralised attentionproject with Eran Zaidel lab, UCLA


Thank you !!!

http://aiolos.um.savba.sk/ roman/

References:Trejo L.J., Rosipal R., Nunez P.L. Advanced Physiological Estimation of Cognitive Status (APECS). Final project report,U.S. Army Research Offfice, Research Triangle Park, NC, September 2009.Trejo L.J., Rosipal R., Nunez P.L. Advanced Physiological Estimation of Cognitive Status. The 27th Army ScienceConference, Orlando, Florida, November 29 - December 2, 2010.Rosipal, R., Trejo, L. J., Nunez, P. L. (2009). Application of Multi-way EEG Decomposition for Cognitive WorkloadMonitoring. In Proceedings of the 6th International Conference on Partial Least Squares and Related Methods, Vinzi V.E,Tenenhaus M., Guan R. (eds.), Beijing, China, pp. 145-149, 2009.

Trejo L.J., Knuth K., Prado R., Rosipal R., et al. (2007). EEG-based Estimation of Mental Fatigue: Convergent Evidence

for a Three-State Model. In Proceedings HCII 2007, Beijing, China, Springer, pp. 201-211.

multi-way modelling and analysis of high-dimensional eeg...

Documents