1

EEG Signal Processing for Crew State Monitoring

Matthew N. GrayNIFS Student Intern

Airspace Operations and Safety ProgramCrew Systems and Aviation Operations Branch Mentors: Dr. Angela Harrivel, Chad Stephens

2

Crew State Monitoring (CSM) - Overview

• Part of Commercial Aviation Safety Team (CAST)

• National airspace is becoming increasingly busy and more complex.

• If we could stop just one fatal commercial aviation accident, we could save hundreds of lives and a billion dollars.  3 Boeing Statistical Summary of Commercial Worldwide Jet

Transport Accidents, 2011. Includes only accidents involving turbofan or turbojet airplanes with max takeoff weight > 60,000 lbs., referenced in the CAST Airplane State Awareness Joint Safety Analysis Team Final Report, June 17, 2014

3

• CSM aims to improve pilot operational efficiency during safety-critical operations by: o Aiding in pilot attention training

o Improving the human-machine interface in aircraft

• Move towards high-order classification with multiple physiological sensors rather than single sensors o AFDC studies (1.0, 1.5, 2.0,…)

Crew State Monitoring (CSM) - Overview

Subject in MATB (Multi-Aptitude Task Battery) Training with EEG cap (shown) and EKG nodes (not shown).

AFDC: Augmented Flight Deck Countermeasures

4State Classification via Machine Learning

Physiological monitoring

Mark II by NeXusHeart Rate Variability

Respiration RateGalvanic Skin Response

Eye Trackingby SmartEye

Electro-encephalography

B-Alert x24by Advanced Brain

Monitoring

Body MotionKinect by Microsoft for Xbox

(future)

Pilot and aircraft flight simulation data streams

Data SynchronizationMAPPS by EyesDx

CSM - Physiological Sensor Suite for Flight Simulation

Functional Near Infrared Spectroscopy

by BIOPAC(future)

Figures courtesy of Harrivel, Stephens, Ellis, Pope

5

CSM - Overview• Data Collection

oElectroencephalography (EEG)oEye TrackingoBody Motion Tracking (future studies)oEKG, Respiratory Rate, Galvanic Skin

ResponseoFunctional Near Infrared Spectroscopy

(fNIRS) (future studies)

• Data SynchronizationoMAPPSTM (Labels and Synchronizes)

(Multi-Analysis of Physiological and Psychological Signals)

Figure courtesy of Charles Liles and the LaRC Big Data and Machine Information Team

6

CSM - Overview• Signal Processing

oFiltering, Feature ExtractionoUsing MATLAB, Fieldtrip and EEGLab

• Model TrainingoSupport Vector Machines, Deep

Learning AlgorithmsoOther resources via Big Data Team and

academic partners• Model Evaluation

oTest efficacy of model via accuracyoDetermine which sensors are most

useful to the models

Figure courtesy of Charles Liles and the LaRC Big Data and Machine Information Team

7

EEG Signals - Background

• Neurons produce small, but detectable, electric potentials (μV)

• EEG nodes record these potentials

• Brain waves vary by Frequency:oGamma - γ (30-50 Hz)oBeta - β (14-30 Hz)oAlpha - α (8-14 Hz)oTheta - θ (4-8 Hz)oDelta - δ (1-4 Hz)

8

Signal ProcessingStep 1 - Artifact Removal: 1st Method

• Rolling Window EEG Power Band Calculationso Center value of rolling window compared to

mean of all values in rolling window

o If center value is >2 Standard Deviations from mean of all values in current window, it is labeled as an artifact and replaced with either a mean or median value

• Faster but less precise and higher chance of error

• Faster method is much more desirable, however, for its real-time feedback applications

(a) Brain Waves (b) Artifacts

Delta

Theta

Alpha

Beta

Gamma

EOG

EMG

ECG

Source: Jose Antonio Urigüen and Begoña Garcia-Zapirain 2015 J. Neural Eng. 12 031001 doi:10.1088/1741-2560/12/3/031001

9

Signal ProcessingStep 1 - Artifact Removal: 2nd Method• Fieldtrip and EEGLab

oThird Party MATLAB ToolboxesoVery fast and efficient at

processing raw signal data Useful for real-time implementation

oSegmentation of signal based on markers

oBuilt in pre-processing functions allowing user to apply various filters and select parameters

EEGLab - Brain Activity intensity diagram (source: cognitrn.psych.indiana.edu)

open source MATLAB toolbox

10

Signal ProcessingStep 2 – Feature Extraction

• Characterize EEG signals based on brain wave type

• Compute/Plot Power Spectral Density via Fast Fourier Transform

• Calculate area under curve for each frequency interval

• Using area under curve, calculate percent brain wave composition

• Engagement Index (EI):

Fast Fourier Transform (fft) (Calculates power of each frequency in signal)

11

Signal ProcessingStep 2 - Feature Extraction (cont.)

• Determine Stationarity of Raw SignaloDetermines minimum segment size for

gaining useful time-dependent information.

• Calculate Engagement Index for each moving window. oCreates time series ‘Alertness’ profile for

use in machine learning algorithms.

Features Extracted from EEG SignalsEngagement Index

(EI) % Power Comp.

Gamma 24.83%Beta

= 0.998524.49%

Alpha 11.30%Theta 13.23%Delta 26.16%

12

Signal Processing Outcome: Attention Meter• Real-time feedback of cognitive

engagement/awareness.

• Based on Engagement Index (EI)

• Could give real-time feedback to operator/trainer about engagement

• Future work to make Engagement Index more robustoDetermine source of cognitive activity

(internal or external?)

EI

13

Side Projects

• Functional Near-Infrared Spectroscopy (fNIRS)o BIOPAC fNIRS Imager 100

o Integrate into measured physiological sensors for next study (AFDC 2.0)

Record fNIRS and EEG simultaneously

• AFDC 1.5 Data CollectionoHelped pilots become cyborgs

o Assisted in data collection

fNIRS Imager 100 by BIOPAC, Commercial Product

14

Future Work

• Modify code to process multiple EEG signals (20+) o Parallel Computing

• Continue to optimize EEG filteringo Minimize artifacts while maximizing signal

• Create Time-dependant EI values for Machine Learning Efforts

• Internal vs. External Engagement Index

• Develop/Enhance experimental procedure for fNIRS in future studies (AFDC 2.0)