This content has been downloaded from IOPscience. Please scroll down to see the full text.

Download details:

IP Address: 132.203.227.63

This content was downloaded on 14/06/2014 at 20:41

Please note that terms and conditions apply.

EEG predictors of covert vigilant attention

View the table of contents for this issue, or go to the journal homepage for more

2014 J. Neural Eng. 11 035009

(http://iopscience.iop.org/1741-2552/11/3/035009)

Home Search Collections Journals About Contact us My IOPscience

Journal of Neural Engineering

J. Neural Eng. 11 (2014) 035009 (11pp) doi:10.1088/1741-2560/11/3/035009

EEG predictors of covert vigilant attention

Adrien Martel1,3, Sven Dahne2 and Benjamin Blankertz1

1 Neurotechnology Group, Berlin Institute of Technology, Berlin, Germany2 Machine-Learning Group, Berlin Institute of Technology, Berlin, Germany

E-mail: adrien.mm.martel@campus.tu-berlin.de

Received 30 August 2013, revised 31 October 2013Accepted for publication 15 November 2013Published 19 May 2014

AbstractObjective. The present study addressed the question whether neurophysiological signalsexhibit characteristic modulations preceding a miss in a covert vigilant attention task whichmimics a natural environment in which critical stimuli may appear in the periphery of thevisual field. Approach. Subjective, behavioural and encephalographic (EEG) data of 12participants performing a modified Mackworth Clock task were obtained and analysed offline.The stimulus consisted of a pointer performing regular ticks in a clockwise sequence across 42dots arranged in a circle. Participants were requested to covertly attend to the pointer and pressa response button as quickly as possible in the event of a jump, a rare and random event. Mainresults. Significant increases in response latencies and decreases in the detection rates werefound as a function of time-on-task, a characteristic effect of sustained attention tasks knownas the vigilance decrement. Subjective sleepiness showed a significant increase over theduration of the experiment. Increased activity in the α-frequency range (8–14 Hz) wasobserved emerging and gradually accumulating 10 s before a missed target. Additionally, asignificant gradual attenuation of the P3 event-related component was found to antecedemisses by 5 s. Significance. The results corroborate recent findings that behavioural errors arepresaged by specific neurophysiological activity and demonstrate that lapses of attention canbe predicted in a covert setting up to 10 s in advance reinforcing the prospective use ofbrain–computer interface (BCI) technology for the detection of waning vigilance in real-worldscenarios. Combining these findings with real-time single-trial analysis from BCI may pavethe way for cognitive states monitoring systems able to determine the current, and predict thenear-future development of the brain’s attentional processes.

Keywords: EEG, BCI, covert vigilant attention, P3, α-rhythm

S Online supplementary data available from stacks.iop.org/JNE/11/035009/mmedia

(Some figures may appear in colour only in the online journal)

1. Introduction

Several work environments and daily activities require thecontinuous and constant attention of a human operator. Thisis especially true in the wake of mass computerization andubiquitous automation which have made human supervision,and concurrently, surveillance tasks a widespread necessity.These types of tasks are usually characterized by extensiveperiods of monotonous and attention-demanding observationof a particular environment or interface and to remain on

3 Author to whom any correspondence should be addressed.

the qui vive for rare critical stimuli (Grier et al 2003).Researchers investigating the capacity of operators to sustaintheir attention in such conditions have consistently observed asignificant deterioration of the ability to detect critical stimuliover time, named the vigilance decrement (Mackworth 1948,1968, Davies and Parasuraman 1982, Matthews 2000). Inthe context of high-risk environments (Sarter et al 2007)such as nuclear power plants (Sheridan 2002) or securityscreenings (Hancock and Hart 2002) momentary lapses ofattention harbour the potential for serious and even disastrousconsequences. The human capacity to endogenously maintainfocused attention on a portion of the environment or a certain

1741-2560/14/035009+11$33.00 1 © 2014 IOP Publishing Ltd Printed in the UK

J. Neural Eng. 11 (2014) 035009 A Martel et al

task to monitor for changes is referred to as vigilance. Mostcognitive processes are intrinsically difficult to assess on asecond-to-second basis with traditional methods relying eitheron retrospective subjective reports or behavioural responses.Vigilance belongs to so-called covert aspect of user state(CAUS) (Zander et al 2010) and is particularly difficult tobe determined in real-time since it requires long monitoringperiods with few interspersed target stimuli. Besides theability to improve the agency and communication of patientswith severe motor impairments (Wolpaw et al 2002), brain–computer interface (BCI) represents a powerful method forthe assessment of macroscopic brain states such as intentions,emotions, attention and/or accuracy capabilities in real-time (Blankertz et al 2010, Zander et al 2010). In viewof this possibility, BCI-based systems are being envisionedwhich would enhance human abilities (He et al 2013) orgreatly promote the synergistic cooperation between users andmachines (Grubb and Cohn 2012). Neuroergonomic adaptiveautomation (Parasuraman and Wilson 2008), augmentedcognition (Hutchison et al 2011), neuroadaptive systems(Stanney 2012) and passive BCIs (Zander and Jatzev 2012)all share the aim of developing enhanced human–machineinteraction (HMI) by dynamically adapting the interactionaccording to the operator’s or user’s mental state. Previouswork demonstrated that BCI is a viable method for thereal-time monitoring and partial alleviation of workload(Kohlmorgen et al 2007) or the real-time detection andcorrection of errors (Chavarriaga and del Millan 2010, Schmidtet al 2012). It has been well established that alpha activityand the P3 amplitude are modulated by the time spenton a vigilance task (Gale 1977, Ritter and Vaughan 1969,Parasuraman and Davies 1975, 1977). However, in order toeffectively thwart attentional lapses, the system must have thecapacity to not only detect in real-time but also to reliablypredict the most probable evolution of the cognitive statefrom the currently measured neural activity, i.e. detecting apropensity towards an inadequate attentional level and informthe user before a lapse actually occurs so as to allow forenough time for corrective measures. Recent studies uncoveredneurophysiological precursors of attentional lapses. O’Connellet al (2009) observed a modulation of the α-activity 20 s andof the contingent negative variation (CNV) and P3 four to fiveseconds before a lapse of attention in a temporal expectancytask. Eichele et al (2010) unveiled a modulation of the N2,five trials prior to an error in a modified Eriksen flanker task.The present work aimed at obtaining analogous results in acovert vigilant attention task which more closely resembles areal-world environment where rare critical stimuli may appearon the periphery of the visual field.

2. Methods

2.1. Subjects

Twelve healthy subjects (four female; 27.4 ± 2.9 years)volunteered for participation in this study. All of them reportednormal or corrected-to-normal visual acuity. None of theparticipants reported or exhibited signs of substance abuse

or sleep deprivation as assessed with the FAL questionnaire(‘acquisition of initial condition questionnaire’, Janke 1976).Data from one of the participants were discarded due tohardware malfunctions. The final sample comprised fourfemales and seven males with an average age of 27.2 ( ± 2.9)years. All participants gave written informed consent.

2.2. Experimental design and procedure

Participants performed a modified ‘Mackworth clock test’(Mackworth 1948) implemented in Pyff (Venthur et al 2010).The stimulus, displayed on a LCD monitor, consisted of 42white dots arranged in a circle around a centrally placedfixation cross. One of the dots was coloured in green and, akinto the pointer of a clock, performed regular ‘ticks’, i.e. movedfrom one position to the next, in a clockwise direction everysecond. Pseudo-randomly distributed over the duration of thetask the pointer omitted one position (10% of the stimuli)and performed a ‘jump’, i.e. the target stimulus (figure 1).The time course of the task was divided by the number oftarget stimuli into equally sized segments in which the targetwas random. In order to prevent multiple target stimuli fromimmediately succeeding each other a minimum number of twoin-between ticks was set. The diameter of the dot circle, basedon a viewing distance of 70 cm, was 10◦ of the visual field, thusrestricting foveal stimulus processing and required participantsto covertly shift their attention to follow the progression of thepointer. The participants were instructed to gaze at the fixationcross in the centre, keep track of the pointer using covertattention and to react as fast as possible to the rare jumps bypressing a button with their right index finger. To minimize theeffects of neural habituation and the resulting visual fading ofthe fixated object (Riggs et al 1953) the colour of the fixationcross was implemented in such a manner as to slowly shiftcolour through the spectrum of the rainbow. The subjectswere further instructed to perform periodic microsaccadeswithin the boundaries of the fixation cross in-between jumpsto counteract Troxler fading (Martinez-Conde et al 2006).

To ensure that the stimulus could be perceived participantswere required to complete a short practice session, in whichauditory feedback was given for false positives and omissionerrors, with 30 jumps for 2 min with 100% accuracy. Allthe participants successfully completed the practice. Theexperimental session was divided into four blocks of 15.4 mineach, with 90 jumps per block, by short breaks of at least 5 min.A modified version of the Stanford Sleepiness Scale (Hoddeset al 1973) was applied directly before the experiment started,after the training session and immediately following eachblock. The task was purposefully designed to be unengagingand monotonous as to impose great strain on cognitiveresources (Warm et al 2008), especially endogenously directedattention, and thus encourage attentional drifting.

2.3. Data acquisition

Superficial cortical activity was recorded at 1 kHz samplingfrequency with multichannel EEG amplifiers (BrainAmp DCby Brain Products, Munich, Germany) via an actiCAP activeelectrode system (Brain Products, Munich, Germany). The cap

2

J. Neural Eng. 11 (2014) 035009 A Martel et al

Figure 1. Modified Mackworth Clock. The participants fixated the cross in the centre and covertly attended to the green dot whichperformed regular clockwise ticks (green arrow) and had to respond in the event of a jump (red arrow).

was equipped with 64 Ag/AgCl electrodes placed according tothe extended international 10–20 system with impedances keptbelow 5 k�. Channels were referenced to left/right mastoidin conjunction with a ground electrode positioned on theforehead. The recording of the participant’s responses wascarried out by a RB-830 Response Pad (Cedrus Corporation2010). The markers of the ticks and jumps, obtained vialightsensors, and from the response pad were registeredwith a StimTracker system (Cedrus Corporation 2010) andstored alongside EEG data by BrainVision software (BrainProducts, Munich, Germany). Analyses of behavioural andEEG data were carried out in MATLAB version R2010a(The MathWorks, Natrick, MA, USA) while all statisticalanalyses were performed using IBM R©SPSS Statistics 20 (IBMCompany, Armonk, NY, USA).

2.4. Behavioural analysis

A target trial was labelled as a ‘hit’ if a response was registeredwithin a time window of 100 to 1000 ms, i.e. within the timewindow starting shortly after a jump and lasting until the nexttick. Accordingly, a target trial was categorized as a ‘miss’ inthe absence of a response during that time window. Responsesfaster than 100 ms were discarded as anticipation errors andresponses slower than 1000 ms were not considered. Accuracyrefers to the ratio of hits against the total amount of target trials.In order to depict changes in response time (RT) and accuracyover time, each of the four blocks was divided into 30 timebins of 30 s each plus one block of 22 s, equating to 124time bins over the whole experiment. To better illustrate thetemporal development of behavioural measures, smoothingwas achieved by applying a moving average of period 10calculated for each block.

For the purpose of statistical analysis, each block wassegmented into 5 bins of 3 min and 4.4 s. Two 4 × 5 repeatedmeasures ANOVA were conducted in order to study the effectof the within-subject factors block and time on the dependentvariables accuracy and RT.

Concerning the questionnaire results, a one-way repeatedmeasures ANOVA was conducted with the within-subjectfactor time point of measurement (six levels: before theexperiment, after training and after each of the four blocks).

2.5. EEG processing

For EEG analysis, the data was low-pass filtered using aChebyshev II filter with 80 Hz pass-band and 95 Hz stop-band,and subsequently downsampled to 200 Hz. The continuoussignal was segmented into epochs of 1200 ms, from −200to 1000 ms relative to stimulus onset. Target epochs werecategorized as either belonging to the class hits or the classmisses and labelled accordingly. Non-target epochs werelabelled according to the number of trials remaining until thenext target trial. To overcome data contamination by signalsunrelated with neural activity, such as ocular or muscularactivity, artifactual epochs and channels were automaticallydetected and removed using a 70 μV min-max criterion. Inorder to determine to what extent the EEG signal differsbetween hit and miss trials, the discriminability across classeswas determined with the signed point-biserial correlationcoefficient sgn r2 (Blankertz et al 2011).

The present study aimed at identifying the characteristicneurophysiological dynamics preceding lapses of attention,the EEG analysis therefore focused on the examination of thedifferences between pre-hit and pre-miss trials. For the firststep of the pre-target analysis grand averages were computedby averaging the epochs class-wise over all blocks andparticipants. Subsequently the four trials directly preceding atarget were collapsed across trials by means of class-wiseaverage for the electrodes Fz, Cz, PO7 and PO8. The electrodesand time intervals were chosen according to the temporaland spatial distribution of the event-related potential (ERP)components of interest. The channels PO7 and PO8 wereselected since the N2pc (negative posterior contralateral) isa lateralized negative ERP with a parieto-occipital distribution(over the visual cortex) within the temporal window of the N2(Luck et al 1993). The P3 has been shown to peak between 250and 500 ms after stimulus onset depending on the experimentalparadigm and be fronto-centrally localized (Polich 2007).Accordingly, the channels Fz and Cz were selected and aheuristic search was used to pinpoint the peak latency of theP3. In order to examine the evolution of the pre-hit and pre-miss ERP waveforms, the ten trials preceding targets wereconcatenated. The pre-target analysis was limited to ten trialsprior to target due to the probability of another target stimulus.Visual inspection revealed a gradually emerging divergence inP3 peak amplitude between the two detection classes emerging

3

J. Neural Eng. 11 (2014) 035009 A Martel et al

Figure 2. Mean accuracy against time with the moving average of period 10. The bar diagram depicts the total amount of hits, misses andfalse positives. Each bar bin spans 30 s. The vertical dash–dot lines mark the break between blocks.

seven trials before the target. A linear regression was thenestablished for the difference in mean peak amplitudes acrossclasses for the P3 component against target anteceding trialsextracted from the most discriminative channel (Cz) for thetime intervals 200–300 ms and 235–255 ms for epochs rangingfrom the target trial to the seven trials prior. The temporalwindow for the linear regression was chosen beginning withthe first pre-trial exhibiting peak amplitude divergences acrossclasses and ending with the trial directly preceding the targettrial.

The final step of the analysis was to investigate thecharacteristics of the pre-target neural oscillatory dynamicsin the alpha-band power spectrum leading to a hit ormiss. To examine second-by-second fluctuations in alphaactivity, the continuous EEG was epoched into stimulus-locked periods of 1000 ms, resulting in epochs spanning theentirety of an inter-stimulus interval (ISI). The epochs werecategorized and labelled in the same fashion as for the ERPanalysis. Epochs and channels containing ocular or muscularartefacts were automatically rejected. The EEG channels werethen bandpass filtered using a Butterworth filter for the 8–14 Hz band. Subsequently, the envelope was calculated as abandpower estimate. Lastly, the grand averages were obtainedby averaging class-wise over all participants. The mean peakamplitude was extracted from the most discriminative channel(POz) for epochs ranging from the target trial to ten trials prior.A linear regression was computed for the difference in spectralamplitude between hits and misses.

3. Results

3.1. Questionnaire results

The results of the questionnaire revealed that fatigue increasedover the course of the experiment (supplementary figure 1,

available from stacks.iop.org/JNE/11/035009/mmedia). Onaverage, participants rated their fatigue level as ‘normal’(3.45) at the beginning and mildly drowsy at the end (6.8)with a steady decrease over the course of the experiment. Asignificant effect of time on subjective sleepiness was found,F(2.328, 23.280) = 15.967, p < 0.001, indicating that time-on-task had a detrimental effect on arousal.

3.2. Behavioural results

The results of the modified Mackworth Clock revealedthat on average participants responded correctly to 66%(SD = 13%) of the target trials with an average RT of546.5 ms (SD = 134.5 ms). The rate of false alarmsexhibited average of 12.1% (SD = 9.7%). The overallbehavioural results can be taken from supplementary table1 (available from stacks.iop.org/JNE/11/035009/mmedia).The temporal evolution of the averaged RT and accuracyyielded typical results for a sustained attention task, anincrease of the former and decrease of the latter over thecourse of the experiment, albeit in a noticeably shortertime frame (within the first 5 min). The averaged accuracyand accuracy are depicted in figures 2 and 3. Besides theevident strong decline in accuracy over the course of eachblock (figure 2), the decreasing starting accuracy from blockto block seems to suggest that fatigue had a cumulativeeffect. This effect failed to achieve statistical significance,F(3, 27) = 0.695, p = 0.563. However, the time-on-task effecton accuracy was revealed to be highly significant, F(4, 36) =14.483, p < 0.001. No significant interaction effect was foundbetween block and time, F(12, 108) = 1.067, p = 0.395. Thetemporal evolution of the response latencies exhibits a reversedpattern, increasing over the course of each block (figure 3).The time-on-task increase in RT was found to be significant

4

J. Neural Eng. 11 (2014) 035009 A Martel et al

Figure 3. Mean response latencies against time with the moving average of period 10. The vertical dash–dot lines mark the break betweenblocks.

F(4, 36) = 3.458, p = 0.017. The starting latencies of eachblock did not exhibit an increment and accordingly the effectwas not revealed to be significant F(1.416, 12.747) = 0.729,p = 0.456. No significant interaction was found, F(12, 108) =0.485, p = 0.919.

3.3. ERP results

Each stimulus, i.e. tick, elicited three contiguous waves, aN1, a N2pc and a P3. Previous work has identified theN2pc as indexing covert visual-spatial attention (Luck andHillyard 1994, Woodman and Luck 2003, Robitaille andJolicoeur 2006) with enhanced amplitude over the contralateralhemisphere to the visual hemifield in which the stimuliis presented (see supplementary figure 2, available fromstacks.iop.org/JNE/11/035009/mmedia). Figure 4 illustratesthe spatial distribution and the temporal dynamics of the N1,N2pc and P3, which peaked at around 150, 185 and 240 msrelative to stimulus onset, respectively. While, on average,the first two waves demonstrated only slight deviations acrossclasses (with a maximum sgn r2 value of −0.002), the P3described a distinctively stronger positive deflection for fronto-central channels in hit preceding trials, with sgn r2-valuesreaching up to 0.0048.

Figure 5 depicts the grand average ERP waveforms of theCz electrode for the seven hit- and miss-preceding trials. Scalptopographies of the inter-class contrast expressed in sgn r2 aregiven for the highlighted peaks of the P3. Visual inspectionreveals a gradually emerging gap between pre-miss and pre-hitP3 peaks, beginning six trials before the target. The noticeablereduction in P3 amplitude anteceding misses is also reflectedby the inter-class contrast values reaching up to 0.0042.

The results of the linear regression for the difference inpeak P3 amplitude for the 235–255 ms relative to tick onset

interval are depicted in figure 6. While the positive correlationbetween P3 peak amplitude differences and pre-trials failedto reach significance for the 200–300 ms time interval(r = 0.193, p = 0.093), it reached statistical significance(r = 0.257, p = 0.024) for the 235–255 ms interval whichmore accurately encloses the moment P3 amplitude reachesits maximum.

The amplitudinal P3 difference between hits and missesamounted, on average, to 0.97 μV ( ± 1.35) for trialsimmediately prior to targets. Correspondingly, an increasinglywidening gap of 0.163 μV ( ± 0.07) was estimated across thepre-trials, emerging six trials before the target trial.

3.4. Amplitude spectrum results

Figure 7 illustrates the spatial distribution of relative changes inalpha activity for each trial of the 9 s time window immediatelypreceding a target trial. Additionally the scalp topographiesof the resulting inter-class contrast are shown expressed interms of sgn r2. The pre-miss trials reveal an alpha-increaseover parieto-occipital areas, in particular at electrode POz,which gradually amplifies when approaching a target trial.In comparison, pre-hit trials show lower alpha activity overparieto-occipital areas and do not reveal inter-trial variability.The results of the linear regression for the channel POz aredepicted in figure 8. The negative correlation between alphaspectral amplitude differences measured over POz and pre-trials did reach statistical significance (r = −0.193, p = 0.043).

4. Discussion

The present behavioural results, i.e. the increase in subjectivesleepiness, the significant decrease in accuracy and increase

5

J. Neural Eng. 11 (2014) 035009 A Martel et al

Figure 4. Upper plot shows grand average ERP waveforms for the four trials preceding hits or misses for channels Cz, Fz, PO7 and PO8.Three intervals were chosen enclosing the peaks of the CNV, N2pc and P3 and their averages illustrated as scalp topographies for each class.The bottom row shows the topography of the resulting class differences, depicted in terms of sgn r2-values.

Figure 5. Upper plot shows the grand average ERP waveforms for the 7200 ms preceding hits and misses for the channel Cz. Seven intervalswere manually chosen enclosing the peak of the P3 of each trial. The bottom row shows the topography of the class differences, depicted interms of sgn r2-values.

6

J. Neural Eng. 11 (2014) 035009 A Martel et al

Figure 6. Scatter plot showing the hits–misses amplitude differences of the P3 peak amplitude extracted from channel Cz over the interval235–255 ms following stimulus presentation for each subject against the pre-trials. The line represents the best fitting curve.

Figure 7. Two upper rows show the spatial distribution of the alpha activity for nine pre-hit and pre-miss trials for the interval 0–1000 msfollowing presentation of the stimulus. The bottom row shows the topography of the resulting class differences, depicted in terms of sgnr2-values.

in RT as a function of time-on-task are characteristic ofa vigilance decrement (Davies and Parasuraman 1982,Hester et al 2005, Helton and Russell 2011). The relativelygood and stable accuracy of block 1 is most likely due to thenovelty of the task. On average, the excellent starting accuracydecreased dramatically within the first 5 min. The appearanceof a vigilance decrement in such a short time frame can beattributed to the difficulty of the discrimination of low saliencystimuli with covert attention and to the relatively high stimulus

rate discrimination which is consistent with previous studies(Nuechterlein et al 1983, Tiwari et al 2009). The mechanismsunderlying the vigilance decrement are still heavily debatedupon (Allan Cheyne et al 2009, Ariga and Lleras 2011,Bonnefond et al 2010, Greene et al 2009, Grier et al 2003,Helton and Warm 2008, MacLean et al 2009, Pattyn et al2008, Robertson et al 1997, Sawin and Scerbo 1995, Warmet al 2008). Although it was not the aim of the present studyto adjudicate between the two predominant theories, the

7

J. Neural Eng. 11 (2014) 035009 A Martel et al

Figure 8. Scatter plot shows the hits–misses alpha-activity differences extracted from channel POz over the interval 0–1000 ms followingstimulus presentation for each subject against the pre-trials. The line represents the best fitting curve.

mindlessness and the resource theory, and therefore workloadwas not assessed, our results seem to be in accordance withthe over-load theory. While it could be argued that the drop inaccuracy reflects a growing indifference towards the task, theparticipants were well aware that the task would remain thesame and it therefore seems more likely that the high startingaccuracies of blocks 2, 3 and 4 are the result of the short restingperiod allowing attentional resources to replenish rather thana renewed interest in the task. This observation is in line withthe results of the performance monitoring study by Blankertzet al (2010). A slight gradual decline in starting accuracy canbe observed across blocks and while this effect was not foundto be significant it might indicate that recuperation took placebut was limited and fatigue accumulated over time. Addition-ally, the significant increase of subjective is consistent withprevious reports examining the subjective impact of sustainedattention tasks and has been proposed to be the result of adrain of cognitive resources due to prolonged attentional effort(Grier et al 2003). One of the advantages of the paradigm usedhere is the close semblance of the regular stimuli with thetarget stimuli imposing high demands since the task requiresthe focused and continuous attention of the subject. Moreover,misses due to perceptual difficulties can be ruled out since allparticipants successfully performed a short training task with100% accuracy. The P3 amplitude has been shown to indexthe top–down attentional enhancement of stimuli (Hopfingerand West 2006), the cognitive evaluation of the visual stimuli(Patel 2005) as well as resource allocation of attention(Isreal et al 1980, Kramer et al 1985, Wickens et al 1983).Consequently, the gradual attenuation of the P3 preceding

lapses observed here, suggests a retraction or exhaustion ofavailable attentional resources. In light of these results it seemsreasonable to assume that the vigilance decrement was theresult of a shortfall in attentional resources brought about bythe high mental demands of a constant endogenously-drivenredeployment of attention. It should be noted that whetherthis assumption is valid or not does not bear fundamental im-plications for the present work as lapses of attention resultingfrom workload or boredom still represents a safety concernin real-world settings. It may, however, be of consequence forthe selection and implementation of adaptive measures mostappropriate to mitigate the vigilance decrement.

The present study delivers the first evidence that taskdisengagement can be predicted up to 10 s in advance in acovert sustained attention task, reinforcing the feasibility of aBCI-based human–machine system (HMS) able to provideeffective countermeasures to lapses in visually complexenvironments in which low-salient stimuli hold safety-criticalinformation (e.g. multi-monitor surveillance, airport luggagescreening, medical screenings, long-distance driving andcockpit monitoring). Two predictors of lapses of attentioncould be identified, an increase in α-activity and a gradualreduction in P3 amplitude developing 9 to 10 s and 6 to7 s before a lapse, respectively. Interestingly, the predictiveneurophysiological signals were extracted from only twochannels and considering the tedious process of setting upnumerous electrodes, a system requiring only a small subsetwould represent an improvement in ergonomics and increasethe likelihood of adoption of BCI technology for operationalapplications.

8

J. Neural Eng. 11 (2014) 035009 A Martel et al

These results strongly corroborates the findings ofO’Connell et al (2009), both the P3 component and alpha-band activity were found to differentiate hits from misses.Analogously the strongest differences in EEG signals werefound over central channels for the P3 and over parieto-occipital channels for alpha-activity. Previous work has linkedincreased alpha activity with cortical idling, i.e. a decrease ofmeaningful activity or cognitive processing (Pfurtscheller et al1996, Mantini et al 2007). As noted by O’Connell et al (2009)idling of the visual cortex preceding misses would imply areduced deployment of visual attention. However, this standsin contradiction with the observed dynamics of the N2pc,which is hypothesized to reflect target enhancement (Eimer1996), i.e. selection of a target stimulus relevant to the currenttask. The quasi identical temporal and spatial distributions ofthe N1 and N2pc across classes suggest that bottom-up visualprocessing was not attenuated preceding a miss. This, in turn,leads to the same conclusion as O’Connell et al (2009), thatthe maladaptive increase in α-amplitude observed here doesnot reflect the idling of visual regions but rather of regionsassociated with monitoring and attentional processes. This isconsistent with reports of Muller et al (2008) where a left-occipital increase of alpha activity was linked with decreasedconcentration in a sustained visual discrimination task. Thetaactivity (4–8 Hz) was also investigated since it has been shownto correlate with accuracy during a monitoring task (Beattyet al 1974) but did not exhibit any interclass differences inthe present study. While previous studies (Smallwood et al2008, O’Connell et al 2009) and the present results identifythe P3 as the strongest predictor of an inadequate level ofdeficient of attention, its use in a real-world environment islimited. As pointed out by Berka et al (2007) it would requirethe introduction of a recurrent artificial stimulus to elicit anelectrophysiological marker of the current attentional state ofthe operator. Many applications are still conceivable especiallyin controlled environments such as simulated or training taskswith neuro-feedback to enhance sustained attention (Friel2007, O’Connell et al 2008). This limitation and the resultsof the present study hint towards alpha as the best predictorfor passively monitoring attentional levels without the needto alter the HMI. The single-channel analysis presented hererevealed gradual hit–miss differences in alpha activity. Whilebeing statistically significant, the effect is comparably smallin magnitude. Multivariate analysis using SPoC (Dahne et al2013) is currently under way, promising to amplify the effectand thus yielding improved predictability. Further researchto disentangle the specific neural mechanisms reflected byfluctuations in the alpha-frequency range would be beneficialas it has been found to reflect a wide range of different cognitiveprocesses dependent on the task (Palva and Palva 2007, Minet al 2008, Deiber et al 2012, Bonnefond and Jensen 2013).

Despite the high variability exhibited byneurophysiological signals, machine learning and adaptivesignal processing techniques have rendered the single trialanalyses of EEG data in real-time operational conditionspossible (Muller et al 2008). As exemplified by the growingnumber of studies demonstrating the feasibility of monitoringbrain states in real-time and adapting the HCI accordingly

(Chavarriaga and del Millan 2010, Freeman et al 2004,Gangadhar et al 2009, Kohlmorgen et al 2007, Muller et al2008, Papadelis et al 2007, Schmidt et al 2012) it would be ofutmost interest, based on the present findings, to investigatewhether the propensity towards task disengagement can bedetected online on a single-trial basis. Future studies could thenaddress which types of adaptive measures are most effectiveat preventing the vigilance decrement or generate novel formof self-training in the form of neuro-feedback applications.

References

Allan Cheyne J, Solman G J, Carriere J S and Smilek D 2009Anatomy of an error: a bidirectional state model of taskengagement/disengagement and attention-related errorsCognition 111 98–113

Ariga A and Lleras A 2011 Brief and rare mental ‘breaks’ keep youfocused: deactivation and reactivation of task goals preemptvigilance decrements Cognition 118 439–43

Beatty J, Greenberg A, Deibler W P and O’Hanlon J F 1974Operant control of occipital theta rhythm affects performancein a radar monitoring task Science 183 871–3

Berka C, Levendowski D J, Lumicao M N, Yau A, Davis G,Zivkovic V T, Olmstead R E, Tremoulet P D and Craven P L2007 EEG correlates of task engagement and mental workloadin vigilance, learning, and memory tasks Aviat. Space Environ.Med. 78 (Suppl. 5) B231–B244

Blankertz B, Lemm S, Treder M, Haufe S and Muller K-R 2011Single-trial analysis and classification of ERP components–atutorial Neuroimage 56 814–25

Blankertz B et al 2010 The Berlin brain–computer interface:non-medical uses of BCI technology Front. Neurosci. 4 198

Bonnefond A, Doignon-Camus N, Touzalin-Chretien Pand Dufour A 2010 Vigilance and intrinsic maintenance ofalert state: an ERP study Behav. Brain Res. 211 185–90

Bonnefond M and Jensen O 2013 The role of gamma and alphaoscillations for blocking out distraction Commun. Integr. Biol.6 e22702

Chavarriaga R and del Millan J R 2010 Learning from EEGerror-related potentials in noninvasive brain-computerinterfaces IEEE Trans. Neural Syst. Rehabil. Eng. 18 381–8

Dahne S, Meinecke F C, Haufe S, Hohne J, Tangermann M,Muller K-R and Nikulin V V 2013 SPoC: a novel frameworkfor relating the amplitude of neuronal oscillations tobehaviorally relevant parameters Neuroimage (in press)

Davies D R and Parasuraman R 1982 The Psychology of Vigilance(London: Academic)

Deiber M-P, Sallard E, Ludwig C, Ghezzi C, Barral J and Ibanez V2012 EEG alpha activity reflects motor preparation rather thanthe mode of action selection Front. Integr. Neurosci. 6 59

Eichele H, Juvodden H T, Ullsperger M and Eichele T 2010Mal-adaptation of event-related EEG responses precedingperformance errors Front. Hum. Neurosci. 4 65

Eimer M 1996 The N2pc component as an indicator of attentionalselectivity Electroencephalogr. Clin. Neurophysiol. 99 225–34

Freeman F G, Mikulka P J, Scerbo M W and Scott L 2004 Anevaluation of an adaptive automation system using a cognitivevigilance task Biol. Psychol. 67 283–97

Friel P N 2007 EEG biofeedback in the treatment of attention deficithyperactivity disorder Altern. Med. Rev. 12 146–51

Gale A 1977 Some EEG correlates of sustained attention Vigilanceed R R Mackie (Boston, MA: Springer) pp 263–83

Gangadhar G, Chavarriaga R and Millan J D R 2009 Fastrecognition of anticipation-related potentials IEEE Trans.Biomed. Eng. 56 1257–60

9

J. Neural Eng. 11 (2014) 035009 A Martel et al

Greene C M, Bellgrove M A, Gill M and Robertson I H 2009Noradrenergic genotype predicts lapses in sustained attentionNeuropsychologia 47 591–4

Grier R A, Warm J S, Dember W N, Matthews G, Galinsky T L,Szalma J L and Parasuraman R 2003 The vigilance decrementreflects limitations in effortful attention, not mindlessnessHum. Factors 45 349–59

Grubb J and Cohn J 2012 The evolution of human systems: a briefoverview Neuroadaptive Systems, Ergonomics Design &Mgmt. Theory & Applications ed D Schmorrow (Boca Raton,FL: CRC Press) pp 60–6

Hancock P A and Hart S G 2002 Defeating terrorism: what canhuman factors/ergonomics offer? Ergon. Des. 10 6–16

He B, Gao S, Yuan H and Wolpaw J R 2013 Brain–computerinterfaces Neural Engineering ed B He (Boston, MA: Springer)pp 87–151

Helton W S and Russell P N 2011 Feature absence–presence andtwo theories of lapses of sustained attention Psychol. Res.75 384–92

Helton W S and Warm J S 2008 Signal salience and themindlessness theory of vigilance Acta Psychol. 129 18–25

Hester R, Foxe J J, Molholm S, Shpaner M and Garavan H 2005Neural mechanisms involved in error processing: a comparisonof errors made with and without awareness Neuroimage27 602–8

Hoddes E, Zarcone V, Smythe H, Phillips R and Dement W C 1973Quantification of sleepiness: a new approach Psychophysiology10 431–6

Hopfinger J B and West V M 2006 Interactions betweenendogenous and exogenous attention on cortical visualprocessing Neuroimage 31 774–89

Hutchison D et al 2011 Foundations of augmented cognitionDirecting the Future of Adaptive Systems vol 6780 (Berlin:Springer)

Isreal J B, Chesney G L, Wickens C D and Donchin E 1980 P300and tracking difficulty: evidence for multiple resources indual-task performance Psychophysiology 17 259–73

Janke W 1976 Fragebogen zur Erfassung der Ausgangslage (FAL)Dusseldorf: Institut fur Psychologie II derHeinrich-Heine-Universitat.

Kohlmorgen J, Dornhege G, Braun M, Blankertz B, Muller K-Rand Curio G 2007 Improving human performance in a realoperating environment through real-time mental workloaddetection Toward Brain-Computer Interfacinged G Dornhege, J d R Millan, T Hinterberger, D J McFarlandand K-R Muller (Cambridge, MA: MIT Press) pp 409–22

Kramer A F, Wickens C D and Donchin E 1985 Processing ofstimulus properties: evidence for dual-task integrality J. Exp.Psychol.: Hum. Percept. Perform. 11 393–408

Luck S J, Fan S and Hillyard S A 1993 Attention-related modulationof sensory-evoked brain activity in a visual search task J. Cogn.Neurosci. 5 188–95

Luck S J and Hillyard S A 1994 Electrophysiological correlates offeature analysis during visual search Psychophysiology31 291–308

Mackworth J F 1968 Vigilance, arousal and attention Psychol. Rev.75 308–22

Mackworth N H 1948 The breakdown of vigilance during prolongedvisual search Q. J. Exp. Psychol. 1 6–21

MacLean K A, Aichele S R, Bridwell D A, Mangun G R,Wojciulik E and Saron C D 2009 Interactions betweenendogenous and exogenous attention during vigilance Atten.Percept. Psychophys. 71 1042–58

Mantini D, Perrucci M G, Del Gratta C, Romani G L and Corbetta M2007 Electrophysiological signatures of resting state networksin the human brain Proc. Natl Acad. Sci. USA 104 13170–5

Martinez-Conde S, Macknik S L, Troncoso X G and Dyar T A 2006Microsaccades counteract visual fading during fixation Neuron49 297–305

Matthews G 2000 Human Performance: Cognition, Stress,and Individual Differences (Philadelphia, PA: Taylor andFrancis)

Min B-K, Park J Y, Kim E J, Kim J I, Kim J-J and Park H-J 2008Prestimulus EEG alpha activity reflects temporal expectancyNeurosci. Lett. 438 270–4

Muller K-R, Tangermann M, Dornhege G, Krauledat M,Curio G and Blankertz B 2008 Machine learning for real-time single-trial EEG-analysis: from brain-computerinterfacing to mental state monitoring J. Neurosci. Methods167 82–90

Nuechterlein K, Parasuraman R and Jiang Q 1983 Visual sustainedattention: image degradation produces rapid sensitivitydecrement over time Science 220 327–9

O’Connell R G, Bellgrove M A, Dockree P M, Lau A, Fitzgerald Mand Robertson I H 2008 Self-alert training: volitionalmodulation of autonomic arousal improves sustained attentionNeuropsychologia 46 1379–90

O’Connell R G, Dockree P M, Robertson I H, Bellgrove M A,Foxe J J and Kelly S P 2009 Uncovering the neural signature oflapsing attention: electrophysiological signals predict errors upto 20 s before they occur J. Neurosci. 29 8604–11

Palva S and Palva J M 2007 New vistas for alpha-frequency bandoscillations Trends Neurosci. 30 150–8

Papadelis C, Chen Z, Kourtidou-Papadeli C, Bamidis P D,Chouvarda I, Bekiaris E and Maglaveras N 2007 Monitoringsleepiness with on-board electrophysiological recordings forpreventing sleep-deprived traffic accidents Clin. Neurophysiol.118 1906–22

Parasuraman R and Davies D R 1975 Response and evokedpotential latencies associated with commission errors in visualmonitoring Percept. Psychophys. 17 465–8

Parasuraman R and Davies D R 1977 A taxonomic analysis ofvigilance performance Vigilance ed R R Mackie (Boston, MA:Springer) pp 559–74

Parasuraman R and Wilson G F 2008 Putting the brain to work:neuroergonomics past, present, and future Hum. Factors50 468–74

Patel S H 2005 Characterization of N200 and P300: selected studiesof the event-related potential Int. J. Med. Sci. 2 147

Pattyn N, Neyt X, Henderickx D and Soetens E 2008Psychophysiological investigation of vigilance decrement:boredom or cognitive fatigue? Physiol. Behav. 93 369–78

Pfurtscheller G, Stancak A and Neuper C 1996 Event-relatedsynchronization (ERS) in the alpha band—anelectrophysiological correlate of cortical idling: a review Int. J.Psychophysiol. 24 39–46

Polich J 2007 Updating P300: an integrative theory of P3a and P3bClin. Neurophysiol. 118 2128–48

Riggs L A, Ratliff F, Cornsweet J C and Cornsweet T N M 1953The disappearance of steadily fixated visual test objects J. Opt.Soc. Am. 43 495

Ritter W and Vaughan H G 1969 Averaged evoked responses invigilance and discrimination: a reassessment Science164 326–8

Robertson I H, Manly T, Andrade J, Baddeley B T and Yiend J 1997‘Oops!’: performance correlates of everyday attentionalfailures in traumatic brain injured and normal subjectsNeuropsychologia 35 747–58

Robitaille N and Jolicoeur P 2006 Fundamental properties of theN2pc as an index of spatial attention: effects of masking Can.J. Exp. Psychol. 60 101–11

Sarter N B, Mumaw R J and Wickens C D 2007 Pilots’ monitoringstrategies and performance on automated flight decks: anempirical study combining behavioral and eye-tracking dataHum. Factors 49 347–57

Sawin D A and Scerbo M W 1995 Effects of instruction type andboredom proneness in vigilance: implications for boredom andworkload Hum. Factors 37 752–65

10

J. Neural Eng. 11 (2014) 035009 A Martel et al

Schmidt N M, Blankertz B and Treder M S 2012 Online detection oferror-related potentials boosts the performance of mentaltypewriters BMC Neurosci. 13 19

Sheridan T B 2002 Humans and Automation: System Design andResearch Issues (Santa Monica, CA: Human Factors andErgonomics Society)

Smallwood J, Beach E, Schooler J W and Handy T C 2008 GoingAWOL in the brain: mind wandering reduces cortical analysisof external events J. Cogn. Neurosci. 20 458–69

Stanney K 2012 Augmenting human performance NeuroadaptiveSystems, Ergonomics Design & Mgmt. Theory & Applicationsvol 20125874 ed D Schmorrow (Boca Raton, FL: CRC Press)

Tiwari T, Singh A L and Singh I L 2009 Task demand andworkload: effects on vigilance performance and stress J. IndianAcad. Appl. Psychol. 35 265–75

Venthur B, Scholler S, Williamson J, Dahne S, Treder M S,Kramarek M T, Muller K-R and Blankertz B 2010Pyff—a pythonic framework for feedback applications andstimulus presentation in neuroscience Front. Neurosci.4 179

Warm J S, Parasuraman R and Matthews G 2008 Vigilance requireshard mental work and is stressful Hum. Factors 50 433–41

Wickens C, Kramer A, Vanasse L and Donchin E 1983 Performanceof concurrent tasks: a psychophysiological analysis of thereciprocity of information-processing resources Science221 1080–2

Wolpaw J R, Birbaumer N, McFarland D J, Pfurtscheller Gand Vaughan T M 2002 Brain-computer interfaces forcommunication and control Clin. Neurophysiol. 113 767–91

Woodman G F and Luck S J 2003 Serial deployment of attentionduring visual search J. Exp. Psychol.: Hum. Percept. Perform.29 121–38

Zander T O and Jatzev S 2012 Context-aware brain–computerinterfaces: exploring the information space of user, technicalsystem and environment J. Neural Eng. 9 16003

Zander T O, Kothe C, Jatzev S and Gaertner M 2010 Enhancinghuman-computer interaction with input from active andpassive brain-computer interfaces Human-ComputerInteraction Series ed D S Tan and A Nijholt (London:Springer) pp 181–99

11