1

EEG-informed attended speaker extraction fromrecorded speech mixtures with application in

neuro-steered hearing prosthesesSimon Van Eyndhoven, Tom Francart, and Alexander Bertrand, Member, IEEE .

Abstract—Objective: We aim to extract and denoisethe attended speaker in a noisy, two-speaker acousticscenario, relying on microphone array recordings froma binaural hearing aid, which are complemented withelectroencephalography (EEG) recordings to infer thespeaker of interest. Methods: In this study, we proposea modular processing flow that first extracts the twospeech envelopes from the microphone recordings, thenselects the attended speech envelope based on the EEG,and finally uses this envelope to inform a multi-channelspeech separation and denoising algorithm. Results:Strong suppression of interfering (unattended) speechand background noise is achieved, while the attendedspeech is preserved. Furthermore, EEG-based auditoryattention detection (AAD) is shown to be robust to theuse of noisy speech signals. Conclusions: Our resultsshow that AAD-based speaker extraction from micro-phone array recordings is feasible and robust, evenin noisy acoustic environments, and without access tothe clean speech signals to perform EEG-based AAD.Significance: Current research on AAD always assumesthe availability of the clean speech signals, which limitsthe applicability in real settings. We have extended thisresearch to detect the attended speaker even when onlymicrophone recordings with noisy speech mixtures areavailable. This is an enabling ingredient for new brain-computer interfaces and effective filtering schemes inneuro-steered hearing prostheses. Here, we providea first proof of concept for EEG-informed attendedspeaker extraction and denoising.

Index Terms—EEG signal processing, speechenhancement, auditory attention detection, brain-computer interface, auditory prostheses, blind sourceseparation, multi-channel Wiener filter

This work was carried out at the ESAT Laboratory of KU Leuven,in the frame of KU Leuven Research Council BOF/STG-14-005,CoE PFV/10/002 (OPTEC), Research Projects FWO nr. G.0931.14‘Design of distributed signal processing algorithms and scalable hard-ware platforms for energy-vs-performance adaptive wireless acousticsensor networks’, and HANDiCAMS. The project HANDiCAMSacknowledges the financial support of the Future and EmergingTechnologies (FET) programme within the Seventh Framework Pro-gramme for Research of the European Commission, under FET-Opengrant number: 323944. The scientific responsibility is assumed by itsauthors.

S. Van Eyndhoven and A. Bertrand are with KU Leuven,Department of Electrical Engineering (ESAT), StadiusCenter for Dynamical Systems, Signal Processing and DataAnalytics, Kasteelpark Arenberg 10, box 2446, 3001 Leuven,Belgium (e-mail: simon.vaneyndhoven@esat.kuleuven.be,alexander.bertrand@esat.kuleuven.be). T. Francart is with KULeuven, Department of Neurosciences, Research Group ExperimentalOto-rhino-laryngology (e-mail: tom.francart@med.kuleuven.be).

I. Introduction

In order to guarantee speech intelligibility in a noisy,multi-talker environment, often referred to as a ‘cocktailparty scenario’, hearing prostheses can greatly benefitfrom effective noise reduction techniques [1], [2]. Whilenumerous and successful efforts have been made to achievethis goal, e.g. by incorporating the recorded signals ofmultiple microphones [2]–[4], many of these solutionsstrongly rely on the proper identification of the targetspeaker in terms of voice activity detection (VAD). In anacoustic scene with multiple competing speakers, this is ahighly non-trivial task, complicating the overall problemof noise suppression. Even when a good speaker separationis possible, a fundamental problem that appears in suchmulti-speaker scenarios is the selection of the speaker ofinterest. To make a decision, heuristics have to be used,e.g., selecting the speaker with highest energy, or thespeaker in the frontal direction. However, in many real-life scenarios, such heuristics fail to adequately select theattended speaker.

Recently however, auditory attention detection (AAD)has become a popular topic in neuroscientific and audio-logical research. Different experiments have confirmed thefeasibility of a decoding paradigm that, based on record-ings of brain activity such as the electroencephalogram(EEG), detects to which speaker a subject attends in anacoustic scene with multiple competing speech sources [5]–[10]. A major drawback of all these experiments is thatthey place strict constraints on the methodological design,which limits the practical use of their results. More pre-cisely all of the proposed paradigms employ the separate‘clean’ speech sources that are presented to the subjects(to correlate their envelopes to the EEG data), a conditionwhich is never met in realistic acoustic applications suchas hearing prostheses, where only the speech mixtures asobserved by the device’s local microphone(s) are available.In [11] it is reported that the detection performance dropssubstantially under the effect of crosstalk or uncorrelatedadditive noise on the reference speech sources that are usedfor the auditory attention decoding. It is hence worthwhileto further investigate AAD that is based on mixturesof the speakers, such as in the signals recorded by themicrophones of a hearing prosthesis.

Nonetheless, devices such as neuro-steered hearing pros-theses or other brain-computer interfaces (BCIs) thatimplement AAD, can only be widely applied in realistic

This paper is published in IEEE Transactions on Biomedical Engineering (2016) and is under copyright. Please cite this paper as:S. Van Eyndhoven, T. Francart, and A. Bertrand, ”EEG-informed attended speaker extraction from recorded speech mixtures withapplication in neuro-steered hearing prostheses”, IEEE Trans. Biomedical Engineering, vol. x(x), pp. x-x, 2016.

arX

iv:1

602.

0570

2v3

[cs

.SD

] 1

4 Ju

l 201

6

2

scenarios if they can operate reliably in these noisy condi-tions. End users with (partial) hearing impairment couldgreatly benefit from neuro-steered speech enhancementand denoising technologies, especially if they are imple-mented in compact mobile devices. EEG is the preferredchoice for these emerging solutions, due to its cheap andnon-invasive nature [12]–[17]. Many research efforts havebeen focused on different aspects of this modality to enablethe development of small scale, wearable EEG devices.Several studies have addressed the problem of wearabilityand miniaturization [13]–[16], data compression and powerconsumption [16], [17].

In this study, we combine EEG-based auditory atten-tion detection and acoustic noise reduction, to suppressinterfering sources (including the unattended speaker)from noisy multi-microphone recordings in an acousticscenario with two simultaneously active speakers. Ouralgorithm enhances the attended speaker, using EEG-informed AAD, based only on the microphone recordingsof a hearing prosthesis, i.e., without the need for the cleanspeech signals1. The final goal is to have a computationallycheap processing chain that takes microphone and EEGrecordings from a noisy, multi-speaker environment atits input and transforms these into a denoised audiosignal in which the attended speaker is enhanced, andthe unattended speaker is suppressed. To this end, wereuse experimental data from the AAD experiment in[9] and use the same speech data as in [9] to synthesizemicrophone recordings of a binaural hearing aid, basedon publicly available head-related transfer functions whichwere measured with real hearing aids [18]. As we willshow further on, non-negative blind source separation is aconvenient tool in our approach, as we need to extract thespeech envelopes from the recorded mixtures. To this end,we rely on [19], where a low-complexity source separationalgorithm is proposed that can operate at a sampling ratethat is much smaller than that of the microphone signals,which is very attractive from a computational point ofview. We investigate the robustness of our processingscheme by adding varying amounts of acoustic interferenceand testing different speaker setups.

The outline of the paper is as follows. In section II, wegive a global overview of the problem and an introductionto the different aspects we will address; in section IIIwe explain the techniques for non-negative blind sourceseparation, and cover the extraction of the attended speechfrom (noisy) microphone recordings; in section IV we de-scribe the conducted experiment; in section V we elaborateon the results of our study; in section VI we discuss theseresults and consider future research directions; in sectionVII we conclude the paper.

1We still use clean speech signals to design the EEG decoder inan initial training or calibration phase. However, once this decoderis obtained, our algorithm operates directly on the microphonerecordings, without using the original clean speech signals as side-channel information.

II. Problem statementA. Noise reduction problem

We consider a (binaural) hearing prosthesis equippedwith multiple microphones, where the signal observed bythe i-th microphone is modeled as a convolutive mixture:

mi[t] = (hi1 ∗ s1)[t] + (hi2 ∗ s2)[t] + vi[t] (1)= xi1[t] + xi2[t] + vi[t]. (2)

In (1), mi[t] denotes the recorded signal at microphonei, which is a superposition of contributions xi1[t] andxi2[t] of both speech sources and a noise term vi[t]. xi1[t]and xi2[t] are the result of the convolution of the clean(‘dry’) speech signals s1[t] and s2[t] with the head-relatedimpulse responses (HRIRs) hi1[t] and hi2[t], respectively.These HRIRs are assumed to be unknown and model theacoustic propagation path between the source and thei-th microphone, including head-related filtering effectsand reverberation. The term vi[t] bundles all backgroundnoise impinging on microphone i and contaminating therecorded signal.

Converting (1) to the (discrete) frequency domain, weget

Mi(ωj) = Hi1(ωj)S1(ωj) +Hi2(ωj)S2(ωj) + Vi(ωj) (3)= Xi1(ωj) +Xi2(ωj) + Vi(ωj) (4)

for all frequency bins ωj . In (3), Mi(ωj), S1(ωj), S2(ωj)and Vi(ωj) are representations of the recorded sig-nal at microphone i, the two speech sources and thenoise at frequency ωj , respectively. Hi1(ωj) and Hi2(ωj)are the frequency-domain representations of the HRIRs,which are often denoted as head-related transfer func-tions (HRTFs). All microphone signals and speech con-tributions can then be stacked in vectors M(ωj) =[M1(ωj) . . .MK(ωj)]T , X1(ωj) = [X11(ωj) . . . XK1(ωj)]T

and X2(ωj) = [X12(ωj) . . . XK2(ωj)]T , where K is thenumber of available microphones. Our aim is to enhancethe attended speech component and suppress the inter-fering speech and noise in the microphone signals. Moreprecisely, we arbitrarily select a reference microphone (e.g.r = 1) and, assuming without loss of generality thats1[t] is the attended speech, try to estimate Xr1(ωj)by filtering M(ωj), which is the full set of microphonesignals2. Hereto, a linear minimum mean-squared error(MMSE) cost criterion is used [2], [3]:

J(W(ωj)) = E{|W(ωj)HM(ωj)−Xr1(ωj)|2

}(5)

in which W is a K-channel filter, represented by a K-dimensional complex-valued vector, where the superscriptH denotes to the conjugate transpose. Note that a differentW is selected for each frequency bin, resulting in a spatio-spectral filtering, which is equivalent to a convolutivespatio-temporal filtering when translated to the time-domain. In section III-C, we will minimize (5) by means

2In the case of a binaural hearing prosthesis, we assume thatthe microphone signals recorded at the left and right ear can beexchanged between both devices, e.g., over a wireless link [4].

3

of the so-called multi-channel Wiener filter (MWF).Up to now, it is not known which of the speakers is the

target or attended speaker. To determine this, we need toperform auditory attention detection (AAD), as describedin the next subsection. Furthermore, the MWF paradigmrequires knowledge of the times at which this attendedspeaker is active. To this end, we need a speaker-dependentvoice activity detection (VAD), which will be discussed insubsection III-D. We only have access to the envelopes ofthe microphone signals, which contain significant crosstalkdue to the presence of two speakers. Hence, relying onthese envelopes would lead to suboptimal performance (i.e.misdetections of the VAD), motivating the use of an inter-mediate step to obtain better estimates of these envelopes.As stated, we employ non-negative blind source separationto obtain more accurate estimates of the envelopes, whichwill prove to relax the VAD problem (see III-B).

B. Auditory attention detection (AAD) problem

In (1), either s1[t] or s2[t] can be the attended speech.Earlier studies showed that the low frequency variations ofspeech envelopes (between approximately 1 and 9 Hz) areencoded in the evoked brain activity [20], [21], and thatthis mapping differs whether the speech is attended to bythe subject (or not) in a multi-speaker environment [6]–[8], [22], [23]. This mapping can be reversed to categorizethe attention of a listener from recorded brain activity.In brief, the AAD paradigm works by first training aspatiotemporal filter (decoder) on the recorded EEG datato reconstruct the envelope of the attended speech bymeans of a linear regression [5], [9]–[11]. This decoderwill reconstruct an auditory envelope, by integrating themeasured brain activity across κ channels and for τmaxdifferent lags, described by

sA

[n] =τmax∑τ=0

κ∑k=1

rk[n+ τ ]dk[τ ] (6)

in which rk[n] is the recorded EEG signal at channel k andtime n, dk[τ ] is the decoder weight for channel k at a post-stimulus lag of τ samples, and s

A[n] is the reconstructed

attended envelope at time n. We can rewrite this expres-sion in matrix notation, as s

A= R d, in which s

Ais a vec-

tor containing the samples of the reconstructed envelope,d = [ d0[0] . . . d0[τmax] . . . dκ[0] . . . dκ[τmax] ]T is a vectorwith the stacked spatiotemporal weights, of length chan-nels× lags, and where the matrix with EEG measurementsis structured as R = [ r1 . . . rN ]T , where there is a vectorrn = [ r0[n] . . . r0[n + τmax] . . . rκ[n] . . . rκ[n + τmax]]T forevery sample n = 1 . . . N of the envelope. We find thedecoder by solving the following optimization problem:

d = arg mind‖ s

A− s

A‖2 (7)

= arg mind‖R d− s

A‖2 (8)

in which sA

is the real envelope of the attended speech.Using classical least squares, we compute the decoderweights as

d = (RTR)−1RTsA. (9)

The matrix RTR represents the sample autocorrelationmatrix of the EEG data (for all channels and consideredlags) and RTs

Ais the sample cross-correlation of the EEG

data and the attended speech envelope. Hence, the decoderd is trained to optimally reconstruct the envelope s

Aof the

attended speech sources. If the sample correlation matricesare estimated on too few samples, a regularization termcan be used, like in [10]. As motivated in subsection IV-B,we omitted regularization in this study.

The decoding is successful if the decoder reconstructsan envelope that is more correlated with the envelope ofthe attended speech than with that of the unattendedspeech. Mathematically, this translates to r

A> r

U, in

which rA

and rU

are the Pearson correlation coefficientsof the reconstructed envelope s

Awith the envelopes of

the attended and unattended speech, respectively. In thispaper, rather than requiring the separate speech envelopesto be available, we make the assumption that we only haveaccess to the recorded microphone signals (except for thetraining of the EEG decoder based on (9)). In section III,we address the problem of speech envelope extraction fromthe speech mixtures in the microphone signals, to still beable to perform AAD using the approach explained above.

III. Algorithm pipelineHere, we propose a modular processing flow that com-

prises a number of steps towards the extraction anddenoising of the attended speech, shown as a block di-agram in Fig. 1. We compute the energy envelopes ofthe recorded microphone mixtures (represented by the‘env’-block and explained in subsection III-A) and usethe multiplicative non-negative independent componentanalysis (M-NICA) algorithm to estimate the originalspeech envelopes from these mixtures (subsection III-B).These speech envelopes are fed into the AAD processingblock described in previous subsection, which will indicateone of both as belonging to the attended speaker, basedon the EEG recording (arrows on the right). Voice activitydetection is carried out on the estimated envelopes, andthe VAD track that is selected during AAD serves as inputto the multi-channel Wiener filter (subsection III-D). TheMWF filters the set of microphone mixtures, based on thisVAD track, yielding one enhanced speech signal at theoutput (subsection III-C).

A. Conversion to energy domain (ENV)In order to apply the AAD algorithm described in

subsection II-B, we need the envelopes of the individualspeech sources. Since we are only interested in the speechenvelopes, we will work in the energy domain, allowingto solve a source separation problem at a much lowersampling rate than the original sampling rate of the

4

Fig. 1. Pipeline of the proposed processing flow.

microphone signals. Furthermore, energy signals are non-negative, which can be exploited to perform real-timesource separation based only on second-order statistics[24], rather than higher-order statistics as in many ofthe standard independent component analysis techniques.These two ingredients result in a computationally efficientalgorithm, which is important when it is to be operatedin a battery-powered miniature device such as a hearingprosthesis. A straightforward way to calculate an energyenvelope is by squaring and low-pass filtering a micro-phone signal, i.e., for microphone i this yields the energysignal

Emi [n] = 1T

T∑w=1

mi[nT + w]2 (10)

in which n is the sample index of the energy signal, Tis the number of samples (window length) to compute theshort-time average energy Emi

[n], which estimates the realmicrophone energy, E{m2

i [nT ]}.Based on (1), and assuming the source signals are

independent, we can model the relationship between theenvelopes of the speech sources and the microphone sig-nals as an approximately linear, instantaneous mixture ofenergy signals:

Em[n] ≈ A Es[n] + Ev[n] . (11)

Here, the short-time energies of the K microphone signalsand the S speech sources are stacked in the time-varyingvectors Em[n] and Es[n], respectively, and are relatedthrough the K × S mixing matrix A, defining the overallenergy attenuation between every speech source and everymicrophone. Similarly, the short-term energies of the Nnoise components that contaminate the microphone sig-nals are represented by the vector Ev[n]. For infinitelylarge T and infinitely narrow impulse responses, (11) iseasily shown to be exact. For HRIRs of a finite durationand for finite T , it is a quite rough approximation, butwe found that it still provides a useful basis for thesubsequent algorithm that aims to estimate the originalspeech envelopes from the mixtures, as we succeed toextract the original speech envelopes reasonably well (seenext subsection and section V). The literature also re-ports experiments where the approximation in (11) hassuccesfully been used as a mixing model for separation ofspeech envelopes, even in reverberant environments withlonger impulse responses than the HRIRs that are used

here [19], [25].

B. Speech envelope extraction from mixtures (M-NICA)

The M-NICA algorithm is a technique that exploitsthe non-negativity of the underlying sources [24] to solveblind source separation (BSS) problems in an efficient way.It demixes a set of observed signals, that is the resultof a linear mixing process, into its separate, nonnegativesources. Under the assumption that the source signals areindependent, non-negative, and well-grounded3, it can beshown that a perfect demixing is obtained by a demixingmatrix that decorrelates the signals while preserving non-negativity. Similar to [19], we will employ the M-NICAalgorithm, to find an estimate of Es[n] from Em[n] in (11).The algorithm consists of an iterative interleaved applica-tion of a multiplicative decorrelation step (preserving thenon-negativity), and a subspace projection step (to re-fitthe data to the model). An in-depth description of the M-NICA algorithm is available in [24], which also includesa sliding-window implementation for real-time processing.Attractive properties of M-NICA are that it relies only on2nd order statistics (due to the non-negativity constraints)and that it operates at the low sampling rate of theenvelopes. These features foster the use of M-NICA, asthe algorithm seems to be well matched to the constraintsof the target application, namely the scarce computationalresources and the required real-time operation. Note thatthe number of speech sources must be known a priori.In practice, we could estimate this number by a singularvalue decomposition [19]. We will refer to Em[n] andEs[n] as the microphone envelopes and demixed envelopes,respectively, where ideally Es[n] = Es[n]. As with mostBSS techniques, a scaling and permutation ambiguityremains, i.e., the ordering of the sources and their energycannot be found, since they can be arbitrarily changedif a compensating change is made in the mixing matrix.In real-time, adaptive applications, these ambiguities staymore or less the same as time progresses and are of littleimportance (see [19], where an adaptive implementationof M-NICA is tested on speech mixtures). It is noted that,to perform M-NICA on (11), the matrix A should bewell-conditioned in the sense that it should have at leasttwo singular values that are significantly larger than 0.This means that the energy contribution of each speechsource should be differently distributed over the K mi-crophones. In [19] and [25], this was obtained by placingthe microphone several meters apart, which is not possiblein our application of hearing prostheses. However, we usemicrophones that are on both sides of the head, such thatthe head itself acts as an angle-dependent attenuator foreach speaker location. This results in a different spatialenergy pattern for each speech source and hence in a well-conditioned energy mixing matrix A.

3A signal is well-grounded if it attains zero-valued samples withfinite probability [24].

5

C. Multi-channel Wiener filter (MWF)

For the sake of conciseness, we will omit the frequencyvariable ωj in the remainder of the text. The solution thatminimizes the cost function in (5) is the multi-channelWiener filter W [2]–[4], found as

W = arg minW

E{|WHM−Xr1|2

}(12)

= R−1mmRxx er (13)

= (Rxx + Rvv)−1Rxx er (14)

in which Rmm is the K × K autocorrelation matrixE{MMH} of the microphone signals and Rxx is theK × K speech autocorrelation matrix E{X1XH

1 }, wherethe subscript 1 refers to the attended speech. Likewise,Rvv is the K×K autocorrelation matrix of the undesiredsignal component. Note that the MWF will estimate thespeech signal S1 as it is observed by the selected referencemicrophone, i.e., it will estimate Hr1S1, assuming ther-th microphone is selected as the reference. Hence, eris the r-th column of an identity matrix, which selectsthe r-th column of Rxx corresponding to this referencemicrophone.

The matrix Rxx is unknown, but can be estimated asRxx = Rmm − Rvv, with Rmm the ‘speech plus inter-ference’ autocorrelation matrix, equal to E{MMH} whenmeasuring during periods in which the attended speaker isactive. Likewise, Rvv can be found as E{MMH}, duringperiods when the attended speaker is silent. All of thementioned autocorrelation matrices can be estimated bymeans of temporal averaging in the short-time Fouriertransform domain. Note that more robust ways exist to es-timate Rxx, compared to the straightforward subtractiondescribed here. The MWF implementation we employeduses a generalized eigenvalue decomposition (GEVD) tofind a rank-1 approximation of Rxx as in [3]. The rationalebehind this is that the MWF aims to enhance a singlespeech source (corresponding to the attended speaker)while suppressing all other acoustic sources (other speechand noise). Since Rxx only captures a single speech source,it should have rank 1.

Applying the MWF corresponds to computing (14) andperforming the filtering WHM for each frequency ωjand each time-window in the short-time Fourier domain.Finally, the resulting output in the short-time Fourierdomain can be transformed back to the time domain again.In practice, this is often done using a weighted overlap-add(WOLA) procedure [26].

As mentioned above, when estimating Rxx and Rnn

from the microphone signals M, we rely on a goodidentification of periods or frames in which both (at-tended) speech and interference are present (to estimatethe speech-plus-interference autocorrelation Rmm) versusperiods during which only interference is recorded (toestimate the interference-only correlation Rvv). Makingthis distinction corresponds to voice activity detection,which we discuss next.

D. Voice activity detection (VAD)The short-time energy of a speech signal gives an indica-

tion at what times the target speech source is (in)active. Asimple voice activity detection (VAD) algorithm consistsof thresholding the energy envelope of the target speechsignal. Note that in our target application, the speechenvelopes are also used for AAD. After applying M-NICAon the microphone envelopes, we find two demixed en-velopes, which serve as better estimates of the real speechenvelopes. Based on the correlation with the reconstructedenvelope sA from the AAD decoder in (6), one of thesedemixed envelopes will be identified as the envelope of theattended speech source. This correlation can be computedefficiently in a recursive sliding-window fashion, to updatethe AAD decision over time, which is represented by atime-varying switch in Fig. 1. For each AAD decision,the chosen envelope segment is then thresholded sample-wise for voice activity detection. Ideally, the envelope seg-ments on which the VAD is applied all originate from theattended envelope, although sometimes the unattendedenvelope may be wrongfully selected, depending on theAAD decisions that are made. This will lead to VADerrors, which will have an impact on the denoising andspeaker extraction performance of the MWF.

IV. ExperimentFor every pair of speech sources (1 attended and 1

unattended), we performed the following steps:1) compute the microphone signals, according to (1)2) find the energy-envelope of the microphone signals,

as described in subsection III-A3) demix the microphone envelopes with M-NICA, as

described in subsection III-B4) find the VAD track for the attended speech source,

as described in subsection III-D, based on the resultsof the auditory attention task described in IV-B

5) compute the MWF for the attended speech source,as described in subsection III-C, based on the AAD-selected VAD track from step 4

6) filter the microphone signals with this MWF using aWOLA procedure, to enhance the attended speechsource

Furthermore, we also investigate the overall performanceif step 3 is skipped, i.e., if we use the plain microphoneenvelopes without demixing them with M-NICA. In thatcase, we manually pick the two microphone envelopes thatare already most correlated to either of both speakers.Note that this is a best-case scenario that cannot beimplemented in practice.

A. Microphone recordingsWe synthesized the microphone array recordings usinga public database of HRIRs that were measured usingsix behind-the-ear microphones (three microphones perear) [18]. Each HRIR represents the microphone impulseresponses for a source at a certain azimuthal angle relative

6

to the head orientation and at 3 meters distance fromthe microphone. The HRIRs were recorded in an anechoicroom and had a length of 4800 samples at 48 kHz. Asspeech sources, we used Dutch narrated stories (each witha length of approximately six minutes and a sampling rateof 44.1 kHz), that previously served as the auditory stimuliin the AAD-experiment in [9].

To determine the robustness of our scheme, we includednoise in the acoustic setup. We synthesize the microphonesignals for several speaker positions, ranging from -90◦

to 90◦. The background noise is formed by adding fiveuncorrelated multi-talker noise sources nk[n] at positions−90 ◦, −45 ◦, 0 ◦, 45 ◦ and 90 ◦ and at 3 meters distance,each with a long-term power PNk

= 0.1Ps, in which Psis the long-term power of a single speech source. Notethat these noise sources were not present in the stimuliused in the AAD experiment, and are only added here toillustrate the robustness of M-NICA to a possible noiseterm in (11), and to illustrate the denoising capabilitiesof the MWF. We convolve the two speech signals and fivenoise signals with the corresponding HRIRs to synthesizethe microphone signals described in (1). The term vi[n]thus represents all noise contributions and is calculated as∑k(hik ∗ nk)[n], where the five hik[n] are the HRIRs for

the noise sources.In our study, we evaluate the performance for 12 rep-

resentative setups with varying spatial angle between thetwo speaker locations. Taking 0 ◦ as the direction in frontof the subject wearing the binaural hearing aids, theangular position pairs of the speakers are −90 ◦ and 90 ◦,−75 ◦ and 75 ◦, −90 ◦ and 30 ◦, −60 ◦ and 60 ◦, −90 ◦ and0 ◦, −45 ◦ and 45 ◦, −90 ◦ and −30 ◦, −60 ◦ and 0 ◦, −30 ◦

and 30 ◦, −90 ◦ and −60 ◦, −60 ◦ and −30 ◦, and −15 ◦ and15 ◦.

B. AAD experimentThe EEG data originated from a previous study [9],in which 16 normal hearing subjects participated in anaudiologic experiment to investigate auditory attentiondetection. In every trial, a pair of competing speech stimuli(1 out of 4 pairs of narrated Dutch stories, at a sam-pling rate of 8 kHz) is simultaneously presented to thesubject to create a cocktail party scenario; the cognitivetask requires the subject to attend to one story for thecomplete duration of every trial. We consider a subsetof the experiment in [9], in which the presented speechstimuli have a contribution to each ear - after filteringthem with in-the-ear HRIRs for sources at -90◦ and 90◦ -in order to obtain a dataset of EEG-responses that is morerepresentative for realistic scenarios. That is, both earsare presented with a (different) mixture of both speakers,mimicking the acoustic filtering by the head as if thespeakers were located left and right of the subject. Forevery trial, the recorded EEG is then sliced in frames of 30seconds, followed by the training of the AAD decoder anddetection of the attention for every frame, in a leave-one-frame-out cross-validation fashion. We use the approachof [9], where a single decoder is estimated by computing

(9) once over the full set of training frames, i.e., a singleRTR and RTs

Amatrix is calculated over all samples in the

training set. This is opposed to the method in [5], wherea decoder is estimated for each training frame separately,and the averaged decoder is then applied to the test frame.In [9], it was demonstrated that this approach is sensitiveto a manually tuned regularization parameter and mayaffect performance, which is why we opted for the formermethod. The performance of the decoders depends on themethod of calculating the envelope sA of the attendedspeech stimulus. In [9], it was found that amplitude en-velopes lead to better results than energy envelopes. Forthe present study, we work with energy envelopes (asdescribed in subsection III-A) and take the square rootto convert to amplitude envelopes, when computing thecorrelation coefficients in the AAD task.

The present study inherits the recorded EEG datafrom the experiment described above, and assumes thatdecoders can be found during a supervised training phasein which the clean speech stimuli are known4. Throughoutour experiment, we train the decoders per individualsubject on the EEG data and the corresponding envelopesegments of the attended speech stimuli, calculated bytaking the absolute value of the original speech signals andfiltering between 1 and 9.5 Hz (equiripple finite impulseresponse filter, -3 dB at 0.5 and 10 Hz). Contrary to [5],attention during the trials was balanced over both ears, sothat no ear-specific biasing could occur during training ofthe decoder.

The trained decoder can then be used to detect to whichspeaker a subject attends, as explained in subsection II-B.We perform the auditory attention detection procedurewith the same recorded EEG data (using leave-one-frame-out cross-validation) which is fed through the pre-traineddecoder, and then correlated with different envelopes toeventually perform the detection over frames of 30 seconds.In order to assess the contribution of the M-NICA algo-rithm to the overall performance, we consider two options:either the two demixed envelopes or the two microphoneenvelopes that have the highest correlation with eitherof the speech sources’ envelopes are correlated to theEEG decoder’s output s

A. The motivation for the latter

option is that in some microphones, one of both speechsources will be prevalent, and we can take the envelopeof such a microphone signal as a (poor) estimate of theenvelope of that speech source. This will lead to the best-case performance that can be expected with the use ofenvelopes of the microphones, without using an envelopedemixing algorithm.

C. Preprocessing and parameter selectionSpeech fragments are normalized over the full length

to have equal energy. All speech sources and HRIRs wereresampled to 16 kHz, after which we convolved thempairwise and added the resulting signals to find the setof microphone signals.

4Note that in a real device, only one final decoder would need tobe available (obtained after a training phase).

7

The window length T in (10) is chosen so that the energyenvelopes are sampled at 20 Hz. To find the short-termamplitude in a certain bandwidth, we take the squareroot of all energy-like envelopes and filter them between 1and 9.5 Hz before employing them to decode attention inthe EEG epochs. Likewise, all κ = 64 EEG channels arefiltered in this frequency range and downsampled to 20 Hz.As in [5], τmax in (6) is chosen so that it corresponds to250 ms poststimulus. For a detailed overview of the dataacquisition and EEG decoder training, we refer to [9].

VAD tracks for the envelopes of both the attendedand unattended speech are binary triggers (‘on’ or ‘off’),that are 1 when the energy envelope surpasses the chosenthreshold. The value for this threshold was determined asthe one that would lead to the highest median SNR at theMWF output, for a virtual subject with an AAD accuracyof 100% and in the absence of noise sources. After ex-haustive testing, this value was set to 0.05 max

{Es

}and

0.10 max {Em} for the demixed and microphone envelopes,respectively (see subsection V-D). We form one hybridVAD track by selecting and concatenating segments of30 seconds of these two initial tracks, according to theAAD decision that was made in the same 30-secondtrial of the experiment, as described in subsection IV-B.This corresponds to a non-overlapping sliding windowimplementation with a window length of 30 seconds (notethat the AAD decision rate can be increased by using anoverlapping sliding window with a window shift that issmaller than the window length). Thus, this overall VADtrack, which is an input to the MWF, follows the switchingbehavior of the AAD-driven module shown in Fig. 1.

The MWF is applied on the binaural set of six micro-phone signals (resampled to 8 kHz, conform to the pre-sented stimuli in the EEG experiment), through WOLAfiltering with a square-root Hann window and FFT-lengthof 512. Likewise, the VAD track is expanded to match thisnew sample frequency.

For this initial proof of concept, both M-NICA and theMWF are applied in batch mode on the signals, meaningthat the second-order signal statistics are measured overthe full signal length. In practice, an adaptive implemen-tation will be necessary, which is beyond the scope ofthis paper. However, performance of M-NICA and MWFunder adaptive sliding-window implementations have beenreported in [24], [26], where a significant - but acceptable -performance decrease is observed due to the estimation ofthe second-order statistics over finite windows. Therefore,the reported results in this paper should be interpretedas upper limits for the achievable performance with anadaptive system. For envelope demixing, 100 iterations ofM-NICA are used.

V. ResultsA. Performance measures

The microphone envelopes at the algorithm’s input haveconsiderable contributions of both speech sources. What isdesired - as well for the VAD block as for the AAD block

- is a set of demixed envelopes that are well-separated inthe sense that each of them only tracks the energy of asingle speech source, and thus has a high correlation withonly one of the clean speech envelopes, and a low residualcorrelation with the other clean speech envelope. Hence,we adopt the following measure: ∆r

HLis the difference

rH−r

Lbetween the highest Pearson correlation that exists

between a demixed or microphone envelope and a speechenvelope and the lowest Pearson correlation that is foundbetween any other envelope and this speech envelope. E.g.for speech envelope 1, if the envelope of microphone 3has the highest correlation with this speech envelope, andthe envelope of microphone 5 has the lowest correlation,we assign these correlations to r

Hand r

L, respectively.

For every angular separation of the two speakers, we willconsider the average of ∆r

HLover all speech fragments

of all source combinations, and over all tested speakersetups that correspond to the same separation (see sub-section IV-A). An increase of this parameter indicates aproper behavior of the M-NICA algorithm, i.e., it measuresthe degree to which the microphone envelopes (‘a priori’∆r

HL) or demixed envelopes (‘a posteriori’ ∆r

HL) are

separated into the original speech envelopes. Note thatfor the ‘a priori’ value, we select the microphones whichalready have the highest ∆r

HLin order to provide a fair

comparison. In practice, it is not known which microphoneyields the highest ∆r

HL’s, which is another advantage

of M-NICA: it provides only two signals in which thismeasure already maximized.

The decoding accuracy of the AAD algorithm is thepercentage of trials that are correctly decoded. Analogousto the criterion in subsection II-B, if the reconstructedenvelope s

Aat the output of the EEG decoder is more cor-

related with the (demixed or microphone) envelope thatis associated with the attended speech envelope than withthe other envelope, we consider the decoding successful.Here, we consider a (demixed or microphone) envelope tobe associated to the attended speech envelope s

Aif it has a

higher correlation with the attended speech envelope thanwith the unattended speech envelope.

We evaluate the performance of the MWF by means ofthe improvement in the signal-to-noise ratio (SNR). Forthe different setups of speech sources, we compare the SNRin the microphone with the highest input SNR to the SNRof the output signal of the MWF, i.e.

SNRin = maxi

{‖xi1‖2

2

‖xi2 + vi‖22

}(15)

SNRout =

∥∥∥∑Mi=1 wi ∗ xi1

∥∥∥2

2∥∥∥∑Mi=1 wi ∗ (xi2 + vi)

∥∥∥2

2

(16)

where the samples of the signal and noise contributionsxi1[n], xi2[n], and vi[n] from (1) are stacked in vectorsxi1, xi2, and vi, respectively, covering the full record-ing length, and wi is the time-domain representation ofthe MWF weights for microphone i (where the WOLA

8

0 2 4 6 8 10 12 14 16

0

0.5

1

clean envelope microphone envelope demixed envelope

0 2 4 6 8 10 12 14 16

0

0.5

1

time (s)

norm

aliz

ed e

nerg

y

Fig. 2. Effect of M-NICA, shown for a certain time window. Topfigure: original speech envelope (black) and microphone envelope(green). Bottom figure: original speech envelope (black) and demixedenvelope (red).

procedure implicitly computes the convolution in (16) inthe frequency domain). Note that we again assume thats1 represents the attended speech source and s2 is theinterfering speech source, which is why xi2 is includedin the denominator of (15) and (16) as it contributes tothe (undesired) noise power. Since an unequal number ofspeaker setups were analyzed at every angular separation,we will mostly consider median SNR values.

B. Speech envelope demixing

To illustrate the merit of M-NICA as a source separationtechnique, we plot the different kinds of envelopes in Fig. 2.In the top figure, the green curve represents an envelopeof the speech mixture as observed by a microphone, whilethe black curve is the envelope of one of the underlyingspeech sources. The latter is also shown in the bottomfigure, together with the corresponding demixed envelope(red curve). All envelopes were rescaled post hoc, becauseof the ambiguity explained in subsection III-B. The mi-crophone envelope has spurious bumps, which originatefrom the energy in the other speech source. The demixedenvelope, on the other hand, is a good approximation ofthe envelope of a single speech source. The improvementof ∆r

HLis shown in Fig. 3, for the noise-free and the

noisy case. For all relative positions of the speech sources,applying M-NICA to the microphone envelopes gives asubstantial improvement in ∆r

HL, which indicates that

the algorithm achieves reasonably good separation of thespeech envelopes and hence reduces the crosstalk betweenthem. There is a trend of increasing ∆r

HLfor speech

sources that are wider apart. Indeed, for larger angularseparation between the sources, the HRIRs are sufficientlydifferent due to the angle-dependent filtering effects of thehead, ensuring energy diversity. The mixing matrix A willthen have weights that make the blind source separationproblem defined by (11) better conditioned. When multi-talker background noise is included in the acoustic scene,∆r

HLis seen to be slightly lower, especially for speech

sources close together, when the subtle differences inspeech attenuation between the microphones are easilymasked by noise.

30° 60° 90° 120° 150° 180°0

0.2

0.4

0.6

0.8

1

angular separation

∆ r H

L

Influence of M−NICA, in noise−free and noisy case, on ∆ rHL

microphone envelopes, noise−free case

demixed envelopes, noise−free case

microphone envelopes, noisy case

demixed envelopes, noisy case

Fig. 3. Effect of M-NICA: ∆rHL for different separation between thespeech sources, for microphone and demixed envelopes in the noise-free case (dark and light blue, respectively) and microphone anddemixed envelopes in the noisy case (yellow and red, respectively).

C. AAD performanceFig. 4 shows the average EEG-based AAD accuracy

over all subjects versus ∆rHL

for different speaker separa-tion angles, when the microphone envelopes or demixedenvelopes from the noise-free case are used for AAD.The cluster of points belonging to the demixed envelopeshas moved to the right compared to the cluster of themicrophone envelopes, conform to what was shown inFig. 3. Three setups can be distinguished that have a sub-stantially lower AAD accuracy and ∆r

HLthan the others.

Two of them are setups with a separation of 30 ◦, while thethird one corresponds to a separation of 60 ◦. These resultsare intuitive, as the degree of cross-talk is higher whenthe speakers are located close to each other. The speakersthen have a similar energy contribution to all microphones,which results in lower quality microphone envelopes forAAD and also aggravates the envelope demixing problem,as demonstrated in Fig. 3.

Remarkably, despite the substantial decrease in cross-talk due to the envelope demixing, the average decodingaccuracy does not increase when applying the demixingalgorithm, i.e., both microphone envelopes and demixedenvelopes seem to result in comparable AAD performance.However, it is important to put this in perspective, asthe accuracy measure for AAD in itself is not perfect(and possibly not entirely representative) when the cleanspeech signals are not known. Indeed, a ‘correct’ AADdecision here only means that the algorithm selects thecandidate envelope that is most correlated to the attendedspeaker, even if this candidate envelope still contains alot of crosstalk from the unattended speaker. Therefore,the validity of this measure depends on the quality of thecandidate envelopes, i.e., a correct AAD decision accordingto this principle may have little or no practical relevanceif the selected candidate envelope does not contain a high-quality ‘signature’ of the attended speech that can even-tually be exploited in the post-processing stage (VAD andMWF) to truly identify or extract the attended speaker.Moreover, M-NICA automatically produces as many can-didate envelopes as there are speakers, circumventing theselection of the optimal microphones that would otherwisebe necessary, as explained in section IV.

9

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 175

80

85

90

∆ rHL

deco

ding

acc

urac

y (%

)Decoding accuracy vs ∆ r

HL

demixed envelopesmicrophone envelopes

−90° and −30°

−60° and −30°

−90° and −60°

Fig. 4. Average decoding accuracy over subjects versus ∆rHL

for the twelve tested speaker setups, using microphone envelopes(green) or demixed envelopes (red) from the noise-free case. Thecombinations of speaker positions that lead to the lowest performanceare indicated.

To further illustrate how envelope demixing influencesthe AAD algorithm, we show in Fig. 5 the correlation ofthe EEG decoder’s output s

Awith the true envelopes (in

Fig. 5a), and with the two candidate demixed envelopes(in Fig. 5b) as well as with the two candidate microphoneenvelopes (in Fig. 5c). The point cloud when using thedemixed envelopes (Fig. 5b) better resembles the pointcloud based on the clean speech envelopes, showing theinfluence of the demixing process. However, it seems thatthe variance is higher, as the demixing is not perfect.We observe that the point cloud corresponding to themicrophone envelopes (Fig. 5c) is clustered around themain diagonal. Intuitively, this is explained by the factthat the microphone envelopes are not yet separated intoseparate speech envelopes, and hence they have a consid-erable mutual resemblance.

Finally, we note that a large variability exists in thedecoding accuracy over all subjects, which is illustratedin Fig. 6. It spans a range between 52% and 98%, andprovides the only subject-specific effect on the overall per-formance of our processing scheme. The decoding accuracyusing either microphone envelopes or demixed envelopes isin general lower than the performance which is obtainedusing the clean speech envelopes, in an idealized scenario,as expected. Again, we observe that envelope demixing ingeneral does not improve nor lower the AAD accuracy,even if it raises the ∆r

HL. However, we restate that the

AAD accuracy measure employed here is in itself onlypartially informative. Indeed, this accuracy measure onlyquantifies how well the AAD algorithm is able to selectthe envelope with highest correlation with the attendedspeaker, but not how well this envelope actually representsthe attended speaker. The latter is important to also gen-erate an accurate VAD track that only triggers when theattended speaker is active. For this reason, it is relevantto include the demixing step in the analysis, as we showin the next subsection.

D. Denoising and speech extraction performanceThe median input SNR is shown in Fig. 7, for the

different angular separations between the speakers, and

−0.15 −0.1 −0.05 0 0.05 0.1 0.15 0.2 0.25 0.3

0.3

0.2

0.1

0

−0.1

rA

r U

Correlation with clean envelopes

(a)

−0.15 −0.1 −0.05 0 0.05 0.1 0.15 0.2 0.25 0.3

−0.1

0

0.1

0.2

0.3

rA

r U

Correlation with demixed envelopes

(b)

−0.15 −0.1 −0.05 0 0.05 0.1 0.15 0.2 0.25 0.3

−0.1

0

0.1

0.2

0.3

rA

r U

Correlation with microphone envelopes

(c)

Fig. 5. Scatter plot of the correlation coefficients rA and rU ofthe reconstructed envelope with the envelopes of the attended andunattended speech, respectively, for all trials from the noise-free case.Every trial corresponds to one point and is correctly decoded if thispoint falls below the black decision line rA = rU . The envelopes ofthe attended and unattended speech are either the clean envelopes(a), demixed envelopes (b), or microphone envelopes (c). Note thatthe latter two figures consist of more points than the first one, sinceAAD was performed for 12 different speaker setups.

subject number1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

deco

ding

acc

urac

y (%

)

50

60

70

80

90

100Subject-specific decoding accuracy

clean envelopesmicrophone envelopesdemixed envelopes

Fig. 6. Subject-specific decoding accuracy using the accuracywith clean envelopes (black line) as a reference. Accuracies obtainedby using microphone (green boxplots) or demixed (red boxplots)envelopes from the noise-free case are shown, over all 12 speakersetups.

10

angular separation30° 60° 90° 120 150° 180

inpu

t SN

R (

dB)

-5

0

5SNR at MWF input

noise-free casenoisy case

Fig. 7. Input SNR taken from the microphone with highest SNR,in the noise-free case (blue) and the noisy case (red), for all angularseparations between the speakers.

for both the noise-free and the noisy case. It is noted thatin the noisy scenarios, the inclusion of five uncorrelatednoise sources with an energy that is 10% of that of thespeech sources, lowers the input SNR with approximately10 log10(5 · 0.1) = 3 dB. For equal-energy speech sourcesthat are sufficiently far apart and/or for low noise levels,the input SNR is higher than zero, because in mostmicrophones, one speech source is prevalent over the otherdue to head shadow and thus for every speech source wecan find a microphone signal that gets most of its energyfrom that particular speech source (recall that the inputSNR is defined based on the ‘best’ microphone).

Fig. 8 shows the output SNR for the varying angularseparations between the speech sources, ranging from 30 ◦

to 180 ◦. Boxplots show the variation in MWF performancewhen using the AAD results of each of the 16 subjects(median subject-specific SNR value per angular separa-tion, i.e., 16 values per boxplot). First, we investigate theperformance for acoustic setups without additional noise.The output SNR is much higher when computing theAAD/VAD combination based on the demixed envelopes(see Fig. 8a), compared to the SNR when computing theAAD/VAD based on the original microphone envelopes(see Fig. 8b). In the latter case, the performance of theMWF drops as the speech sources are closer together(smaller angular separation). A similar, but smaller effectis observed for the AAD/VAD based on the demixedenvelopes. Fig. 8c and Fig. 8d show the output SNR inthe presence of multi-talker background noise when usingdemixed and microphone envelopes, respectively. In thiscase, the SNRs are lower - yet still satisfactory, given thesub-zero input SNR - and again the demixed envelopes areseen to be the preferred choice for use in the VAD. Theimprovement in SNR when choosing demixed envelopesfor the AAD/VAD over the microphone envelopes is sig-nificant, both in the noiseless and in the noisy case (p< 10−8, 2-way repeated measures ANOVA). Note thatall variability in the SNR over subjects is purely due tothe difference in the decoding accuracy, as explained inthe previous subsection. The black square markers in thefigures show the output SNR for a virtual subject with adecoding accuracy of 100%. It is seen that the SNR for

subjects with a high decoding accuracy closely approxi-mates this ideal performance, and sometimes even surpassit (as the envelopes used for VAD are still imperfect, thisis a stochastic effect). As a measure of robustness, weanalyzed over which range of VAD thresholds the resultswe found are valid. From Fig. 9, we see that the VADbased on demixed envelopes gives rise to a high outputSNR over a wide range of thresholds. By contrast, whenusing the microphone envelopes, a low SNR is observedfor all thresholds. The VAD thresholds to generate theresults of Fig. 8 were chosen as the optimal values foundwith these curves, and were reported in subsection IV-C.

VI. DiscussionThe difference between the SNR at the input and

output of the MWF is substantial, demonstrating thatMWF denoising can rely on EEG-based auditory attentiondetection to extract the attended speaker from a set ofmicrophone signals. Furthermore, for the first time, theAAD problem is tackled without use of the clean speechenvelopes, i.e., we only use speech mixtures as collectedby the microphones of a binaural hearing prosthesis. Thisserves as a first proof of concept for EEG-informed noisereduction in neuro-steered hearing prostheses.

Even in severe, noisy environments, subzero input SNRsare boosted to acceptable levels. This positive effect issignificantly lower when leaving out the envelope demixingstep, showing the necessity of source separation tech-niques. Rather than applying expensive convolutive ICAmethods on the high-rate microphone signals based onhigher-order statistics the M-NICA algorithm operates inthe low-rate energy domain and only exploits second-orderstatistics, which makes it computationally attractive. Infact, we circumvent an expensive BSS step on the rawmicrophone signals by using the fast envelope processingsteps and that way postpone the spatiotemporal filteringof the set of microphone signals until the multi-channelWiener filter. As opposed to convolutive ICA methods,the MWF only extracts a single speaker from a noisebackground with much lower computational complexityand a higher robustness to noise. From the results in Fig. 8,we see that the demixing using M-NICA has a strongpositive effect on the denoising performance. AlthoughM-NICA indeed slightly improves the AAD accuracy,the use of microphone envelopes without demixing stillyields a comparable performance, which is remarkable.The main reason for this is that we always compare withmicrophones which already have a high ∆r

HL, i.e., micro-

phones in which one of the two speech sources is alreadydominant. Such microphone envelopes with sufficiently lowcrosstalk - resulting in an acceptable AAD accuracy - arepresent due to the angle-dependent attenuation throughthe head. In practice however, we do not know which of themicrophones provide these good envelopes, which meansthat the use of M-NICA is still important to obtain a goodAAD performance, as it requires no microphone selection.Furthermore, based on Fig. 9, M-NICA seems to lead tomore robust VAD results by providing better estimates for

11

0

5

10

15

20

25

30

35

40

30° 60° 90° 120° 150° 180°angular separation

outp

ut S

NR

(dB

)

MWF using demixed envelopes, in noise−free case

(a)

0

5

10

15

20

25

30

35

40

30° 60° 90° 120° 150° 180°angular separation

outp

ut S

NR

(dB

)

MWF using microphone envelopes, in noise−free case

(b)

−5

0

5

10

15

30° 60° 90° 120° 150° 180°angular separation

outp

ut S

NR

(dB

)

MWF using demixed envelopes, in noisy case

(c)

−5

0

5

10

15

30° 60° 90° 120° 150° 180°angular separation

outp

ut S

NR

(dB

)

MWF using microphone envelopes, in noisy case

(d)

Fig. 8. Boxplots of the output SNR over all subjects and for different angles of speaker separation, using (a) demixed envelopes in the noise-free scenario, (b) microphone envelopes in the noise-free scenario, (c) demixed envelopes in the noisy scenario, (d) microphone envelopes inthe noisy scenario. All SNR values represent the median SNR over all pairs of stimuli and possibly multiple speaker setups, per combinationof subject and angular separation. The black squares indicate the output SNR for the ideal case of a subject with perfect AAD, i.e. anaccuracy of 100%.

the speakers’ envelopes, which seems to be the main reasonfor the improved output SNR when using the MWF.

The performance of our algorithm pipeline is seen tobe robust to the relative speaker position, i.e., even forspeakers that are close together, the combination of enve-lope demixing and multi-channel Wiener filtering results insatisfactory speaker extraction and denoising. The simpleVAD scheme proved to be effective, and is insensitiveto its threshold setting over a wide range. Note that astraightforward envelope calculation was used for AAD,and that more advanced methods for envelope calculation[9] or for increased robustness in attention detection [27]may further increase the accuracy. Also increasing thewindow length (larger than 30s) improves AAD accuracy,at the cost of a poorer time resolution (the latter is alsoimproved upon in [27]). The MWF performance in thecase of a perfectly working AAD (shown in Fig. 8) leadsus to believe in the capabilities of the proposed processingflow, especially after incorporation of expected advancesin AAD methods.

Future research should aim at collecting EEG measure-ments from noisy, multi-speaker scenarios over differentangles to validate the proposed processing for both theAAD and the speech enhancement on a unified dataset.It should be investigated whether representative EEG canbe collected in real life using miniature and semi-invisibleEEG devices, e.g., based on in-the-ear [13] or around-

0 0.05 0.1 0.15 0.2 0.250

5

10

15

20

25

30

35

threshold

outp

ut S

NR

(dB

)

Median SNR at MWF output vs threshold

demixed envelopesmicrophone envelopes

Fig. 9. SNR at output of the MWF for thresholds going from 1%to 25% of the maximum short-term energy, using demixed (red) ormicrophone (green) envelopes. No multi-talker noise was added, andan idealized AAD track with accuracy of 100% was used. SNRs aregiven as the median value over all subjects and all angular separationsbetween the speakers.

the-ear EEG [14], and possibly combining multiple suchdevices [16]. A study in [10] has demonstrated that a highAAD accuracy can still be obtained with only 15 EEGchannels, although this study assumed availability of theclean speech signals. It has to be investigated whetherthese results still hold in the case where only the speechmixtures are available, as in this paper.

As a next step, we aim to adjust the proposed processingscheme to an adaptive implementation, which would besuitable for online, real-time applications.

12

VII. ConclusionWe have shown that our proposed algorithm pipeline

for EEG-informed speech enhancement or denoising yieldspromising results in a two-speaker environment, even inconditions with substantial levels of noise. Our techniqueis extensible to multi-speaker scenarios, and except foran initial training phase, the algorithm operates solelyon the microphone recordings of a hearing prosthesis,i.e., without knowledge of the clean speech sources. Wehave demonstrated that, although the AAD performancedecreases, the AAD-informed MWF is still able to extractand denoise the attended speaker with a satisfactoryoutput SNR. All of the elementary building blocks, per-forming speech envelope demixing, voice activity detec-tion, speech filtering, and auditory attention detection,are computationally inexpensive and are implementablein real-time. This renders them very attractive for usein battery-powered hearing prostheses which have severeconstraints on energy usage. With this study, we madethe first attempt to bridge the gap between auditoryattention detection in ideal scenarios with access to cleanspeech envelopes, and neuro-steered attended speech en-hancement in situations that are more representative forreal life environments (without access to the clean speechenvelopes).

AcknowledgementsThe authors would like to thank Neetha Das and Wouter

Biesmans for providing the experimental EEG data andtheir help with the implementation of the AAD algorithm,and Joseph Szurley for the help with the implementationof the MWF.

References[1] H. Dillon, Hearing aids. Thieme, 2001.[2] S. Doclo and M. Moonen, “GSVD-based optimal filtering for

single and multimicrophone speech enhancement,” Signal Pro-cessing, IEEE Transactions on, vol. 50, no. 9, pp. 2230–2244,2002.

[3] R. Serizel et al., “Low-rank approximation based multichannelWiener filter algorithms for noise reduction with application incochlear implants,” Audio, Speech, and Language Processing,IEEE/ACM Transactions on, vol. 22, no. 4, pp. 785–799, 2014.

[4] S. Doclo et al., “Reduced-bandwidth and distributed MWF-based noise reduction algorithms for binaural hearing aids,”Audio, Speech, and Language Processing, IEEE Transactionson, vol. 17, no. 1, pp. 38–51, 2009.

[5] J. A. O’Sullivan et al., “Attentional selection in a cocktail partyenvironment can be decoded from single-trial EEG,” CerebralCortex, p. bht355, 2014.

[6] N. Ding and J. Z. Simon, “Emergence of neural encoding ofauditory objects while listening to competing speakers,” Pro-ceedings of the National Academy of Sciences, vol. 109, no. 29,pp. 11 854–11 859, 2012.

[7] E. M. Z. Golumbic et al., “Mechanisms underlying selectiveneuronal tracking of attended speech at a “cocktail party”,”Neuron, vol. 77, no. 5, pp. 980–991, 2013.

[8] N. Mesgarani and E. F. Chang, “Selective cortical represen-tation of attended speaker in multi-talker speech perception,”Nature, vol. 485, no. 7397, pp. 233–236, 2012.

[9] W. Biesmans et al., “Auditory-inspired speech envelope ex-traction methods for improved EEG-based auditory attentiondetection in a cocktail party scenario,” IEEE Transactions onNeural Systems and Rehabilitation Engineering, vol. PP, no. 99,pp. 1–1, 2016.

[10] B. Mirkovic et al., “Decoding the attended speech stream withmulti-channel EEG: implications for online, daily-life applica-tions,” Journal of neural engineering, vol. 12, no. 4, p. 046007,2015.

[11] A. Aroudi et al., “Auditory attention decoding with EEGrecordings using noisy acoustic reference signals,” in Acoustics,Speech and Signal Processing (ICASSP), 2016 IEEE Interna-tional Conference on. IEEE, 2016.

[12] V. Mihajlovic et al., “Wearable, wireless EEG solutions in dailylife applications: what are we missing?” Biomedical and HealthInformatics, IEEE Journal of, vol. 19, no. 1, pp. 6–21, 2015.

[13] D. Looney et al., “The in-the-ear recording concept: User-centered and wearable brain monitoring,” Pulse, IEEE, vol. 3,no. 6, pp. 32–42, 2012.

[14] M. G. Bleichner et al., “Exploring miniaturized EEG electrodesfor brain-computer interfaces. an EEG you do not see?” Physi-ological reports, vol. 3, no. 4, p. e12362, 2015.

[15] J. J. Norton et al., “Soft, curved electrode systems capableof integration on the auricle as a persistent brain–computerinterface,” Proceedings of the National Academy of Sciences, vol.112, no. 13, pp. 3920–3925, 2015.

[16] A. Bertrand, “Distributed signal processing for wireless EEGsensor networks,” Neural Systems and Rehabilitation Engineer-ing, IEEE Transactions on, vol. 23, no. 6, pp. 923–935, Nov2015.

[17] A. J. Casson et al., “Wearable electroencephalography,” Engi-neering in Medicine and Biology Magazine, IEEE, vol. 29, no. 3,pp. 44–56, 2010.

[18] H. Kayser et al., “Database of multichannel in-ear and behind-the-ear head-related and binaural room impulse responses,”EURASIP Journal on Advances in Signal Processing, vol. 2009,p. 6, 2009.

[19] A. Bertrand and M. Moonen, “Energy-based multi-speakervoice activity detection with an ad hoc microphone array,” inProc. IEEE Int. Conf. Acoustics, Speech, and Signal Processing(ICASSP), Dallas, Texas USA, March 2010, pp. 85–88.

[20] S. J. Aiken and T. W. Picton, “Human cortical responses to thespeech envelope,” Ear and hearing, vol. 29, no. 2, pp. 139–157,2008.

[21] B. N. Pasley et al., “Reconstructing speech from human audi-tory cortex,” PLoS-Biology, vol. 10, no. 1, p. 175, 2012.

[22] N. Ding and J. Z. Simon, “Neural coding of continuous speechin auditory cortex during monaural and dichotic listening,”Journal of neurophysiology, vol. 107, no. 1, pp. 78–89, 2012.

[23] J. R. Kerlin et al., “Attentional gain control of ongoing corticalspeech representations in a “cocktail party”,” The Journal ofNeuroscience, vol. 30, no. 2, pp. 620–628, 2010.

[24] A. Bertrand and M. Moonen, “Blind separation of non-negativesource signals using multiplicative updates and subspace projec-tion,” Signal Processing, vol. 90, no. 10, pp. 2877–2890, 2010.

[25] S. Chouvardas et al., “Distributed robust labeling of audiosources in heterogeneous wireless sensor networks,” in Acoustics,Speech and Signal Processing (ICASSP), 2015 IEEE Interna-tional Conference on. IEEE, 2015, pp. 5783–5787.

[26] A. Bertrand et al., “Adaptive distributed noise reduction forspeech enhancement in wireless acoustic sensor networks,” inProc. of the International Workshop on Acoustic Echo and NoiseControl (IWAENC), Tel Aviv, Israel, August 2010.

[27] S. Akram et al., “Robust decoding of selective auditory attentionfrom meg in a competing-speaker environment via state-spacemodeling,” NeuroImage, vol. 124, pp. 906–917, 2016.