an exploration on the potential of an electroencephalographic headset for human computer interaction

MASTER’S PROJECT, THE COLLEGE OF WILLIAM AND MARY, SPRING 2015 1

An Exploration on the Potential of anElectroencephalographic Headset for Human

Computer InteractionJohnathan Savino, Peter KemperDepartment of Computer ScienceThe College of William and MaryWilliamsburg, VA, 23185, USA{jesavino, kemper}@cs.wm.edu

F

Abstract—As computers become more and more integratedinto our daily life, so does the need for improved ways ofinteracting with them. We look into the possibility of using anEEG Brain Sensing Headset as means of better interfacing witha computer. Through a user study we analyze human brainwave patterns when responding to simple yes - no questions, inaddition to looking at the accuracy of an Emotiv EEG Headsetin recognizing different facial movement patterns. We implementour findings into a brain controlled music player, capable ofrecognizing head movements and certain facial movements toaid turning the player on and off plus allowing rating of playedsongs to be done hands free. We provide data to concludethat both brow motion and head motion provide accurate andreliably recognizable data for deployment into a variety of braincomputer applications.

Index Terms—EEG, Brain Computer Interface, Human Com-puter Interaction, User Study

1 INTRODUCTION

INTERACTING with computers has become socommon in our daily lives that many people

will not go more than one day without spendingsome time in front of one their devices. This inter-action has been developed carefully over the lastfifty years, moving from keyboard only systems,towards more advanced graphical user interfaces.It only seems natural that if we can improve theway that humans interact with their computers, wecan drastically improve one of the more frequent

This project was approved by the College of William and MaryProtection of Human Subjects Committee (Phone 757-221-3966) on2015-03-03 and expires on 2016-03-03.

interactions we make throughout the day. In doingso, we not only reduce the amount of time requiredto complete mundane tasks, but also allow moretime to spent on the problem at hand instead ofinterfacing with the computer. We are also able tobroaden the group of users able to use computersby decreasing the learning curve required to use arange of computing devices.

We would like to utilize Electroencephalo-graphic (EEG) sensing equipment in order to ex-ploit common brain signal patterns which occur intandem with our daily interactions. By harnessingthese patterns in the different regions of the brain, itis possible to track different emotions and recognizeevoked signals based on physical or visual stimuli.EEG sensors work by measuring the electrical sig-nals in different regions of the brain, which, whencombined, allow features such as mood to be ex-tracted in real time. The benefit to EEG based brainmonitoring systems is that EEG is non-invasive; allsignal collection is done with sensors places on topof the head. This allows for easy integration intodaily life.

In order to provide a fully immersive experiencefor your everyday user, there must be a way toaccurately disseminate the signal we get from theEEG sensors from the noise of background brain ac-tivity. While a significant amount of work has beendone mapping different cortical regions of the brainto specific emotions, less has been done looking atthe signals the brain produces throughout everydayinteraction with a computer. More of this will bediscussed in section 2 with other related works.


This study is an attempt to see how the brainresponds throughout interaction with computers.More specifically, we will look at how users af-fective and effective responses change as they arepresented a series of yes and no questions. Thesebinary questions can be assumed similar to confir-mation dialogs often encountered with a computer.In addition, we will look to determine which fa-cial expressions are best utilized as a state controltrigger. The main contributions of this paper can besummarized as follows:

• We present a user study which looks toanalyze how users brains respond through-out interaction with a computer. This infor-mation becomes incredibly powerful whenattempting to interconnect brain and com-puter. From this study we learn typical brainpatterns experience by average users whenthey interact with a computer on an every-day basis.

• We implement a simple binary controllerinto an EEG based brain music player. Thissystem will allow users to make choices inprogram such as accepting a dialog popupusing a gyroscope on their head.

• We implement an on - off controller to pauseand unpause the music player. Users canraise their brow twice to achieve this, whichadds the ability to operate the player handsfree.

The rest of this paper is structured as follows. Wepresent related work and our motivation in section(2), information on the user study in section (3),an analysis of the data in section (4), and finallythe background of our implementation in the musicplayer in section (5). We then conclude and presentour future work.

2 MOTIVATION

While electroencephalography has been around fora number of years, the work done in regards tohuman computer interaction has not been exploredto its full potential.

2.1 EEG Background

While many people have heard of electroen-cephalography, it certainly is not an everyday term.In order to grasp the limitations of EEG as a brainsensing solution, we will first discuss two differentapproaches of sensing.

Before we do this, we will define what an EventRelated Potential (ERP) is. In short, an ERP mea-sures the brains specific responses to some sort ofspecific cognitive, sensory or motor event. We gofurther into how ERPs are used in BCI.

2.1.1 SSVEPThe first approach is the analysis of Steady-StateVisual Evoked Potentials (SSVEP). SSVEP are nat-ural responses to stimulus of specific frequencies[1]. These visually evoked potentials are elicitedby sudden visual stimuli and the repetitive stimulilead to stable oscillations in EEG. These voltageoscillation patterns are called SSVEP [2].

SSVEP is evoked at the frequency of the stim-ulus. When the retina is excited by a visual cue inrange of 3.5 Hz to 75 Hz, the brain generates electri-cal activity mimicking this frequency [2]. This activ-ity can be further broken down into low, medium,and high frequency bands. Because the stimulusis directly related to frequency, SSVEP is a goodindicator of visual disease in a variety of patients.

In relation to BCI, SSVEP functions well in ap-plications that send a large number of commandswhich require a high reliability. A typical setup foran SSVEP-based system involves using one or mul-tiple LED lights to flicker at varying frequencies.SSVEP is ideal for users where small eye movementis allowed, users that are capable of sustained atten-tion effort, and applications where small commanddelays are allowed.

SSVEP could be applied to our application, butwe chose to perform our study with the Emotivheadset because it is a commercial headset. In addi-tion, because we are interested in a users emotionaland physical responses during ordinary computerinteraction, we chose the Emotiv headset over oneusing SSVEP.

2.1.2 P300The second approach is called P300 Evoked Po-tential. This wave is a component of an EventRelated Potential, not limited to auditory, visual orsomatosensory stimuli [1]. P300 is one of the majorpeaks in the ERP wave response. The presentationof stimulus in an oddball paradigm can produce apositive peak in the EEG, approximately 300ms af-ter onset of the stimulus [2]. The triggered responseis called the P300 component of ERP.

P300 sensing nodes are placed along the centerof the skull and the back of the head. The wavecaptured by the P300 component ranges in the 2 to5 µHz, and only lasts 150 to 200ms [2]. Due to the


Fig. 1. An example of a P300 system

small nature of these measurements, one can imag-ine that a significant amount of signal processingmust be done in order to get access to any sort ofmeaningful data.

We show in Figure 1 a simple setup for clas-sifying P300 data to implement a spelling system.EEG Data is first acquired, and then sent for Pre-Processing. In this step, noise is removed from thegathered signal. After that, a Principle ComponentAnalysis is run in order to highlight the signalsthat contribute the most, which are then fed intoa classifier.

In order to understand the basis for P300-basedBCI, we will look at the speller in the system shownin Figure 1. The user is presented with a six bysix grid, and instructed to focus on the letter thatthey would like to choose. The rows of the tableare then randomly flashed, which evokes a P300-response when the row the user is focusing on lightsup. This process is then repeated for the columns,which allows the system to narrow down the letterthe user is interested in.

From this simple example, we can see that P300is a very strong system for BCI, but unfortunately isquite slow. As a result, more recent spelling systemsstill utilize some variation on this simple spellingparadigm.

For our purposes, P300 based sensing could beapplied into any of the decision based sensing, butit does not give us the emotional responses we arelooking for to determine how a user is feeling ata given time. Similarly to the reason we chose notto use SSVEP, the Emotiv headset is commerciallyavailable, which further plays into our decision touse it.

2.2 Related WorkMany applications have been developed for usewith EEG headsets. These applications extend intothe realm of web browsers, gaming systems, andeven mobility control systems [3]. In fact, there has

been some work done to connect these brain sensingmethods to mobile phones, in order to interactwith the smaller devices [4]. Such a wide arrayof applications highlights the desire for a deeperunderstanding of the way our brains interact withcomputers.

Using the Emotiv EPOC headset specifically, theauthors in [5] utilized the gyroscope in the headsetin order to control the movement of a wheelchair.The system was developed to move the chair usingeither one head motion or four head motions. Thissystem shows that the headset is powerful enoughto control larger scale applications, and can be effec-tive enough for day to day use.

In [6], the authors develop a pointing devicesimilar in functionality to a mouse for use byquadriplegic users. They were able to emulate stan-dard mouse movement, and easily able to teachthis new system to quadriplegic individuals. Thisshows us that a gyro-scoped based system can havepractical application both for day-to-day or averageusers, and can possibly help aid disabled users.

Another area of interest in BCI research is theneed to know when a user is attempting to accessthe system. Because of the inherent always-activenature of the human brain, there needs to be a wayto turn the system on and off. Researchers have at-tempted to use complicated Gaussian Probabilities[7] to solve this problem, but the math required isvery advanced. Instead, we attempt to show thatthere may be better ways of doing this, based onthe natural responses of the human brain duringnormal interaction with a computer. Specifically, weattempt to find a facial movement to accurately andreliably turn the EEG listening system on and off.

2.3 Equipment

We use the Emotiv EPOC (seen in Figure 2) Headsetfor our testing. The EPOC comes with fourteen sen-sors and two reference points, and transmits datawirelessly. The headset uses the sequential samplingmethod, which for our purposes entails using thedata available at every time-step to extract featuresfor that time-step. The EPOC headset connects viaWiFi in order to transmit the data back to an ag-gregator. There, signal processing is done to reducethe noise, a simple Principal Component Analysis isdone, and the classification of the features is passedto the user either via the Emotiv Control Panel, orthrough their open API.

The Emotiv headset is capable of measuring anumber of things. The built in two-axis gyro-scope


Fig. 2. The Emotiv Epoc Headset

provides an accurate sensing of any head movementfrom the user. The Emotiv API allows access toa number of other classifications in the emotionaland physical ranges, termed Affectiv and Expressivrespectively. The headset also measures the usersintent to perform specific trained tasks through theCognitiv Suite.

The Expressiv suite offers access to physicalevents. These events are limited to Blink, RightWink, Left Wink, Look Right, Look Left, LookUp, Look Down, Raise Brow, Furrow Brow, Smile,Clench, Right Smirk, Left Smirk, and Smile. For eachof these actions, when the headset classifies one asoccurring, a signal is sent to the application, whichwe then log. All events operate on a binary scale,either they happen or they do not, except for thebrow events, which give a measure of extent. Thesensitivity of the system to each of these events canbe changed, but for our study we use the defaultconfiguration in order to give the best representa-tion of a non-trained user.

The Affectiv suite reports real time changes inthe subjective emotions experienced by the user.This suite looks for universal brainwave charac-teristics, and stores this data so the results can berescaled over time. We select the new user optionfor every study participant so not to bias our resultswith Emotiv’s learning system.

The Affectiv suite offers analysis of five differ-ent emotions, Short-Term Excitement, Long-TermExcitement, Frustration, Engagement and Medita-tion. Short-Term Excitement is a measure of posi-tive physiological arousal. Related emotions to thisinclude titillation, nervousness and agitation. Long-Term Excitement is similar to short-term, but thedetection is fine tuned to changes in excitement overa longer period of time. The definition for the Frus-

-1500

-1000

-500

0

500

1000

1500

Gyr

osc

op

e R

ead

ing

Time (ms)

Gyroscope Data Logged for Head Motion

X Y

Fig. 3. Gyroscope data for Positive / Negative Head Motions

tration measurement parallels the emotion experi-enced in everyday life. Engagement is consideredto be experienced as alertness and the consciousdirection of attention towards task related stim-uli. Related emotions include alertness, vigilance,concentration, stimulation and interest. Meditationis defined as the measurement of how relaxed aperson is.

In the Cognitiv Suite, the headset allows fortraining of thirteen different actions, not limited topush, pull, rotate and disappear. This suite worksby training a classifier with how a user’s brainresponds when thinking about these specific ac-tions. Then, the classifier listens for these patternsto classify up to four of these actions at the sametime. We do not use the Cognitiv Suite in our study,but note that for actions like clicking a mouse orturning the volume down in the music player, thissuite would be very useful.

While the Emotiv headset is very good, it is ata consumer price range. We wanted to assure thatour results were simple enough to be achievablewithout requiring users to own elaborate, bulkyEEG systems. As a result, some of the recognitionpromised by the Emotiv headset is not up to thestandards we could have hoped for. Our user studylooks to determine which actions are best recog-nized and therefore are best suited for integrationinto a reliable application.

2.4 Key FeaturesIn our testing with the Emotiv headset, we cameacross a few key observations. The Emotiv headsetdoes a few things very well, and other things not


Action Percentage of SuccessesRaise Eyebrows 80.00%

Blink 20.00%

Left Wink 10.00%

Right Wink 10.00%

Look in Direction 6.00%

Fig. 4. True Positive Accuracy of Different Effectiv Motions usingthe Emotiv EPOC headset

Fig. 5. The main user interface for control over the EEG loggingapplication.

quite as well. We tested the Effectiv suite for oneperson while learning how to use the headset forour user study. As you can see in Figure 4, the truepositive rates for blinking and eye direction are wellbelow the accuracy one might require in a real-timeapplication. That being said, the eyebrow motiondetection is much stronger. We still record data forall these features with our users, as it is possible thatother users could have better results.

In addition to the fourteen sensors, the EmotivEPOC headset contains a two direction gyroscopebuilt into the headset. In researching the ways thathumans respond to questions in everyday conver-sation, we noticed that head motion played a largefactor in determining a users response from phys-ical cues alone. We tested the Emotiv gyroscopeand extracted data for someone nodding positively,negatively, and vigorously nodding positively andnegatively. As Figure 3 clearly shows, it is notchallenging to discern how a user is responding toa question based on the gyroscope data alone. Asa result, a significant portion of our study will bebased on utilizing the gyroscope to extract humanresponse information.

3 USER STUDY

3.1 Study BackgroundWe implement a testing application in java with twoparts. The first is a logging based system, whichextracts the classification of the EEG data and logsthis data. We log emotional data for Engagement,

Fig. 6. A sample question a user would see throughout the study.In total the user is asked 10 questions.

Frustration, Short Term Excitement, Long Term Ex-citement, and Meditation. We also log the deltaof the motion in the X and Y directions of thegyroscope. Finally, we log the Effective responsesof the participant, which include Eyebrow motion,blinking, winking with both eyes, and directionaleye motion. Of all the Effective responses, onlythe Eyebrow measurement is a measure of extent,ranging from 0 to 1, while the rest are binary, eitherthey happened or they did not at each time-frame.

The second part is a simple survey, run at thesame time as the logging application. Participantsare asked a series of yes - no questions (presentedin full in Appendix A), and they are asked to re-spond using head motions and audible cues. Uponanswering ten yes - no questions, they are asked tospecifically do each of the Effective physical actions.This is to both test the accuracy of the headset formultiple people and get a baseline for analyzingif any of these physical responses are visible insearching for positive or negative responses fromour participants.

Yes - no questions appear on screen for tenseconds, and participants have five seconds be-tween questions. Minimal instruction is given to theparticipants, and the questions are not seen untilthe session begins. The questions are specificallywritten so that there can be no gray area in regardsto their answer. After the session, the responses ofthe participant are recorded so we do have a groundtruth label for each question.

Nine users participated in our study. All par-ticipants were willing, and signed a consent formbefore participating. The pool was made up of fourmales and five females, ranging from undergradu-ates, graduates, and faculty of The College. Mostusers were very excited to have their brain signalslooked at, which we do note when looking at thecollected data.


4 RESULTS AND ANALYSIS

Of the nine participants, eight provided quality re-sults. Only one participant’s data had to be thrownout due to insufficient signal quality throughout theexperiment. The participant had long, thick hair,which may have been the reason for the poor signalquality. While the data was lost, we did learn thatthe Emotiv headset is not an end-all on the hard-ware level. Also, in each of our figures here, weshow the data from one participant instead of alleight. This is because the trends we point out areeasier to observe in one participant, and are seenacross all participants.

4.1 Collected Data

We plot the gyroscope data for the entire survey inFigure 7 and the Affectiv data in Figure 9.

4.1.1 Gyroscope Data

From the Gyroscope figure, we can see that it iseasy to discern what the user is answering. Thisis because nodding one’s head in affirmation con-trasted with the negative shake are actions thatoccur on separate axis. So from an affirmative /negative perspective, realizing a user’s answers is,as expected, quite easy. The interesting result comesfrom a closer analysis of the gyroscope data. Whena user is more emphatic about their response, theyshake their head more vigorously, which showsup in the gyroscope delta. We zoom in on twodifferent yes responses for one participant in Figure8. The two questions asked here were questionsthree and four, or ”Have you ever been to the stateof Virginia?” and ”Do you like chocolate?”. Clearlythe first response is less emphatic than the second,which aligns with how we expected participants toanswer. Because the study was entirely conductedin the state of Virginia, it makes sense that a partic-ipant (assuming they like chocolate) would moreemphatically affirm their taste for chocolate overtheir habitation of Virginia.

This relation also manifests itself when lookingat the average gyroscope delta for both responses.When looking at the two questions, the first hadan average Y magnitude of 32.74, while the secondhad an average Y magnitude of 56.54. Because wecan find such a difference, we can further divideresponses into strong no, no, yes, and strong yes.We will use this information further in the imple-mentation section.

-500

-400

-300

-200

-100

0

100

200

300

400

500

Gyro

scop

e De

lta (I

MU

)

Time (seconds)

Gyroscope Data Collected over Time

gyroX gyroY

Fig. 7. Graph of collected gyroscope data over time for one studyparticipant. The blue lines are the readings for the X axis, andthe orange for the Y axis. It is easy to tell that when the user isanswering yes to a question, the Y axis reports more data, andthat the X axis reports more motion when the user is answeringno.

-400

-300

-200

-100

0

100

200

300

400

Gyro

scop

e De

lta (I

MU

)

Time (seconds)

Y Axis Data for Questions 3 and 4

Fig. 8. Graph of one participant’s answers to two individualquestions. The first answer is to the question ”Have you everbeen to the state of Virginia?” and the second is to the question”Do you like chocolate?”.

4.1.2 Affectiv Data

Looking at the Affectiv Data for one participant,we can see the various emotions throughout theentire survey. As we would expect, the Long-TermExcitement and Short-Term Excitement scores bothdrop as the survey goes on. Initially, users are ex-cited to being using EEG sensing equipment, likelyfor the first time. But once the novelty wears off,the mundane task of answering simple questionsand moving features of their face around becomes


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Emot

iona

l Res

pons

e

Time (seconds)

Emotional Data over Time

Frustration Short Term Excitement Long Term Excitement

Fig. 9. Graph of the collected Affectiv Data over the course ofthe study. The emotional data is normalized to the [0,1] range,with 1 being a strong feeling and 0 being no feeling.

boring, and the excitement scores drop off.Frustration rises and falls throughout the survey,

but peaks the most near the end for all participants.Because the frustration score can be paralleled withboredom, once the user hits the end of the study,all of the novelty has worn off, and their brain isresetting to a more natural pattern. It is expectedthat answering ten simple questions and repeatingeight actions would wear on most people over time.

We did mention meditation and engagementwhen discussing the Emotiv Affectiv Suite. For allparticipants in the study, every participant recordedconstant data throughout the survey for these twoemotions. At this time we are not sure if this wasdue to a hardware malfunction with the specificheadset we tested on, a weak signal strength, orsome other unforeseen error.

4.1.3 Expressiv DataIn addition to logging the gyroscope data and emo-tional data, we looked to verify our hypothesisabout the facial motions recognized by the EmotivHeadset. We plot the recording of blinking and eye-brow events in Figure 11 and Figure 10 respectively.From these we look to see if either motion couldbe used to accurately and consistently turn a musicplayer on and off.

First looking at Figure 11, we can see that thereis no periodic pattern. We also note that there isno significant difference between the number ofblinks recorded at a given time compared to thenumber recorded in the ten second period where theparticipant was consciously, continuously, blinking.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Mag

nitu

de o

f Eye

brow

Rai

se

Time (seconds)

Recorded Magnitude of Brow Raising over Time

Fig. 10. Graph of recorded brow raise magnitudes throughoutthe study. This magnitude is on a zero to one scale, and is ameasure of extent. The red outline is where we asked the user torepetitively raise their brow. We can see that there is a significantincrease in brow motion in this time period.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Mag

nitu

de o

f the

Blin

k

Time (seconds)

Recorded Blinking Over Time

Fig. 11. Graph of recorded blink events over the course of thesurvey. The blink event is recorded on a binary scale; either ithappens or it does not. The red outline is where we asked theuser to repetitively blink their eyes. We can see that there areonly 5 or 6 blinks recorded in this time frame, far less than thenumber of times the user blinked.

This further supports our original hypothesis thatblinking would not be an accurate facial motion tobe used with any sort of consistency.

On the other hand, Figure 10 shows the extentto which the Emotiv headset recorded the browbeing raised. By looking at the graph we can seethe exact period when the participant was askedto repeatedly raise their eyebrows. Other than thisone period, eyebrow extent at any one period isrelatively controlled. From this we back up ourhypothesis that eyebrow motion could be used as


an accurate state transition trigger.We also looked at the recording of winking

with individual eyes in addition to the recordingof which way a person is looking. Neither actionreturned significant data so we do not include thefigures in this report. We determined that neithermotion would provide consistent or accurate resultsas a trigger for state changes.

4.2 Statistical Analysis

As part of our analysis on emotions, we looked atthe statistical correlation between positive answerand short-term excitement, in addition to the cor-relation between negative answers and frustration.We would expect both to have positive relation-ships.

We ran a Pearson’s Correlation test on Frustra-tion, Short-Term Excitement, yes answers and noanswers. We took the absolute value of the gyro-scope data to end up with a function which is largerwhen the participant was answering. We can seethe results of our analysis in Figure 12. When thePearson Correlation Coefficient is close to one itsignifies a positive, linearly correlated relationship,and close to negative one implies the opposite. Aswe can see, the only relationship which exists isthe fact that there is a moderately strong positiverelationship between short-term excitement and yesanswers. While the results are not strong enough tobe used in an implementation, we note that there isa strong relationship between short-term excitementand yes answers.

We also look at the average magnitude of themotion collected by the gyroscope. In Figure 13 weplot the average magnitudes of the motion we col-lect for all users. We only look at data greater than40 inertial units because that allows us to see onaverage how much a user moves their head whileanswering a question specifically. As we can see,yes answers, on average, require more motion thanno answers. This makes sense when we considerhow the human body has a larger range of motionlooking up and down compared to left and right.We use this information in our implementation ofthe results classification system to set thresholds foreach rating. We also see that in developing a system,it would make sense to train a classifier for eachindividual. Because people move in different ways,a classifier would allow the ratings to be tailored tothe individual. We leave this for future work.

Frustration Excitement No YesFrustration 1.00000Excitement -0.16022 1.00000

X = No 0.04362 0.07417 1.00000Y = Yes -0.05421 0.31528 0.10533 1.00000

Fig. 12. The Pearson Coefficient’s of Frustration and Short-TermExcitement compared with X and Y axis gyroscope data.

0

50

100

150

200

250

300

1 2 3 4 5 6 7 8Av

erag

e N

on-Z

ero

Mot

ion

Dete

cted

Participant Number

Average Magnitude of Yes and No Responses

xAxis yAxis

Fig. 13. The average magnitude of yes answers and no answersfor each user. We can see that yes answers require more motionin the head on average.

5 IMPLEMENTATION

We implement our findings in a Brain-Controlledmusic player.

5.1 Existing Application

As a result of the study, we implement our responseclassification system and state transition classifieron top of a music player. The existing architec-ture is shown in Figure 14. The base structure isbuilt on top of the .NET Windows Media Player,which is shown in Figure 15. The system connectsto the Emotiv EEG headset using their API. Oncewe have access to the headset, extracting both theRaw EEG data in addition to the classifications theEmotiv Signal Processing gives us, both of thesedata sets are stored in a database. This information,in addition to the rating system we discuss next, iscompiled to drive a song recommendation engine.When the media player loads the first and subse-quent songs to play, the recommendation enginedrives this decision process.

The user has the option to select between Work,Study, and Relax modes, and music is played which


Fig. 14. The existing architecture of the BCI controlled musicplayer. Input is done through the Emotiv Headset which is thestored in the database and used to recommend songs for theuser to listen to.

Fig. 15. The user interface of the Brain Controlled Music Player.

matches their mood for each mode. In addition, theplayer starts in a mode which matches the timeof day, so the initial music played is more likelyto match what the user would like to listen to.The information that drives the selection engine isextracted from a series of ratings the user gives eachsong once it is done playing. The rating is done ona 1 to 5 scale, which ranges from ”I hated this song”to ”I loved this song”. In addition to taking thepure rating, the users Arousal and Valence levelsare recorded using the Emotiv headset, and all ofthis is compiled to rate the song for the given mode.This information is in turn used to select songs forthe user as the player is used more.

5.2 On - Off Switch

As we discussed, one of the things we were lookingto evaluate in the user study was which facial eventwould provide reliable and accurate recognition for

use turning the Emotiv system on and off. We deter-mined from the study that the Emotiv system bestrecognizes the motion of the brow, and this is whatwe add to the existing music player application.While the main idea was to look for turning thesystem on and off, there is no reason at the momentto stop recording EEG data while the player isrunning. Instead, we allow the user to pause andunpause the music player using two raises of theireyebrows. We have found this works with reliableconsistency while using the application. In addition,we have shown that this can easily be extended toan application which relies more heavily on brainsignals as input, and could quite easily be added totell the EEG headset to start and stop listening.

5.3 Gyroscope Based Rating Dialog

As we learned in the user study, head motions suchas nodding and shaking the head are both quiteeasy to detect and easy to extract more informationthan yes and no out of. Using this information, weimplement a classification scheme for the user toanswer how much they enjoyed listening to a song.In the existing implementation this dialog appearswhen the user stops listening to a song, either bypressing the ”next” button or when the song ends.We extend this usage scenario by asking the usersimply if they liked the song that was playing. Thisquestion can be see in Figure 16.

Structurally, the main thread spawns off twochildren, one to show the prompt and one to listenfor the gyroscope. Access to the EEG headset ispassed through the BCIEngine which is responsiblefor recording the BCI data about the song playing.Once we have access to the headset, it is easy toextract the gyroscope delta and average the absolutevalue of this over time. Once enough data has beencollected (we select 300 as a constant number ofpoints) we classify the response into either a strongno, no, neutral, yes, or a strong yes. This is repre-sented to the user in the dialog box shown in Figure17. The user is able to simply accept the rating bywaiting five seconds or clicking the yes button, or,if the system did not correctly classify how they feltabout the song, the user can change the rating.

The overall rating structure works very well. Forusers with enough variance between a standard an-swer and a strong answer the system will accuratelyclassify their rating of the song within one pointevery time. However, we do notice from our datain the user study that setting static thresholds forthe different classifications is not optimal. We leave


Fig. 16. The dialog that appears when the user finishes playinga song. The system listens for a response from the user or waitsfive seconds before resolving to the default neutral value.

Fig. 17. The verification dialog that appears once a user hasclassified a song. The user is allowed to change the rating ifthey do not believe the system correctly rated the song.

training the system for individual use as futurework.

6 CONCLUSION

We have explored the use of the Emotiv EEG head-set as a means for interacting with your computer.While we were unable to find significant correla-tions between reported brain activity and the an-swers to simple yes and no questions, we were ableto determine that one’s head motions can provide areasonable scale of agreement and disagreement. In

addition, we learned that the Emotiv headset doesbest when listening for motion in the brow. Utilizingthese two facts we implement an on-off switch andrating classification system on top of an existing BCImusic player. Both of these contributions can easilybe extended to a myriad of different applications,and have been shown to work. We are hopeful thatour research will be utilized to improve future braincomputer interfaces as both hardware capabilitiesand consumer demands increase in the years tocome.

6.1 Future WorkThere are a few areas in which we could extend ourcurrent research. Because the user study was doneusing one Emotiv EEG headset, we would like totest these concepts using other commercial headsetsto see if we can find more significant correlationsbetween mood and how a user answers a question.As EEG research becomes a bigger field in com-puter science, more data will need to collected fora variety of different headsets to fully understandwhat parts of the brain fire for specific computerinteractions.

We also would like to use the emotional data andgyroscope data to improve our rating classificationsystem in the music player. By analyzing how ausers head is moving while they listen to a song, itmight be possible to adjust our rating system to aneven finer grained scale to truly understand whena user liked a certain song. Moreover, if we cancombine these results with the fluctuations in mood,we might be able to come up with an even strongerguess of how the user felt about a particular song.

APPENDIX ASURVEY GIVEN TO PARTICIPANTS

The survey as presented is shown here. Each ques-tion / action was displayed to the participant for tenseconds before disappearing. Another five secondselapsed between questions.

1) Have you had a meal in the last twenty fourhours?

2) Have you left the country in the last twentyfour hours?

3) Have you ever been to the state of Virginia?4) Do you like chocolate?5) Have you ever been to Havana?6) Can you run a mile in under 5 minutes?7) Can you ride a bike?8) Can you whistle?


9) Have you ever purchased a television?10) Did you have coffee this morning?11) Please move your eyebrows up and down

for ten seconds.12) Please blink slowly for ten seconds.13) Please wink with your left eye only for ten

seconds.14) Please wink with your right eye only for ten

seconds.15) Please use your eyes to look left and back to

center. Repeat this for ten seconds.16) Please use your eyes to look right and back

to center. Repeat this for ten seconds.17) Please use your eyes to look up and back to

center. Repeat this for ten seconds.18) Please use your eyes to look down and back

to center. Repeat this for ten seconds.

REFERENCES

[1] F. Beverina, G. Palmas, S. Silvoni, F. Piccione, and S. Giove,“User adaptive bcis: SSVEP and P300 based interfaces,”PsychNology Journal, 2003.

[2] S. Amiri, A. Rabbi, L. Azinfar, and R. Fazel-Rezai, “Areview of P300, SSVEP and hybrid P300/SSVEP brain-computer interface systems,” Biomedical Image and SignalProcessing Laboratory, Department of Electrical Engineering,University of North Dakota, 2013.

[3] M. Jackson and M. Rudolph, “Applications for brain-computer interfaces,” IEEE, 2010.

[4] A. Stopczynski, J. E. Larsen, C. Stahlhut, M. K. Petersen,and L. K. Hansen, “A smartphone interface for a wirelessEEG headset with real-time 3d reconstruction.”

[5] E. J. Rechy-Ramirez, H. Hu, and K. McDonald-Maier,“Head movements based control of an intelligentwheelchair in an indoor environment,” IEEE, 2012.

[6] C. Pereira, R. Neto, A. Reynaldo, M. Canndida de Mi-randa Luza, and R. Oliveira, “Development and evalua-tion of a head-controlled human-computer interface withmouse-like functions for physically disabled users,” Clinics,2009.

[7] S. Fazli, M. Danoczy, F. Popescu, B. Blankertz, and K.-R.Muller, “Using rest class and control paradigms for braincomputer interfacing,” IWANN, 2009.

an exploration on the potential of an electroencephalographic headset for human computer interaction

Documents