assessing drivers’ physiological responses using consumer

Assessing Drivers’ Physiological Responses Using Consumer Grade DevicesTimotej Gruden * , Kristina Stojmenova, Jaka Sodnik and Grega Jakus
University of Ljubljana, Faculty of Electrical Engineering, Traška Cesta 25, 1000 Ljubljana, Slovenia; [email protected] (K.S.); [email protected] (J.S.); [email protected] (G.J.) * Correspondence: [email protected]; Tel.: +386-1-476-8951
Received: 30 October 2019; Accepted: 4 December 2019; Published: 7 December 2019
Abstract: The ability to measure drivers’ physiological responses is important for understanding their state and behavior under different driving conditions. Such measurements can be used in the development of novel user interfaces, driver profiling, advanced driver assistance systems, etc. In this paper, we present a user study in which we performed an evaluation of two commercially available wearable devices for assessment of drivers’ physiological signals. Empatica’s E4 wristband measures blood volume pulse (BVP), inter-beat interval (IBI), galvanic skin response (GSR), temperature, and acceleration. Bittium’s Faros 360 is an electrocardiographic (ECG) device that can record up to 3-channel ECG signals. The aim of this study was to explore the use of such devices in a dynamic driving environment and their ability to differentiate between different levels of driving demand. Twenty-two participants (eight female, 14 male) aged between 18 and 45 years old participated in the study. The experiment compared three phases: Baseline (no driving), easy driving scenario, and demanding driving scenario. Mean and median heart rate variability (HRV), standard deviation of R–R intervals (SDNN), HRV variables for shorter time frames (standard deviation of the average R–R intervals over a shorter period—SDANN and mean value of the standard deviations calculated over a shorter period—SDNN index), HRV variables based on successive differences (root mean square of successive differences—RMSSD and percentage of successive differences, greater than 50 ms—pNN50), skin temperature, and GSR were observed in each phase. The results showed that motion artefacts due to driving affect the GSR recordings, which may limit the use of wrist-based wearable devices in a driving environment. In this case, due to the limitations of the photoplethysmography (PPG) sensor, E4 only showed differences between non-driving and driving phases but could not differentiate between different levels of driving demand. On the other hand, the results obtained from the ECG signals from Faros 360 showed statistically significant differences also between the two levels of driving demand.
Keywords: physiological signals; heart rate variability (HRV); user study; validation; driving environment
1. Introduction
The human factor is still one of the leading causes for road traffic accidents. It has been reported that more years of life were lost due to traffic accidents than due to most human diseases [1]. Driving is a demanding process, mostly relying on the driver’s visual and manual senses, and largely to their auditory and cognitive capabilities. Vehicle manufactures have therefore been constantly working on reducing the driver’s role and improving the driving experience by adding a number of advanced driver-assistance systems (ADAS) and including user-friendly in-vehicle information systems (IVIS). Although the driver’s role is changing, people spend more time in a vehicle compared to ever before,
Appl. Sci. 2019, 9, 5353; doi:10.3390/app9245353 www.mdpi.com/journal/applsci
Appl. Sci. 2019, 9, 5353 2 of 23
thus monitoring the driver’s state is somewhat necessary for further research in current and higher levels of vehicle autonomy.
The measurements of drivers’ physiological signals can be very useful in the development of automated vehicles that tend to imitate human driving, since driving algorithms can be improved with the results of such measurements. Not only vehicles, driving simulators can also use such algorithms to make the scenarios (other vehicles in the scenarios) more realistic. Determining different drivers’ state and behavior from the measured signals is also beneficial to insurance companies for driver profiling and cost calculation. Among other factors, stress levels, cognitive demand, attention, arousal, and fatigue indicators have been used for evaluation of usability of new ADAS and IVIS systems. These indicators are mainly derived from drivers’ physiological signals that represent cardiovascular, electrodermal, pupil, or brain activity.
1.1. Physiological Responses for Assessment of Driver State
Among all studies that measure physiological signals of drivers, driver drowsiness and fatigue are the most often researched topics. Chronologically, the first studies on detection of physiological signals for drowsiness detection were using video analysis with image processing techniques [2,3]. Now, modern research often uses electroencephalography (EEG) [4–7] or a combination of EEG with other physiological signals [8,9]. Another common measure for detecting drowsiness besides EEG, is cardiovascular activity. Drowsiness has been assessed with different cardiovascular signals including photoplethysmography (PPG) [9–12], blood volume pulse (BVP) [11], and electrocardiogram (ECG) [8,12–14]. Some studies report on using heart rate variability (HRV) [9,12,13], electrooculography (EOG) [8,15,16], and galvanic skin response (GSR) [9,11].
Another often studied topic regarding the driver’s physiological signals is monitoring of individual driver’s state such as situational awareness [17], emotional states [18–20], or driver physical fitness [21–23]. Similar methods have also been used for observing effects of use of different modalities when interacting with in-vehicle information systems [24] and to assess drivers’ cognitive workload due to use of such systems [25,26] or experienced stress due to demanding driving conditions [27]. By observing EEG signals, physiological data can also be used for prediction of drivers’ intentions [28,29].
Difficult driving conditions, e.g., interaction with reckless drivers and bikers, impatient pedestrians crossing the road, and operating the vehicle, e.g., gear shifting, can significantly increase the load on drivers [30]. Different levels of driving demand reflect in drivers’ mental and also physical load. What makes the situation even worse is that the driving demand is increasing every day [31]. The driver’s mental workload is usually assessed by subjective reports, measures of task performance, or physiological measures [32]. As Brookhius and de Waard suggest, physical and mental workload have clear impact on physiological signals [31], especially on heart rate, heart rate variability, galvanic skin response, and blood pressure [33].
In summary, the list of common physiological measures for assessing drivers consists of:
• Electroencephalography (EEG), • Electrocardiography (ECG), • Photoplethysmography (PPG), • Heart rate (HR), • Galvanic skin response (GSR), • Electromyography (EMG) and • Eye tracking (pupil diameter—PD).
Combinations often include EEG and ECG or PPG and GSR, sometimes in combination with eye tracking.
Appl. Sci. 2019, 9, 5353 3 of 23
1.2. Devices Used to Measure Driver’s Physiological Responses
There is a great variety of available devices for capturing physiological signals. Most often higher reliability increases the price of the device, and thus reduces the availability. Devices used for signal capturing vary from study to study. Some even make their own custom capturing devices. Among the presented studies, Biopac MP-150 was often used as an overall acquisition system [7,20,27], also FlexComp Infinity system [8,34], Nexus-10 from Mind Media Schepersweg [11], Medac Systems/3 [25], and Geodesic EEG System 300 [35] were mentioned. EEG was measured with NicoletOne Ambulatory EEG [6], EMG was measured with Bagnoli-8™ [23], eye movement with FaceLAB® [25]. Bittium’s Faros was also used in many studies [36–38]. Based on the available research it can be concluded that physiological signals can give important information about the human’s state and prediction of human behavior or medical state. It is also noticeable that with the introduction of wearable devices, the research studies using such signals are no longer limited to medical and laboratory environments, but can be used also in dynamic environments such as vehicles. This can however bring “noise” (e.g., motion artefacts, quantization noise, electro-magnetic interferences from other devices, etc.) to the measured signals and therefore, it is important to be aware of this limitation when choosing a measuring device and later when performing the analysis. Therefore, we assessed suitability, accuracy, and robustness of two commonly used commercially available devices in a simulated driving environment.
The E4 wristband can measure BVP, inter-beat interval (IBI), heart rate (HR), electrodermal activity (EDA) or GSR, skin temperature (ST), and motion with an accelerometer. A detailed presentation of the device can be found in the next section.
In many studies, the raw data from the E4 device was used for assessing stress levels [39–44]. Sevil et al. used all of the available E4 output signals (GSR, ST, HR, BVP, and motion) to detect various types of acute stress such as social, completion, emotional and mental stress, and showed that using their algorithm, they can detect stress with 87% accuracy [39]. Park et al. on the other hand tried to estimate the relationship between EDA signals and increased cognitive workload using the E4 [45]. Their results showed that there is a somewhat linear relationship between these data. Furthermore, it has also been used in studies of affective computing in general [40,46], in medical researches for heart arrhythmia and atrial fibrillation detection [47–49]. Studies report also on using E4 data for assessing overall functional health of people with dementia [50] and creating of personal assistants for forgetful people [51]. Vandecasteele et al. even tried to use the HR data for seizure detection; however, their results showed that detection performance was considerably lower to the usually used hospital and wearable Faros device [52]. Additionally, there seems to be a growing trend of using E4 data in biofeedback applications [53–55].
When compared to other laboratory (stationary and wearable) devices, E4 showed comparable results. Ollander et al. compared the E4 device with stationary sensors for ECG and GSR [41]. Their results showed incomplete IBI data, however mean values of HR and standard deviation of HR showed good stress discrimination power. For the GSR, their results showed that due to the placement, data from the E4 yielded higher stress discrimination power than the signal measured with the stationary sensors placed at the fingers. When compared for emotion recognition to higher grade laboratory equipment—Biopac MP150, it was found that E4 can provide comparable results [40,56]. However, based on currently available data, no research was found on E4 being used in a driving environment.
Faros is a family of ECG devices by Bittium. Faros 360 is able to measure 3-channel ECG, body temperature, and accelerations. It is presented in detail along with different models in the next section.
Different versions of Faros are widely used in research studies for detection of various medical conditions, which affect humans’ cardiovascular activity [57–60].
Since Faros 360 is certified as a medical device, it is also often used as a reference for calibration or validation of new wearable devices [61–63]. However, Faros has been used also for monitoring different profiles of users’ states. For example, it has been used for the assessment of mental stress when interacting with robots, where, based on HRV data, it was found that human operators show higher engagement levels when being successful at completing a task [64]. It has also been used for
Appl. Sci. 2019, 9, 5353 4 of 23
monitoring effects of air pollution and noise due to traffic, and the results showed that HRV and blood pressure (BP) drop when noise increases [65].
Other studies have shown that the ECG signal from Faros can also be used to calculate other variables such as BP [66], or to extract core body temperature with respect to clothing and persons’ activity [67].
Since with wearable devices such as Faros, the experimental conditions differ and are not always performed in a controlled laboratory environment, the gathered data can be affected by motion artefacts. Alikhani et al. found that motion can describe the high frequencies in HRV up to 40% [68]. However, the noise of motion can be eliminated by adding an additional accelerometer on each electrode [69]. With this, it is possible to eliminate the baseline wandering without affecting the ECG signal.
Contrary to E4, Faros has already been used in a driving environment. Biondi et al. used HRV data gathered with Faros to observe the effects of semi-automated driving in Tesla model S [36]. The effects of different levels of automation on drivers’ state were studied also in a driving simulator environment by Radlmayr, et al. [38] and Murase et al. [37]. Their findings were however different, as Radlmayr et al. report on a decreased level of arousal and sympathetic nervous system activity when switching from automation level 2 to level 3, while Murase et al. did not find any effects on the ECG channel for different levels of autonomous driving [37].
1.3. Contribution
This research study explores two new and commercially available devices, which have shown promising results for assessing psychophysiological data in various medical research. The goal was to evaluate the two devices and their usefulness also in a driving environment, which is rather specific, and due to its dynamic component, differs from other experimental environments used in reported studies. Our motivation on evaluating these particular devices was the fact that they are wearable and nonintrusive, and therefore much easier to use in a simulated driving study and also in a real environment. We try to answer to the following research questions:
• Can E4 and Faros be used for assessment of physiological signals in a driving environment? • If so, can E4 and Faros differentiate between different levels of driving demand?
The used methodology and experimental environment are presented in the Materials and Methods section and our findings are reported in Results. The possible applications and found limitations of both devices in a driving environment are discussed in the last section: Discussion and Conclusions.
2. Materials and Methods
The presented study had a within-subject (repeated measures) design and was conducted in a simulated driving environment. The drivers’ responses were assessed using two commercially available devices—E4 by Empatica [70] and Faros 360 by Bittium [71], whereas the driving measurements were recorded using a Nervtech driving simulator [72].
2.1. Empatica E4
E4 is a wearable device in the form of a wristband, equipped with a number of sensors for assessment of electrodermal and cardiovascular activity (Figure 1). The E4 wrist band can measure BVP, inter-beat interval (IBI), heart rate (HR), electrodermal activity (EDA) or GSR, skin temperature (ST), and motion with an accelerometer.
Appl. Sci. 2019, 9, 5353 5 of 23 Appl. Sci. 2019, 9, x 5 of 23
Figure 1. Empatica E4 with positions of its sensors.
One of the main features of the E4 device is a photoplethysmography (PPG) sensor. Photoplethysmography sensors use an optical technique to detect changes in blood volume in the microvascular bed of tissue [73]. The PPG sensor in E4 consists of two red and two green LEDs, which provide two different wavelength light sources, and two sensors for detection of the reflected light, with a sampling frequency fs = 64 Hz. The output is a blood volume pulse (BVP) signal with an 8-bit resolution. Based on these measurements, a proprietary algorithm from Empatica detects the heart beats (peaks) in the signal and calculates the intervals between separate beats (inter-beat intervals (IBI)) in milliseconds. Unlike the BVP signal, which has a fixed sampling frequency, the IBI signal is provided when a beat is detected. However, average heart rate (HR) signal value is provided every second, as it is calculated from the last known IBI sample.
E4 is equipped also with an electrodermal activity (EDA) or galvanic skin response (GSR) sensor. With two 8-mm-wide silver-plated copper electrodes, located at the end of the wristband belt (see Figure 1), E4 captures skin conductance with a sampling frequency fs = 4 Hz.
Additionally, E4 also features an optical infrared temperature sensor that measures skin temperature assessed at the wrist, with a sampling frequency fs = 4 Hz.
E4 is equipped also with a 3-axis accelerometer, which allows monitoring of hand activity (movement). On every E4 device, the X-axis is defined as a vector from the center of device towards the USB connector, Y-axis is defined as a vector from the center of the device towards the shorter end of the wrist belt, and Z-axis is defined as a vector from the center of the device towards the bottom of the device. The dynamic range of the accelerometer is ±2 g, the output is an 8-bit signed integer with the resolution of 1/64 g. The accelerometer’s sampling frequency (fs) is 32 Hz.
With the E4 device, data can be captured in two ways: In a recording mode (retrieve after measurement) and in a streaming mode (retrieved in real-time). E4 has an internal storage of up to 60 h of measurements. It comes prepacked with a special USB dock station and proprietary software for downloading data from the device and uploading it to a cloud platform called Empatica connect [74]. When using the recording mode, raw data can only be downloaded from this cloud platform. In the streaming mode, E4 uses Bluetooth Low Energy (BLE) to transfer data in real-time.
Besides the official software solutions that come with the device, Empatica offers also iOS, Android and Windows development kits for researchers that want to customize and upgrade this solution to their needs. The development kits allow real-time data acquisition with custom software that gives the ability of real-time processing, feature extraction and on-demand visualization.
2.2. Bittium Faros 360
Faros by Bittium is a small electrocardiogram (ECG) measurement device. It comes in three versions:
• Faros 90, which offers simple 1-channel ECG measurements,
Figure 1. Empatica E4 with positions of its sensors.
One of the main features of the E4 device is a photoplethysmography (PPG) sensor. Photoplethysmography sensors use an optical technique to detect changes in blood volume in the microvascular bed of tissue [73]. The PPG sensor in E4 consists of two red and two green LEDs, which provide two different wavelength light sources, and two sensors for detection of the reflected light, with a sampling frequency fs = 64 Hz. The output is a blood volume pulse (BVP) signal with an 8-bit resolution. Based on these measurements, a proprietary algorithm from Empatica detects the heart beats (peaks) in the signal and calculates the intervals between separate beats (inter-beat intervals (IBI)) in milliseconds. Unlike the BVP signal, which has a fixed sampling frequency, the IBI signal is provided when a beat is detected. However, average heart rate (HR) signal value is provided every second, as it is calculated from the last known IBI sample.
E4 is equipped also with an electrodermal activity (EDA) or galvanic skin response (GSR) sensor. With two 8-mm-wide silver-plated copper electrodes, located at the end of the wristband belt (see Figure 1), E4 captures skin conductance with a sampling frequency fs = 4 Hz.
Additionally, E4 also features an optical infrared temperature sensor that measures skin temperature assessed at the wrist, with a sampling frequency fs = 4 Hz.
E4 is equipped also with a 3-axis accelerometer, which allows monitoring of hand activity (movement). On every E4 device, the X-axis is defined as a vector from the center of device towards the USB connector, Y-axis is defined as a vector from the center of the device towards the shorter end of the wrist belt, and Z-axis is defined as a vector from the center of the device towards the bottom of the device. The dynamic range of the accelerometer is ±2 g, the output is an 8-bit signed integer with the resolution of 1/64 g. The accelerometer’s sampling frequency (fs) is 32 Hz.
With the E4 device, data can be captured in two ways: In a recording mode (retrieve after measurement) and in a streaming mode (retrieved in real-time). E4 has an internal storage of up to 60 h of measurements. It comes prepacked with a special USB dock station and proprietary software for downloading data from the device and uploading it to a cloud platform called Empatica connect [74]. When using the recording mode, raw data can only be downloaded from this cloud platform. In the streaming mode, E4 uses Bluetooth Low Energy (BLE) to transfer data in real-time.
Besides the official software solutions that come with the device, Empatica offers also iOS, Android and Windows development kits for researchers that want to customize and upgrade this solution to their needs. The development kits allow real-time data acquisition with custom software that gives the ability of real-time processing, feature extraction and on-demand visualization.
2.2. Bittium Faros 360
Faros by Bittium is a small electrocardiogram (ECG) measurement device. It comes in three versions:
• Faros 90, which offers simple 1-channel ECG measurements,
Appl. Sci. 2019, 9, 5353 6 of 23
• Faros 180, which offers 1-channel ECG measurements and is able to stream data via Bluetooth, and • Faros 360, which enables 3-channels ECG measurements and is able to stream data via Bluetooth.
Each version can be mounted to the participant’s body in three ways: Fast-Fix, using a cable set, or using a stingray adapter. Fast-Fix, which is a Bittium’s proprietary electrode, is designed for quick mounting and is considered as the simplest and most convenient way of using the device. Faros can be mounted also by using cable sets (see Figure 2 left) with up to five electrodes for 3-channel measurements (or three electrodes for 1-channel). This version is considered as the most accurate and reliable version of Faros for measuring ECG because it enables best skin contact during the measurement process. The third mounting option does not require placing any electrodes on the participant’s body. It instead uses an elastic textile belt with two electrodes and a mounting pad for Faros (see Figure 2 right).
Appl. Sci. 2019, 9, x 6 of 23
• Faros 180, which offers 1-channel ECG measurements and is able to stream data via Bluetooth, and
• Faros 360, which enables 3-channels ECG measurements and is able to stream data via Bluetooth.
Figure 2. Faros 360 with cable set for measuring with electrodes (left) and Faros 360 with textile belt for electrode-free measurements (right).
The most important feature of Faros is its ECG sensor. The sampling frequency of the provided raw ECG data can be fs1 = 125 Hz, fs2 = 250 Hz, fs3 = 500 Hz, or fs4 = 1000 Hz depending on the user’s needs. From the ECG signal, Faros automatically detects independent heart beats as points R in a PQRST model of the ECG signal, where P, Q, R, S, and T represent specific waves in the signal as presented in Figure 3. It also derives heart rate variability (HRV), sometimes also referred to as R–R or N–N interval, from the time intervals between the consecutive R points, and presents it as a single output in addition to raw ECG.
Figure 3. PQRST model and R–R interval.
Faros is equipped also with a temperature sensor. According to the manufacturer’s specifications [75], its characteristic is linear only between 35 and 45 degrees Celsius when the sampling frequency is fs = 5 Hz.
Faros features also an accelerometer with two possible sampling frequencies: fs1 = 25 Hz and fs2 =
100 Hz. Its dynamic range can be set to ±2 g, ±4 g or ±16 g. Faros 360 can operate in one of two modes: Datalogger mode and online mode. To use the datalogger
mode, Faros has to be connected to a computer, and the appropriate measurement settings have to be set using the Faros Manager software [71]. Raw data is stored on Faros’ internal storage in a European data format (EDF) file. It can be viewed using the provided EDF viewer or can easily be converted to acceleration (ASC) or heart rate variability (SDF) file format. In online mode, Faros periodically (five times in a second) sends data via Bluetooth. After Faros is paired with a computer (or another Bluetooth device), the computer sees Faros as an external serial device accessible over the serial communication (COM) port. The protocol itself and data packet specifications are provided in Bittium’s document 810378 [75].
Appl. Sci. 2019, 9, 5353 7 of 23
Appl. Sci. 2019, 9, x 6 of 23
• Faros 180, which offers 1-channel ECG measurements and is able to stream data via Bluetooth, and
• Faros 360, which enables 3-channels ECG measurements and is able to stream data via Bluetooth.
Figure 3. PQRST model and R–R interval. Figure 3. PQRST model and R–R interval.
2.3. Nervtech Driving Simulator Overview
To ensure a controllable and repetitive driving environment, the study was performed in a driving simulator. We used Nervtech’s high fidelity driving simulator [72], using a triple screen set-up (Figure 4) with adjustable car seat, Fanatec ClubSport Wheel Base V2 with dynamic feedback, Fanatech ClubSport Pedals V3 with dynamic feedback, and Fanatec gear box. Total resolution of the three Samsung Curved 48′ TVs is 5760 × 1080 px, using Nvidia GeForce GTX 1070 Ti graphic card. The on-screen cockpit included speedometer and fuel gauge. Motion cues are provided by slightly moving the cockpit up and down. The simulator uses OKTAL’s Scanner Studio, which is managed by a supervising platform OktalControl by Nervtech. Based on the definition of high physical and functional fidelity as defined by Kinkade and Wheaton [76] and Hays [77], the simulator used in this study qualifies as a high-fidelity driving simulator.
Appl. Sci. 2019, 9, x 7 of 23
Faros is equipped also with a temperature sensor. According to the manufacturer’s specifications [75], its characteristic is linear only between 35 and 45 degrees Celsius when the sampling frequency is fs = 5 Hz.
Faros features also an accelerometer with two possible sampling frequencies: fs1 = 25 Hz and fs2 = 100 Hz. Its dynamic range can be set to ±2 g, ±4 g or ±16 g.
Faros 360 can operate in one of two modes: Datalogger mode and online mode. To use the datalogger mode, Faros has to be connected to a computer, and the appropriate measurement settings have to be set using the Faros Manager software [71]. Raw data is stored on Faros’ internal storage in a European data format (EDF) file. It can be viewed using the provided EDF viewer or can easily be converted to acceleration (ASC) or heart rate variability (SDF) file format. In online mode, Faros periodically (five times in a second) sends data via Bluetooth. After Faros is paired with a computer (or another Bluetooth device), the computer sees Faros as an external serial device accessible over the serial communication (COM) port. The protocol itself and data packet specifications are provided in Bittium’s document 810378 [75].
2.3. Nervtech Driving Simulator Overview
To ensure a controllable and repetitive driving environment, the study was performed in a driving simulator. We used Nervtech’s high fidelity driving simulator [72], using a triple screen set- up (Figure 4) with adjustable car seat, Fanatec ClubSport Wheel Base V2 with dynamic feedback, Fanatech ClubSport Pedals V3 with dynamic feedback, and Fanatec gear box. Total resolution of the three Samsung Curved 48′ TVs is 5760 × 1080 px, using Nvidia GeForce GTX 1070 Ti graphic card. The on-screen cockpit included speedometer and fuel gauge. Motion cues are provided by slightly moving the cockpit up and down. The simulator uses OKTAL’s Scanner Studio, which is managed by a supervising platform OktalControl by Nervtech. Based on the definition of high physical and functional fidelity as defined by Kinkade and Wheaton [76] and Hays [77], the simulator used in this study qualifies as a high-fidelity driving simulator.
Figure 4. Nervtech driving simulator with three Samsung Curved TV 48’.
Figure 4. Nervtech driving simulator with three Samsung Curved TV 48’.
Appl. Sci. 2019, 9, 5353 8 of 23
2.4. Technical Setup
E4 was placed on the participant’s left hand (Figure 5). For this study, we used streaming mode to store data captured with E4 using Windows Streaming Server, which requires a specific BLE USB dongle to connect to the E4 wristband [78]. Once the Windows Streaming Server and E4 are connected, users or software can connect to the streaming server via the TCP socket. For this study, we created a custom software that connects to the streaming server and stores real-time data for further analysis.
From the three available versions for Faros, we used Faros 360, which enables 3-channel ECG measurement and is able to stream data via Bluetooth. For convenience of use, the stingray adapter was used for mounting the device on the participants (Figure 5). Following the recommendations by Laborde et al. we used the sampling frequency fs = 500 Hz [79].
Appl. Sci. 2019, 9, x 8 of 23
2.4. Technical Setup
E4 was placed on the participant’s left hand (Figure 5). For this study, we used streaming mode to store data captured with E4 using Windows Streaming Server, which requires a specific BLE USB dongle to connect to the E4 wristband [78]. Once the Windows Streaming Server and E4 are connected, users or software can connect to the streaming server via the TCP socket. For this study, we created a custom software that connects to the streaming server and stores real-time data for further analysis.
From the three available versions for Faros, we used Faros 360, which enables 3-channel ECG measurement and is able to stream data via Bluetooth. For convenience of use, the stingray adapter was used for mounting the device on the participants (Figure 5). Following the recommendations by Laborde et al. we used the sampling frequency fs = 500 Hz [79].
Figure 5. Positioning of Empatica E4 and Bittium Faros 360. E4 should be placed on the wrist of a non- dominant hand in a way that electrodes are on the line that goes between middle and ring finger. Faros with the textile belt should be placed right below the chest muscle [80].
2.5. User Study
2.5.1. Participants
Since the distribution of different factors that could affect the measurement in different ways is usually unknown, the American Association of Psychologists recommends using a slightly corrected Cohen’s effect sizes. This study was aimed to detect large effect sizes with 80% statistical power. Study [81] made some calculations regarding statistical considerations and it turned out that for statistical power of 80% at α = 0.05 in order to detect large effect sizes, a sample size of at least 21 participants is required.
Therefore, 22 drivers, aged from 18 to 45 participated in this study, eight of them (36%) were female. There were no drivers with any kind of heart diseases or other relevant health problems. A requirement for every driver to participate was a valid driving license.
All subjects gave their informed consent for inclusion before they participated in the study. The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the head of the Laboratory of Information Technologies, Faculty of Electrical Engineering, University of Ljubljana.
2.5.2. Tasks
This study had a within-subject design. It consisted of three phases: Baseline, easy driving, and demanding driving. The duration of each phase was 6–8 min to ensure at least a 5-min-long window of data, as recommended for short-term recordings of physiological signals by the European Society of Cardiology [79,82].
Figure 5. Positioning of Empatica E4 and Bittium Faros 360. E4 should be placed on the wrist of a non-dominant hand in a way that electrodes are on the line that goes between middle and ring finger. Faros with the textile belt should be placed right below the chest muscle [80].
2.5. User Study
2.5.1. Participants
Since the distribution of different factors that could affect the measurement in different ways is usually unknown, the American Association of Psychologists recommends using a slightly corrected Cohen’s effect sizes. This study was aimed to detect large effect sizes with 80% statistical power. Study [81] made some calculations regarding statistical considerations and it turned out that for statistical power of 80% at α = 0.05 in order to detect large effect sizes, a sample size of at least 21 participants is required.
Therefore, 22 drivers, aged from 18 to 45 participated in this study, eight of them (36%) were female. There were no drivers with any kind of heart diseases or other relevant health problems. A requirement for every driver to participate was a valid driving license.
All subjects gave their informed consent for inclusion before they participated in the study. The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by the head of the Laboratory of Information Technologies, Faculty of Electrical Engineering, University of Ljubljana.
2.5.2. Tasks
This study had a within-subject design. It consisted of three phases: Baseline, easy driving, and demanding driving. The duration of each phase was 6–8 min to ensure at least a 5-min-long window
Appl. Sci. 2019, 9, 5353 9 of 23
of data, as recommended for short-term recordings of physiological signals by the European Society of Cardiology [79,82].
Phase 1: Baseline
Physiological signals such as HR and HRV differ among people therefore, we first had to collect baseline data for each participant. Since there is not a standardized procedure for baseline measurements, we followed the recommendations by Laborde et al. [79]. To create similar environment with the following driving tasks, the baseline measurements were also collected in a sitting position. The test drivers were told to sit in a chair with their legs straight (not crossed), keeping their knee angles at 90 degrees. They were asked to try to relax and stay calm, to lean their head back, keep their eyes closed, and to ensure normal blood flow to the wrists, place their arms on the supporters with hands facing up (Figure 6a).
Appl. Sci. 2019, 9, x 9 of 23
Phase 1: Baseline
Physiological signals such as HR and HRV differ among people therefore, we first had to collect baseline data for each participant. Since there is not a standardized procedure for baseline measurements, we followed the recommendations by Laborde et al. [79]. To create similar environment with the following driving tasks, the baseline measurements were also collected in a sitting position. The test drivers were told to sit in a chair with their legs straight (not crossed), keeping their knee angles at 90 degrees. They were asked to try to relax and stay calm, to lean their head back, keep their eyes closed, and to ensure normal blood flow to the wrists, place their arms on the supporters with hands facing up (Figure 6a).
The 5-min measurement began with a short beep from Faros that indicated a successful initialization of measurement system. During this period, all light and noise sources were turned off and remained off until the end of the study process in order to reduce potential distractions that could affect the measurement process.
Phase 2: Easy Driving
Phase 2 and 3 were performed in a driving simulator. The participants’ task was safe driving in a very simple driving scenario (Figure 6b). The participants were asked to drive on a two-way one- lane six-kilometer country road with no other vehicles or pedestrians. The speed limit was 90 km/h, and participants were not asked to follow any pre-defined path.
Phase 3: Demanding Driving
In the third phase, the participants’ primary task remained safe driving. However, in this task the participants had to follow a specific route communicated with a navigation system displayed on a built-in head-up display. In this phase, the driving scenario was much more demanding as it featured six kilometers of two-lane highway and two kilometers of city road with a lot of vehicles, intersections and pedestrians.
While on the highway (Figure 6c), the participants faced many critical situations, including a truck driving in the opposite direction, work on the road, unsafe overtaking by another vehicle, and traffic accident while driving in fog. The driver experienced critical situations also on the city road such as pedestrians crossing the road outside crosswalks, a bike forcing the driver’s right of way, a child on the road running after a ball, etc.
Figure 6. Driver’s position during three phases of measurement: (a) Baseline, (b) easy driving, (c) demanding driving [80]. Figure 6. Driver’s position during three phases of measurement: (a) Baseline, (b) easy driving, (c) demanding driving [80].
The 5-min measurement began with a short beep from Faros that indicated a successful initialization of measurement system. During this period, all light and noise sources were turned off and remained off until the end of the study process in order to reduce potential distractions that could affect the measurement process.
Phase 2: Easy Driving
Phase 2 and 3 were performed in a driving simulator. The participants’ task was safe driving in a very simple driving scenario (Figure 6b). The participants were asked to drive on a two-way one-lane six-kilometer country road with no other vehicles or pedestrians. The speed limit was 90 km/h, and participants were not asked to follow any pre-defined path.
Phase 3: Demanding Driving
In the third phase, the participants’ primary task remained safe driving. However, in this task the participants had to follow a specific route communicated with a navigation system displayed on a built-in head-up display. In this phase, the driving scenario was much more demanding as it featured
Appl. Sci. 2019, 9, 5353 10 of 23
six kilometers of two-lane highway and two kilometers of city road with a lot of vehicles, intersections and pedestrians.
While on the highway (Figure 6c), the participants faced many critical situations, including a truck driving in the opposite direction, work on the road, unsafe overtaking by another vehicle, and traffic accident while driving in fog. The driver experienced critical situations also on the city road such as pedestrians crossing the road outside crosswalks, a bike forcing the driver’s right of way, a child on the road running after a ball, etc.
2.5.3. Variables
All output signals from both Faros 360 and E4 were recorded and processed. Comparison between the two devices could be made with HRV and temperature signals. Since only Faros provides raw ECG signal, the evaluation of other ECG features than the ones already extracted by the devices (HRV) is not feasible. The difference with HRV is only in signal capturing methods which could provide significantly different results. Faros measures ECG signal to obtain R–R intervals (HRV) while E4 uses photoplethysmography (PPG) to obtain BVP and IBI. Two temperature readings could also provide significantly different results since measures take part at different body parts with different thermal conductance. In addition to the comparison, GSR signal from E4 was also analyzed to provide the full picture of E4’s usability.
Independent variable in our tests was each phase’s driving difficulty that could take one of three values: 1 = baseline establishment, 2 = easy driving, 3 = demanding driving.
Dependent variables for both Faros and E4 were:
• Mean and median HRV, • HRV SDNN—standard deviation of the R–R intervals (also known as normal-to-normal or N–N
intervals, Figure 3), which reflects cyclic components during the measurement, • HRV SDANN—standard deviation of the average R–R intervals over a shorter period (10 s), it
reflects changes due to longer cycles, • HRV SDNN index—mean of the standard deviations, calculated over a shorter period (10 s), it
reflects changes due to shorter cycles, • HRV RMSSD—root mean square of successive differences, which reflects parasympathetic nerve
system activity and is not affected by respiration process, • HRV pNN50—the number of successive differences, greater than 50 ms, derived by the number of
total N–N differences, should be highly correlated to RMSSD, • Temperature mean and standard deviation.
Mean and standard deviation of recorded GSR signals were also analyzed as dependent variables from E4.
2.5.4. Procedure
The study began with a demographic questionnaire and a consent form for participants’ personal data processing. After that, Empatica E4 and Bittium Faros 360 were disinfected, turned on, and placed on participants’ wrist and chest respectively. Exact position of both devices is shown in Figure 5.
In [82], Task Force of the European Society of Cardiology set standards for ECG and HRV measurements. According to them, measurements of short events should take at least 5 min. Therefore, the baseline and both tasks lasted at least 5 min and to make the sample size equal, we only observed the last 5 min. Phase three was usually longer than 7 min (it depended on participants’ driving), so we only cut samples from the first half-minute to avoid having artefacts from initial driver movements.
Before the driving, each participant was introduced to and shown how to use the simulator. Participants were asked to follow the traffic rules and try to drive as close as possible as they would in real life.
Appl. Sci. 2019, 9, 5353 11 of 23
After completing all three phases, the drivers were asked to subjectively assess their feelings regarding simulation sickness. The values are equally distributed on a 10-point scale, where 1 indicated “I am perfectly fine”, and 10 meant “I have to throw up”.
2.5.5. Statistical Analysis
The statistical analysis included Shapiro-Wilk normality test and Maunchly’s test of sphericity for every dependent variable. Where the distribution of the data was found normal, repeated measures ANOVA (RMANOVA) and (when the null hypothesis was rejected) Bonferroni post-hoc test. For normally distributed data with violated assumption of sphericity, Greenhouse-Geisser correction was applied.
For variables with non-normally distributed data we used Friedman’s non-parametric test and (when the null hypothesis was rejected) Dunn’s post-hoc test with Bonferroni correction.
3. Results
The results of HRV measurements are presented in four groups: Mean, standard deviation, shorter time-frame variables and successive differences, followed by skin temperature and GSR measurements. A summary of the results is provided in Table 1.
Table 1. Results of each device (columns) and each dependent variable (rows). ‘<’ indicates a statistically significant increase in the corresponding dependent variable between the two phases. X indicates that no significant differences among three phases of driving were found. N/A indicates the measurements were not applicable for the corresponding device.
E4 Faros 360
HRV SDNN baseline < easy driving baseline < demanding driving
easy driving < demanding driving baseline < demanding driving
HRV SDANN baseline < easy driving baseline < demanding driving
easy driving < demanding driving baseline < demanding driving
HRV SDNN Index X X
HRV RMSSD baseline < easy driving baseline < demanding driving X
HRV pNN50 baseline < easy driving baseline < demanding driving X
Mean skin temperature baseline < easy driving baseline < demanding driving
baseline < easy driving baseline < demanding driving
easy driving < demanding driving
Mean and standard deviation of GSR baseline < demanding driving N/A
It is important to mention that E4’s built-in algorithm for calculating the HRV did often not provide any output. This resulted in missing HRV samples from E4 at easy driving (58%) and demanding driving (59%). It is important to notice that the missing values may have influenced the results of HRV successive differences, therefore they should be interpreted with caution. On the other hand, Faros provided all HRV samples during every measurement phase.
Appl. Sci. 2019, 9, 5353 12 of 23
3.1. Mean Heart Rate Variability (HRV)
RMANOVA test for mean values of HRV (Figure 7) did not reveal any statistically significant differences among the data collected during the three phases for E4 (F(1.166, 22.148) = 0.308, p = 0.619) or Faros (F(1.331, 27.941) = 2.330, p = 0.131). The same test did not reveal any statistically significant differences also in median HRV among the observed phases for E4 (F(1.219, 23.154) = 0.366, p = 0.593) or Faros (F(1.323, 27.774) = 2.332, p = 0.131).Appl. Sci. 2019, 9, x 12 of 23
Figure 7. Mean heart rate variability (HRV) for each participant in each phase for E4 (left) and Faros 360 (right).
3.2. HRV SDNN
RMANOVA test for SDNN (Figure 8) showed statistically significant differences among the data from different phases for E4 (F(2, 38) = 10.096, p < 0.001). Bonferroni post-hoc test revealed that the HRV SDNN increased statistically significantly when comparing the baseline and easy driving (p = 0.002) and between baseline and demanding driving (p = 0.010). No statistically significant differences were found between the easy driving and demanding driving phase (p = 1.000).
Figure 8. HRV SDNN for each participant in each phase for E4 (left) and Faros 360 (right).
RMANOVA test results for data captured with Faros 360 also showed that there are some statistically significant differences among the data from different phases (F(2, 42) = 6.967, p = 0.002). Bonferroni post-hoc test showed that the HRV SDNN increased statistically significantly when comparing the baseline and demanding driving (p = 0.010) and between easy driving and demanding driving (p = 0.027). The test did not show any statistically significant differences between the baseline and easy driving phase (p = 0.794).
3.3. HRV Variables for Shorter Timeframes
For further analysis, the data was divided into 10-s timeframes. The analysis showed statistically significant differences among the data from the observed phases for SDANN (Figure 9) but did not reveal any differences in the SDNN Index, for data from the both devices (Table 2).
3.2. HRV SDNN
Appl. Sci. 2019, 9, x 12 of 23
3.2. HRV SDNN
RMANOVA test results for data captured with Faros 360 also showed that there are some statistically significant differences among the data from different phases (F(2, 42) = 6.967, p = 0.002). Bonferroni post-hoc test showed that the HRV SDNN increased statistically significantly when comparing the baseline and demanding driving (p = 0.010) and between easy driving and demanding driving (p = 0.027). The test did not show any statistically significant differences between the baseline and easy driving phase (p = 0.794).
For further analysis, the data was divided into 10-s timeframes. The analysis showed statistically significant differences among the data from the observed phases for SDANN (Figure 9) but did not reveal any differences in the SDNN Index, for data from the both devices (Table 2).
RMANOVA test results for data captured with Faros 360 also showed that there are some statistically significant differences among the data from different phases (F(2, 42) = 6.967, p = 0.002). Bonferroni post-hoc test showed that the HRV SDNN increased statistically significantly when
Appl. Sci. 2019, 9, 5353 13 of 23
comparing the baseline and demanding driving (p = 0.010) and between easy driving and demanding driving (p = 0.027). The test did not show any statistically significant differences between the baseline and easy driving phase (p = 0.794).
For further analysis, the data was divided into 10-s timeframes. The analysis showed statistically significant differences among the data from the observed phases for SDANN (Figure 9) but did not reveal any differences in the SDNN Index, for data from the both devices (Table 2).Appl. Sci. 2019, 9, x 13 of 23
Figure 9. HRV SDANN for 10-s timeframes for each participant in each phase for E4 (left) and Faros 360 (right).
When observing the SDANN data measured with E4, Bonferroni post-hoc test showed that it increased statistically significantly from the baseline compared to the easy driving phase (p < 0.001) and between the baseline and demanding driving phase (p < 0.001); no statistically significant differences were found between the easy driving and demanding driving phase (p = 1.000).
When observing the SDANN data measured with Faros 360, Bonferroni post-hoc test showed that it increased statistically significantly from the baseline compared to the demanding driving phase (p < 0.001) and between the easy driving and the demanding driving phase (p = 0.016). The test showed no statistically significant differences between baseline and easy driving (p = 0.086).
Table 2. Statistical tests’ results for HRV variables for shorter timeframes.
SDANN SDNN Index
F(1.586, 33.303) = 3.923, p = 0.03811
Faros 360 RMANOVA:
F(1.398, 29.364) = 3.405, p = 0.062 1 Bonferroni post-hoc test showed no statistically significant differences between phases, p(1,2) = 0.166, p(2,3) = 0.641, p(1,3) = 0.091.
3.4. HRV Successive Differences
Statistically significant differences in the data from E4 data were found among the phases in both RMSSD (F(2, 38) = 44.822, p < 0.001) and pNN50 (F(1.411, 26.816) = 57.508, p < 0.001) (Figure 10 and Figure 11). Bonferroni post-hoc test showed that the values increased statistically significantly between the baseline and the easy driving phase (p < 0.001) and between the baseline and the demanding driving phase (p < 0.001), for both variables, RMSSD and pNN50. The test showed no statistically significant differences between easy driving and demanding driving (p = 0.964 (RMSSD), p = 0.530 (pNN50)).
In the data, captured with Faros 360, Friedman’s non-parametric tests did not reveal any statistically significant differences in RMSSD (χ2(2) = 3.545, p = 0.170) or pNN50 (χ2(2) = 0.364, p = 0.834).
Figure 9. HRV SDANN for 10-s timeframes for each participant in each phase for E4 (left) and Faros 360 (right).
Table 2. Statistical tests’ results for HRV variables for shorter timeframes.
SDANN SDNN Index
RMANOVA with Greenhouse-Geisser correction: F(1.586, 33.303) = 3.923, p = 0.0381 1
Faros 360 RMANOVA: F(2, 42) = 13.312, p < 0.001
RMANOVA with Greenhouse-Geisser correction: F(1.398, 29.364) = 3.405, p = 0.062
1 Bonferroni post-hoc test showed no statistically significant differences between phases, p(1,2) = 0.166, p(2,3) = 0.641, p(1,3) = 0.091.
When observing the SDANN data measured with E4, Bonferroni post-hoc test showed that it increased statistically significantly from the baseline compared to the easy driving phase (p < 0.001) and between the baseline and demanding driving phase (p < 0.001); no statistically significant differences were found between the easy driving and demanding driving phase (p = 1.000).
When observing the SDANN data measured with Faros 360, Bonferroni post-hoc test showed that it increased statistically significantly from the baseline compared to the demanding driving phase (p < 0.001) and between the easy driving and the demanding driving phase (p = 0.016). The test showed no statistically significant differences between baseline and easy driving (p = 0.086).
3.4. HRV Successive Differences
Statistically significant differences in the data from E4 data were found among the phases in both RMSSD (F(2, 38) = 44.822, p < 0.001) and pNN50 (F(1.411, 26.816) = 57.508, p < 0.001) (Figures 10 and 11). Bonferroni post-hoc test showed that the values increased statistically significantly between the baseline and the easy driving phase (p < 0.001) and between the baseline and the demanding driving phase (p <
0.001), for both variables, RMSSD and pNN50. The test showed no statistically significant differences between easy driving and demanding driving (p = 0.964 (RMSSD), p = 0.530 (pNN50)).
Appl. Sci. 2019, 9, 5353 14 of 23
Appl. Sci. 2019, 9, x 14 of 23
Figure 10. HRV RMSSD for each participant in each phase for E4 (left) and Faros 360 (right).
Figure 11. HRV pNN50 for each participant in each phase for E4 (left) and Faros 360 (right).
3.5. Skin Temperature
During the experiment, the temperature was rising at what appears to be an exponential curve (see Figure 12).
Statistically significant differences among phases (Table 3) were found in mean and standard deviation of the skin temperature readings (Figure 13, Figure 14) from the both devices.
Table 3. Statistical tests’ results for skin temperature.
Skin Temperature Mean Skin Temperature Standard Deviation
E4 RMANOVA: F(2, 42) = 29.313, p < 0.001
Friedman’s nonparametric test: χ2(2) = 9.818, p = 0.007
Faros 360 RMANOVA
RMANOVA with Greenhouse-Geisser correction:
F(1.207, 25.350) = 78.100, p < 0.001
Appl. Sci. 2019, 9, x 14 of 23
During the experiment, the temperature was rising at what appears to be an exponential curve (see Figure 12).
E4 RMANOVA: F(2, 42) = 29.313, p < 0.001
Faros 360 RMANOVA
F(1.207, 25.350) = 78.100, p < 0.001
In the data, captured with Faros 360, Friedman’s non-parametric tests did not reveal any statistically significant differences in RMSSD (χ2(2) = 3.545, p = 0.170) or pNN50 (χ2(2) = 0.364, p = 0.834).
During the experiment, the temperature was rising at what appears to be an exponential curve (see Figure 12).Appl. Sci. 2019, 9, x 15 of 23
Figure 12. Faros’ temperature readings from the beginning of the baseline phase. Data for each participant is drawn with different color. Straight lines interpolate the temperature between the end of one phase and the beginning of the next one.
Figure 13. Mean skin temperature for each participant in each phase for E4 (left) and Faros 360 (right).
Figure 14. Standard deviation of skin temperature for each participant in each phase for E4 (left) and Faros 360 (right).
Bonferroni post-hoc test for mean skin temperature measured with E4 showed statistically significant increase between the baseline and the easy driving phase (p < 0.001) and between the
Appl. Sci. 2019, 9, 5353 15 of 23
E4 RMANOVA: F(2, 42) = 29.313, p < 0.001
Faros 360 RMANOVA with
Appl. Sci. 2019, 9, x 15 of 23
Appl. Sci. 2019, 9, x 15 of 23
Bonferroni post-hoc test for mean skin temperature measured with E4 showed statistically significant increase between the baseline and the easy driving phase (p < 0.001) and between the baseline and the demanding driving phase (p < 0.001). No statistically significant differences were found between the easy and the demanding driving phase (p = 0.770).
Bonferroni post-hoc test for mean skin temperature measure with Faros 360 showed that it increased statistically significantly for each consecutive phase (p < 0.001).
Dunn’s post-hoc test with Bonferroni correction for standard deviation of the skin temperature measured with E4 showed it increased statistically significantly between the baseline and the easy driving phase (p = 0.020) and between the baseline and the demanding driving phase (p = 0.020). The test did not show any statistically significant differences between the easy driving and the demanding driving phase (p = 1.000).
Appl. Sci. 2019, 9, 5353 16 of 23
Similarly, when analyzing the standard deviation of skin temperature measured with Faros 360, Bonferroni post-hoc test showed it increased statistically significantly from the baseline compared to the easy driving phase (p < 0.001), and from the baseline and compared to the demanding driving phase (p < 0.001). The test did not show any statistically significant differences between the easy driving and the demanding driving phase (p = 0.473).
3.6. E4’s GSR
Non-parametric Friedman test revealed that statistically significant differences among the data from the observed phases for mean GSR (χ2(2) = 8.273, p = 0.016) and standard deviation of GSR (χ2(2) = 6.909, p = 0.032) (Figure 15). In both cases, Dunn’s post-hoc test with Bonferroni correction revealed significant increase in the values from the baseline compared to the demanding driving phase (mean GSR: p = 0.013, standard deviation of GSR: p = 0.048), however there were no significant differences in these values when comparing the baseline and the easy driving, and the easy driving and the demanding driving phase (p > 0.1).
Appl. Sci. 2019, 9, x 16 of 23
baseline and the demanding driving phase (p < 0.001). No statistically significant differences were found between the easy and the demanding driving phase (p = 0.770).
Bonferroni post-hoc test for mean skin temperature measure with Faros 360 showed that it increased statistically significantly for each consecutive phase (p < 0.001).
Dunn’s post-hoc test with Bonferroni correction for standard deviation of the skin temperature measured with E4 showed it increased statistically significantly between the baseline and the easy driving phase (p = 0.020) and between the baseline and the demanding driving phase (p = 0.020). The test did not show any statistically significant differences between the easy driving and the demanding driving phase (p = 1.000).
Similarly, when analyzing the standard deviation of skin temperature measured with Faros 360, Bonferroni post-hoc test showed it increased statistically significantly from the baseline compared to the easy driving phase (p < 0.001), and from the baseline and compared to the demanding driving phase (p < 0.001). The test did not show any statistically significant differences between the easy driving and the demanding driving phase (p = 0.473).
3.6. E4’s GSR
Non-parametric Friedman test revealed that statistically significant differences among the data from the observed phases for mean GSR (χ2(2) = 8.273, p = 0.016) and standard deviation of GSR (χ2(2) = 6.909, p = 0.032) (Figure 15). In both cases, Dunn’s post-hoc test with Bonferroni correction revealed significant increase in the values from the baseline compared to the demanding driving phase (mean GSR: p = 0.013, standard deviation of GSR: p = 0.048), however there were no significant differences in these values when comparing the baseline and the easy driving, and the easy driving and the demanding driving phase (p > 0.1).
Figure 15. Mean and standard deviation of GSR for each participant in each phase for E4 measurements.
3.7. Sickness
The majority of drivers (86%) rated their sickness level with a score of equal to or less than 4 (out of 10). Only three participants reported higher sickness levels, with scores of 6, 8 and 9 (Figure 16).
Figure 15. Mean and standard deviation of GSR for each participant in each phase for E4 measurements.
3.7. Sickness
The majority of drivers (86%) rated their sickness level with a score of equal to or less than 4 (out of 10). Only three participants reported higher sickness levels, with scores of 6, 8 and 9 (Figure 16).Appl. Sci. 2019, 9, x 17 of 23
Figure 16. Drivers’ self-reports of sickness on a 10-point scale, where 1 indicates “I am perfectly fine”, and 10 means “I have to throw up”.
4. Discussion and Conclusions
In this paper, we compared two commonly used wearable devices for measuring driver’s physiological signals—Empatica E4 and Bittium Faros 360. We performed a user study with which we wanted to evaluate the use of these devices in a dynamic driving environment. Additionally, we explored if they can be used for differentiating between different levels of driving difficulty.
4.1. Photoplethysmography (PPG) Limitations
E4 uses PPG, which is according to the literature less accurate than ECG, used by Faros [79]. PPG can provide somewhat different (delayed, malformed, etc.) results than ECG since the blood vessels are elastic and the pulse transit times may vary. Furthermore, it is considered that PPG can provide a valid representation of IBI during rest, but not during activity [83]. This suggests additional caution when assessing ECG data with wearable devices that use PPG and not ECG sensors.
When examining the raw data from E4, multiple inter-beat interval (IBI) samples were incomplete for most drivers in both of the driving phases; furthermore, the collected data were completely missing for two drivers. However, the IBI samples recordings were complete for the baseline phase, suggesting the set-up was performed correctly.
There are two potential reasons for incomplete IBI samples when recording with E4. As the missing samples occurred mainly in the dynamic driving environment, we can assume motion artefacts caused by steering negatively influence the E4. Furthermore, using the accelerometer and consequentially its internal algorithm for obtaining IBI from blood volume pulse (BVP), the device might detect intensive motion activity, which may have invalidated some of the IBI samples. Since there is an entire sequence of missing samples recorded, it is impossible to determine the differences between successive samples. Therefore, analysis of variables based on successive differences of such samples (i.e., RMSSD and pNN50) cannot be used for making reliable conclusions.
4.2. HRV Analysis
The results of HRV analysis based on mean and median tests did not show any statistically significant differences among the tested phases. However, the SDNN and SDANN signals showed that it is possible to differentiate between driving and non-driving situations, and to differentiate between different levels of driving demand.
When comparing the standard deviation (SDNN) of HRV, the analysis showed that E4 can differentiate only between non-driving and driving, but it cannot differentiate between different levels of driving demand, where drivers had to move their hands in order to steer the wheel. On the other hand, the analysis showed that with the signals recorded with Faros 360 it was possible to differentiate between the two different levels of driving difficulty. We speculate that the different
0
2
4
6
8
10
12
14
1 2 3 4 5 6 7 8 9 10
N um
be r
of d
ri ve
Reported sickness level
Figure 16. Drivers’ self-reports of sickness on a 10-point scale, where 1 indicates “I am perfectly fine”, and 10 means “I have to throw up”.
Appl. Sci. 2019, 9, 5353 17 of 23
4. Discussion and Conclusions
In this paper, we compared two commonly used wearable devices for measuring driver’s physiological signals—Empatica E4 and Bittium Faros 360. We performed a user study with which we wanted to evaluate the use of these devices in a dynamic driving environment. Additionally, we explored if they can be used for differentiating between different levels of driving difficulty.
4.1. Photoplethysmography (PPG) Limitations
E4 uses PPG, which is according to the literature less accurate than ECG, used by Faros [79]. PPG can provide somewhat different (delayed, malformed, etc.) results than ECG since the blood vessels are elastic and the pulse transit times may vary. Furthermore, it is considered that PPG can provide a valid representation of IBI during rest, but not during activity [83]. This suggests additional caution when assessing ECG data with wearable devices that use PPG and not ECG sensors.
When examining the raw data from E4, multiple inter-beat interval (IBI) samples were incomplete for most drivers in both of the driving phases; furthermore, the collected data were completely missing for two drivers. However, the IBI samples recordings were complete for the baseline phase, suggesting the set-up was performed correctly.
There are two potential reasons for incomplete IBI samples when recording with E4. As the missing samples occurred mainly in the dynamic driving environment, we can assume motion artefacts caused by steering negatively influence the E4. Furthermore, using the accelerometer and consequentially its internal algorithm for obtaining IBI from blood volume pulse (BVP), the device might detect intensive motion activity, which may have invalidated some of the IBI samples. Since there is an entire sequence of missing samples recorded, it is impossible to determine the differences between successive samples. Therefore, analysis of variables based on successive differences of such samples (i.e., RMSSD and pNN50) cannot be used for making reliable conclusions.
4.2. HRV Analysis
The results of HRV analysis based on mean and median tests did not show any statistically significant differences among the tested phases. However, the SDNN and SDANN signals showed that it is possible to differentiate between driving and non-driving situations, and to differentiate between different levels of driving demand.
When comparing the standard deviation (SDNN) of HRV, the analysis showed that E4 can differentiate only between non-driving and driving, but it cannot differentiate between different levels of driving demand, where drivers had to move their hands in order to steer the wheel. On the other hand, the analysis showed that with the signals recorded with Faros 360 it was possible to differentiate between the two different levels of driving difficulty. We speculate that the different results could be a result of the PPG limitations discussed earlier. Another possibility is that the driving demand did not differ significantly, and the sensors could not detect any differences because of the small differences in the driving demand.
The analysis of the HRV variables for shorter time frames (SDANN, SDNN index) showed statistically significant differences only in the values of SDANN, which suggests that changes in activities with periods longer than 10 s can be detected. However, there were no statistically significant differences in the SDNN index results, suggesting that neither E4 nor Faros can detect changes in different driving difficulties, based on recordings shorter than 10 s.
The analysis revealed statistically significant differences in RMSSD and pNN50 between the baseline and driving phases only in the data acquired with E4 but not in the data acquired with Faros although the HRV signal was measured with both devices simultaneously. We believe that the reason lies in E4’s limitations. As described in the third paragraph of the “PPG limitations” subsection, when there are missing samples in a sequence, successive differences cannot be determined. Therefore, we
Appl. Sci. 2019, 9, 5353 18 of 23
could perform analysis only on the signals measured with Faros, which however did not provide statistically significant results.
Based on these results, we can conclude that from the observed HRV signals, only SDNN or SDANN can be used for differentiating between the two observed levels of driving demand. However, we cannot rule out the possibility that the differences could be captured also in the other HRV variables, which could be investigated, for example, by using longer trials for each condition. Furthermore, it is better to use Faros than E4 as motion artefacts influence the measurement process and collected data.
4.3. Skin Temperature Analysis
Direct comparison of skin temperature data between the two devices is not possible, since they measure temperatures of different body parts. As shown in Figure 12 the temperature measured with Faros 360 was constantly rising. This could be due to the fact that the experiment lasted less than 20 min and it is possible that Faros was still warming up. Consequently, with slower warming up, the standard deviation significantly decreased in later phases. The skin temperature sensor in Faros 360 would provide more reliable data in studies that involve longer measurement periods (several hours) or, alternatively, participants would have to wear the measuring devices at least a few hours before the measurement process. This phenomenon was not evident with the E4’s skin temperature sensor.
4.4. Galvanic Skin Response (GSR) Analysis
The galvanic skin response (GSR) measurements with E4 revealed statistically significant increased conductance between demanding driving and baseline phases. However, there was a high standard deviation and standard error in the collected GSR signals during the driving phases, which can imply that the E4 may not have had skin contact during the whole recordings, probably due to the hand movements while steering.
4.5. Driver Sickness
Eighty-six percent of participants rated their feeling of sickness with a score of lower than 4 out of 10, and 91% of participants evaluated it with equal or lower than 6, suggesting that the results were not influenced by simulation/motion sickness. The measurements of the rest of the 9% of participants did not involve any outliers in the observed dependent variables.
4.6. Conclusions
The results from this study suggest that devices with PPG sensors such as E4 may be less reliable for assessing drivers’ signals compared to an ECG sensor-based device such as Faros 360. The main disadvantage is the sensitivity to motion artefacts, which can influence the signal recording quality. This is also in line with the placement of both devices, where Faros 360 is placed on the torso, which is relatively still during driving, whereas the E4 device is placed on the wrist, which is involved in movement due to steering the wheel. Therefore, in this particular case, we think Faros 360 could provide more reliable results for assessing drivers’ physiological signals.
Author Contributions: Conceptualization, T.G. and J.S.; methodology, K.S., G.J. and T.G.; software, T.G.; validation, K.S., J.S. and G.J.; formal analysis, T.G. and G.J.; investigation, T.G. and K.S.; resources, J.S.; data curation, T.G.; writing—original draft preparation, T.G.; writing—review and editing, K.S. and G.J.; visualization, T.G.; supervision, G.J. and J.S.; project administration, T.G. and K.S.; funding acquisition, J.S.
Funding: This work was partly supported by the Slovenian Research Agency within the research program ICT4QoL—Information and Communications Technologies for Quality of Life, grant number P2-0246, and the research project Neurophysiological and Cognitive Profiling of Driving Skills, grant number L2-8178.
Conflicts of Interest: The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.
Appl. Sci. 2019, 9, 5353 19 of 23
References
1. Petridou, E.; Moustaki, M. Human factors in the causation of road traffic crashes. Eur. J. Epidemiol. 2000, 16, 819–826. [CrossRef] [PubMed]
2. Vitabile, S.; Paola, A.D.; Sorbello, F. Bright Pupil Detection in an Embedded, Real-Time Drowsiness Monitoring System. In Proceedings of the 2010 24th IEEE International Conference on Advanced Information Networking and Applications, Perth, Australia, 20–23 April 2010; pp. 661–668.
3. Bergasa, L.M.; Nuevo, J.; Sotelo, M.A.; Barea, R.; Lopez, M.E. Real-time system for monitoring driver vigilance. IEEE Trans. Intell. Trans. Syst. 2006, 7, 63–77. [CrossRef]
4. Lin, C.; Chen, Y.; Huang, T.; Chiu, T.; Ko, L.; Liang, S.; Hsieh, H.; Hsu, S.; Duann, J. Development of Wireless Brain Computer Interface With Embedded Multitask Scheduling and its Application on Real-Time Driver’s Drowsiness Detection and Warning. IEEE Trans. Biomed. Eng. 2008, 55, 1582–1591. [CrossRef] [PubMed]
5. King, L.M.; Nguyen, H.T.; Lal, S.K.L. Early Driver Fatigue Detection from Electroencephalography Signals using Artificial Neural Networks. In Proceedings of the 2006 International Conference of the IEEE Engineering in Medicine and Biology Society, New York, NY, USA, 30 August–3 September 2006; pp. 2187–2190.
6. Li, W.; He, Q.; Fan, X.; Fei, Z. Evaluation of driver fatigue on two channels of EEG data. Neurosci. Lett. 2012, 506, 235–239. [CrossRef]
7. Eoh, H.J.; Chung, M.K.; Kim, S.-H. Electroencephalographic study of drowsiness in simulated driving with sleep deprivation. Int. J. Ind. Ergon. 2005, 35, 307–320. [CrossRef]
8. Khushaba, R.N.; Kodagoda, S.; Lal, S.; Dissanayake, G. Driver Drowsiness Classification Using Fuzzy Wavelet-Packet-Based Feature-Extraction Algorithm. IEEE Trans. Biomed. Eng. 2011, 58, 121–131. [CrossRef]
9. Giusti, A.; Zocchi, C.; Rovetta, A. A Noninvasive System for Evaluating Driver Vigilance Level Examining Both Physiological and Mechanical Data. IEEE Trans. Intell. Trans. Syst. 2009, 10, 127–134. [CrossRef]
10. Lee, B.; Chung, W. Driver Alertness Monitoring Using Fusion of Facial Features and Bio-Signals. IEEE Sens. J. 2012, 12, 2416–2422. [CrossRef]
11. Bundele, M.M.; Banerjee, R. Detection of Fatigue of Vehicular Driver Using Skin Conductance and Oximetry Pulse: A Neural Network Approach. In Proceedings of the 11th International Conference on Information Integration and Web-based Applications & Services, Kuala Lumpur, Malaysia, 14–16 December 2009; ACM: New York, NY, USA, 2009; pp. 739–744.
12. Shin, H.; Jung, S.; Kim, J.; Chung, W. Real time car driver’s condition monitoring system. In Proceedings of the 2010 IEEE SENSORS, Kona, HI, USA, 1–4 November 2010; pp. 951–954.
13. Jung, S.-J.; Shin, H.-S.; Chung, W.-Y. Driver fatigue and drowsiness monitoring system with embedded electrocardiogram sensor on steering wheel. IET Intell. Trans. Syst. 2014, 8, 43–50. [CrossRef]
14. Matsuda, T.; Makikawa, M. ECG monitoring of a car driver using capacitively-coupled electrodes. In Proceedings of the 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Vancouver, BC, Canada, 20–25 August 2008; pp. 1315–1318.
15. Thum Chia Chieh; Mustafa, M.M.; Hussain, A.; Hendi, S.F.; Majlis, B.Y. Development of vehicle driver drowsiness detection system using electrooculogram (EOG). In Proceedings of the 2005 1st International Conference on Computers, Communications, Signal Processing with Special Track on Biomedical Engineering, Kuala Lumpur, Malaysia, 14–16 November 2005; pp. 165–168.
16. Ma, Z.; Li, B.C.; Yan, Z. Wearable driver drowsiness detection using electrooculography signal. In Proceedings of the 2016 IEEE Topical Conference on Wireless Sensors and Sensor Networks (WiSNet), Austin, TX, USA, 24–27 January 2016; pp. 41–43.
17. Shih, C.; Cheng, S. A driver’s physiological monitoring system based on photodiode sensor with reflective optics element. In Proceedings of the 2016 IEEE International Conference on Vehicular Electronics and Safety (ICVES), Beijing, China, 10–12 July 2016; pp. 1–4.
18. Healey, J.; Seger, J.; Picard, R. Quantifying driver stress: Developing a system for collecting and processing bio-metric signals in natural situations. Biomed. Sci. Instrum. 1999, 35, 193–198.
19. Healey, J.; Picard, R. SmartCar: Detecting driver stress. In Proceedings of the 15th International Conference on Pattern Recognition. ICPR-2000, Barcelona, Spain, 3–7 September 2000; Volume 4, pp. 218–221.
20. Fairclough, S.H.; Spiridon, E. Cardiovascular and electrocortical markers of anger and motivation during a simulated driving task. Int. J. Psychophysiol. 2012, 84, 188–193. [CrossRef] [PubMed]
21. Walter, M.; Eilebrecht, B.; Wartzek, T.; Leonhardt, S. The Smart Car Seat: Personalized Monitoring of Vital Signs in Automotive Applications. Personal Ubiquitous Comput. 2011, 15, 707–715. [CrossRef]
22. Yang, C.; Wu, C.; Chou, C.; Yang, T. Vehicle driver’s ECG and sitting posture monitoring system. In Proceedings of the 2009 9th International Conference on Information Technology and Applications in Biomedicine, Larnaca, Cyprus, 4–7 November 2009; pp. 1–4.
23. Balasubramanian, V.; Adalarasu, K. EMG-based analysis of change in muscle activity during simulated driving. J. Bodyw. Mov. Ther. 2007, 11, 151–158. [CrossRef]
24. Reimer, B.; Mehler, B. The Effects of a Production Level “Voice-Command” Interface on Driver Behavior: Summary Findings on Reported Workload, Physiology, Visual Attention, and Driving Performance; MIT AgeLab White Paper No. 2013-18A: Cambridge, MA, USA, 2013.
25. Son, J.; Oh, H.; Park, M. Identification of driver cognitive workload using support vector machines with driving performance, physiology and eye movement in a driving simulator. Int. J. Precis. Eng. Manuf. 2013, 14, 1321–1327. [CrossRef]
26. Solovey, E.T.; Zec, M.; Garcia Perez, E.A.; Reimer, B.; Mehler, B. Classifying Driver Workload Using Physiological and Driving Performance Data: Two Field Studies. In Proceedings of the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Toronto, ON, Canada, 26 April–1 May 2014; ACM: New York, NY, USA, 2014; pp. 4057–4066.
27. Lee, H.B.; Kim, J.S.; Kim, Y.S.; Baek, H.J.; Ryu, M.S.; Park, K.S. The relationship between HRV parameters and stressful driving situation in the real road. In Proceedings of the 2007 6th International Special Topic Conference on Information Technology Applications in Biomedicine, Tokyo, Japan, 8–11 November 2007; pp. 198–200.
28. Liu, Y.; Ji, X.; Hayama, R.; Mizuno, T. A novel estimating method for steering efficiency of the driver with electromyography signals. Chin. J. Mech. Eng. 2014, 27, 460–467. [CrossRef]
29. Kim, I.-H.; Kim, J.-W.; Haufe, S.; Lee, S.-W. Detection of braking intention in diverse situations during simulated driving based on EEG feature combination. J. Neural Eng. 2014, 12, 016001. [CrossRef]
30. ZEIER, H. Concurrent physiological activity of driver and passenger when driving with and without automatic transmission in heavy city traffic. Ergonomics 1979, 22, 799–810. [CrossRef]
31. Brookhuis, K.A.; de Waard, D. Monitoring drivers’ mental workload in driving simulators using physiological measures. Accid. Anal. Prev. 2010, 42, 898–903. [CrossRef]
32. Brookhuis, K.A.; Waard, D. de On the assessment of (mental) workload and other subjective qualifications. Ergonomics 2002, 45, 1026–1030. [CrossRef]
33. Mulder, L.B.J.; de Waard, D.; Brookhuis, A.K. Estimating Mental Effort Using Heart Rate and Heart Rate Variability. Available online: https://www.taylorfrancis.com/ (accessed on 14 November 2019).
34. Wang, J.; Gong, Y. Recognition of multiple drivers’ emotional state. In Proceedings of the 2008 19th International Conference on Pattern Recognition, Tampa, FL, USA, 8–11 December 2008; pp. 1–4.
35. Calder&oa

assessing drivers’ physiological responses using consumer

Documents