Georgia Institute of Technology

Driver Drowsiness Detection System

Measuring heart rate variability on an embedded system

Yang Liu Adhithya Rajasekaran

Abhinav Uppal Yichen Wang

ECE 4781: Biomedical Sensing Systems Design Project Report

December 3, 2015

(all members made an excellent contribution to the project)

A. Executive Summary

Sleep deprived driving is now a serious problem in the United States and worldwide,

resulting in a large number of deaths and injuries.1 Most available drowsiness monitoring systems for drivers rely on either vehicle-based or behavioral based sensing. However, these systems are unable to alarm drivers until after they have fallen asleep, making these devices unreliable in preventing accidents. To that end, we have developed a physiologically-based system that captures the electrocardiogram (ECG) of a user and assesses his / her drowsiness level by analyzing heart rate variability (HRV.)

A user’s ECG was transduced from the body using gel-based electrodes. The signal was then filtered using an integrated analog-front end, sampled with an ADC at 333 Hz, and serially transmitted to computer via a USB isolator. The digitized ECG signal was processed using Physionet’s WaveForm DataBase (WFDB) library and HRV toolkit to extract the R-R time intervals, and hence, the heart rate variability (HRV.) Finally, the power spectrum density of HRV was plotted to view its spectrum in the frequency domain. Our results suggest a good correlation between the low frequency band (0.04-0.15 Hz) over high frequency band (0.15-0.40 Hz) power density (LF/HF) ratio and drowsiness. A lower ratio indicated that the user was in a drowsier state. Therefore, using this system, we could physiologically monitor a driver’s drowsiness level, and issue warnings in time to reduce accidents.

In the future, our focus would be on the development of an ear-based ECG sensing system to improve wearability of the device. Also, an ear-based photoplethysmogram (PPG) sensing system could also be developed to achieve the same goal without the use of electrodes.

B. Introduction and Significance

Each year, over 37,000 people die in road crashes in the United States, and nearly 1.3

million people die in road crashes worldwide. While we hear a lot about the shocking statistics of car accidents, we may not know some astonishing facts about the causes underlying these crashes. One major cause of car accidents is sleep-deprived driving (i.e. tired driving, drowsy driving, or fatigued driving). According to a poll by the National Sleep Foundation, 60% of adult drivers admitted that they felt drowsy while driving in 2014, and over one-third drivers said that they had actually fallen asleep while driving. According to the National Highway Traffic Safety Administration, sleep-deprived driving is responsible for 1,550 deaths, 71,000 injuries, and monetary costs of $12.5 billion annually in the US.1

In recent years, many in-vehicle technologies have been available to address real time monitoring of a driver’s state. However, most of these monitoring devices and systems assess drivers’ drowsiness by measuring behavioral aspects such as eye closure, pupil occlusion, etc. by implementing state-of-the-art computer vision technologies using remotely mounted cameras.2,3,4 Another approach for assessing drowsiness uses vehicle based measurements. For example, the number of micro-corrections on the steering wheel is measured for detecting drowsiness level.5 Even though both vehicle-based and behavioral-based measures are accurate, they become apparent only after the driver starts to sleep, which is often too late to prevent an accident.1

A few emerging technologies have attempted to measure drowsiness using physiological measurements such as brain wave activity and heart rate.6,7 The reliability of physiological measurements for drowsiness detection is very high, as physiological signals start changing in earlier stages of drowsiness. Nevertheless, one problem with current physiological measurements is that they are intrusive, as they require electrode placements that could interfere with driving comfort.1

To our best knowledge, only a few of the currently employed in-vehicle devices can accurately assess and quantify drivers’ consciousness, including the state of alertness, drowsiness, sensory awareness, etc. To this end, we worked on designing a wearable device that can measure physical parameters correlated with stages of drowsiness, and could potentially send automatic alerts to the driver based on its assessment of his/her drowsiness level.

According to past studies, HRV is related with drowsiness.1,6 Deriving measures of HRV requires accurate determination of successive heart beats. For this purpose, ECG is a widely used non-invasive method for extracting R-R peaks for HRV measurement. Our design thus employed ECG for HRV measurement. We also began development on an analogous system employing PPG. The ratio of LF to HF in the ECG decreases as the driver has an increased level of drowsiness.6,8

C. Project Narrative

This project was divided into two parallel branches, each approaching the problem of

measuring heart rate (and hence HRV) by sensing and processing either ECG or PPG. Fig. 1. shows the hardware and software components associated with both of these branches, and the rest of this section elaborates on the analog front-ends (AFEs), sampling approaches, and analysis tools used for obtaining a user’s HRV at different levels of drowsiness.

Figure 1. Various stages of the project, showing the hardware and software tools utilized. Developing a PPG Sensing Platform using AFE4400

Given the size constraints associated with performing a non-obtrusive measurement of heart rate (variability) in a driving setting, we decided to base our sensing system on a compact, integrated AFE. To our knowledge, the Texas Instruments (TI) AFE440010 is the only integrated AFE available in the market dedicated to PPG capture, and with the associated evaluation module being relatively big in size, and costing $149, we decided to design a smaller evaluation platform based on reference designs provided by TI10. AFE4400: Features and Design Considerations

The AFE4400 is a highly configurable AFE, with an integrated H-bridge LED driver (on the Tx) and a transimpedance amplifier (on the Rx) with digitally adjustable parameters. The chip supports an SPI interface for programming registers and transferring data, digital control lines for issuing interrupts to a microcontroller, and a range of supplies for supporting the operation of different LEDs and detectors.

To simplify our prototyping, we decided to design a board that provided pads for easily interfacing the AFE with a microcontroller, LED, and detector by soldering connecting wires. We also chose to breakout the supply pins for performing initial testing of the chip and LEDs using bench top-supplies, which could subsequently be replaced with a power regulation daughter card once the prototype’s current draw and voltage requirements have been characterized. For the LED(s) and detector, we chose to use Osram’s SFH-705011 in our system, as it integrates three LEDs (red, green, and IR), and a detector in one compact package, making it easier to experiment with different LED placements and wavelengths for sensing PPG.

Schematic Capture and Layout

An open source electronic design automation software, KiCAD12, was used to make the platform’s schematic, generate the required footprints, and layout a 2-layer PCB. The schematic and layout were then posted on TI’s e2e medical forum for review13. Feedback from a TI applications engineer, Praveen Aroul, was used to revise the PCB design (including removal of extraneous components.) We then requested the School of ECE to mill the revised PCB, and the milling was completed on December 1, 2015. Team Morpheus would like to thank Mr. James Steinberg for his help. Unfortunately, the PCB’s vias did not plate, hence the board development is on hold for now.

Figure 2. Top layer of the 2-layer AFE4400 PCB. The SFH-7050 was mounted on a separate 2-layer PCB. Designing an ECG system for extracting HRV Being aware of the delays inherent to PCB layout and milling, we worked on an ECG-based sensing and HRV analysis system in parallel to designing the AFE4400 board. A high-level block diagram of this system is shown in Fig. 3.

Figure 3. ECG-based HRV analysis system, for capturing ECG and processing it on a computer. AD8232: Single Lead ECG AFE

With the same size concern as with the PPG solution, we chose to go with an integrated AFE for conditioning an ECG signal acquired by gel electrodes placed across a user’s chest, with a reference electrode on the right ankle. The Analog Devices AD8232 chip was used as the ECF AFE because of the availability of a low-cost development board from SparkFun14.

SparkFun’s AD8232 board implements the “cardiac monitoring configuration” provided in AD8232’s datasheet15. This configuration has a band-pass frequency response (Fig. 4.) suitable for monitoring the shape of the ECG waveform, and it assumes that the signal is relatively free of motion artifact (with the subject remaining stationary.) As this may not always apply to our target users (drivers), the chip can be reconfigured to have a narrow pass-band. For this project, however, we chose to restrain our motion during subsequent measurements, so as to focus development efforts towards digitizing and analyzing the ECG.

Figure 4. Cardiac monitoring configuration of the AD8232 (left), highlighting its cascaded high pass and low pass filtering stages. This configuration results in a band-pass response (right.) A sampling of the output of AD8232 is shown in Fig. 5. The subject remained still while these measurements were taken, and the analog output of the AFE was sampled using an MSP432 (see next section.)

Figure 5. Amplified ECG outputted by AD8232, digitized with MSP432 and plotted in MATLAB. Sampling AD8232’s output with MSP432 LaunchPad

The Texas Instruments MSP432 LaunchPad is a low-cost microcontroller platform with a 14-bit 1MSPS ADC16. It also supports a back-channel UART for transmitting signals to a computer over USB. With access to a free LaunchPad from one of Texas Instruments’

development workshops held at Georgia Tech this October, we decided to implement a sampling routine for sending sampled data to a computer over USB.

Initially, the Arduino-based Energia IDE was used to program the MSP432 to sample and transmit data in a loop, with fixed delays inside the loop being used to enforce a sampling time of 4 ms. The sampling time was observed to be variable over a few milliseconds (the in-built millis()function was used to output time-stamps along with the sampled data.) To get a consistent sampling rate, attempts were made to switch to TI’s CodeComposerStudio IDE for writing an interrupt service routine for triggering ADC captures from a timer. While these attempts were being debugged, we came across Energia’s “Multitasking” feature employing TI’s real-time operating systems (RTOS), which was used to give a consistent sampling period of 3 ms (Fig. 6.)

We also found generating time stamps on the MSP432 board itself to be more reliable as compared to using the computer receiving data over USB for timing the captures. Processing3 was run on a Windows machine to capture the serial data outputted by the MSP432, and variability in sampling time (as stamped by Processing3) can be seen in Fig. 6. Hence, the MSP432 was used to generate reliable time-stamps for all subsequent data captures, and Processing3 was used to write captured ECG data to a file.

Figure 6. A comparison of intervals between consecutive timestamps generated by the MSP432 (RTOS) with each serial (UART) write (top), and the timestamps generated by Processing3 with each line write to the data collection file. RTOS is seen to be more reliable (only 4 out of 204828 intervals are off) for calculating the ECG’s sampling frequency than Processing3 (running on a non-real time, Windows OS.) QRS detection and HRV analysis with Physionet’s software tools

With access to the ECG data thus generated by the MSP432 and Processing3, Physionet’s software tools (WFDB17 and HRV-toolkit18) were setup to perform analysis on different team members’ computers. As a result, different parts of the ECG analysis were performed on different platforms (Fig. 7.): Windows (running Cygwin), Ubuntu, and Debian (kernel 3.14), Debian (kernel 3.8). Regular laptops were used to serve as the Windows and Ubuntu machines, and two BeagleBone Blacks (BBBlack) ran different linux kernels for

Debian. For ease of use, we have begun porting all steps of Fig. 7. to the Beaglebone Blacks, however, the ports have not been completely verified this far.

Figure 7. Physionet’s WFDB and hrv-toolkit commands used for analyzing sampled ECG signals. The tasks were distributed among different team members’ computers, but these can also be executed on a single system (BBBlack). Various functions from the Physionet tool-kits were used to analyze the ECG data collected using the MSP432 and Processing3. The sampled data was first converted to a Physionet compatible record using the wrsamp function. Then, the gqrs function was used to annotate these record files with QRS peak detections. The annotations were then verified by using wave, and only a few (less than 10) errors were observed over three datasets spanning approx. 10 minutes each. These errors were manually corrected using the wave tool. Next, the QRS annotation files were used to obtain RR intervals with ann2rr, which were then plotted using the hrfft and hrlomb functions for generating FFTs and lomb periodograms on the Windows machine running Cygwin. The lomb periodograms for three different datasets (corresponding to different levels of the user’s self-reported drowsiness) are provided in Fig. 9. On the other hand, the get_hrv function from Physionet’s HRV toolkit was used on the same Physionet record files (transferred from Windows to BBBlack over sftp) to obtain LF/HF ratios summarized in Fig. 10. Fig. 11. shows the get_hrv functions output as printed to the BBBlack’s console. Porting all processing to one BBBlack

As BBBlack is a complete Linux computer running the Debian operating system supporting analog inputs, we decided to implement everything from sampling the ECG to analyzing HRV on one BBBlack (Fig. 8.) The Physionet tools have been installed and tested independently. The entire chain runs through one node.js script, but the script is yet to be tested with the the AD8232 board connected to the BBBlack.

Figure 8. Node.js implementation for performing ECG sampling and analysis on one embedded platform.

Figure 9. Cygwin generated Lomb periodograms (least-squares spectra) associated with three 10-minute ECG datasets collected with different levels of drowsiness. Later data collection times were assumed to be correlated with more drowsiness. The LF/HF ratio is seen to increase with the subject’s drowsiness (top to bottom.)

Figure 10. Comparison of LF/HF ratios obtained for three ten-minute ECG datatsets collected on the same user reporting different levels of drowsiness over different days.

Figure 11. Screenshot of the HRV analysis results outputted to the BBBlack’s console.

D. Conclusions and Future Directions

Based on preliminary data, our study suggested that heart rate variability has a good

correlation with people’s level of drowsiness. In the frequency domain, the power density of the HRV tends to concentrate in the LF band (0.04-0.15 Hz) for people in energetic state. Conversely, as people getting drowsier, the power density starts to shift toward the HF band (0.15-0.4 Hz). The LF/HF ratio will thus decrease as people becoming more and more sleepy. Therefore, by accurately measuring the LF/HF ratio for drivers in real-time, alarming signals could be sent to the drivers on time to alert them and prevent any car accidents. However, more data still need to be collected to show a statistical significance and reliably predict a cutoff ratio to send out the alarm for users.

In the future, to further improve driver’s convenience and comfortability using our device, a ear-based monitoring system could be developed. This will both decrease the size and enhance the wearability of our device. To do so, an electrode array might have to be developed to accurately capture ECG signal from the ear.

On the other hand, a potential alternative is to determine heart rate through PPG. PPG involves determining cardiac cycles by monitoring changes in blood volume in the peripheral microvasculature. PPG can be used to quantify changes in oxygenation levels of blood, which has been shown to be correlated with drowsiness.9

E. References

[1] A. Sahayadhas, K. Sundaraj, and M. Murugappan, “Detecting driver drowsiness based

on sensors: a review,” Sensors (Basel, Switzerland), vol. 12, pp. 16937-16953, 2012. [2] M. Lew, N. Sebe, T. Huang, E. Bakker, E. Vural, M. Cetin, A. Ercil, G. Littlewort,

M. Bartlett, J. Movellan, “Drowsy driver detection through facial movement analysis,” Human-Computer Interaction, Vol. 4796, Springer; Berlin, Germany: 2007; pp. 6–18.

[3] F. Xiao, C.Y. Bao, F.S. Yan, “Yawning detection based on gabor wavelets and LDA.” J. Beijing Univ. Technol. 2009;35:409–413

[4] Z. Zhang, J. Zhang, “A new real-time eye tracking based on nonlinear unscented Kalman filter for monitoring driver fatigue,” J. Contr. Theor. Appl. 2010;8:181–188.

[5] F. Ruijia, Z. Guangyuan, C. Bo, “An on-Board System for Detecting Driver Drowsiness Based on Multi-Sensor Data Fusion Using Dempster-Shafer Theory,” Proceedings of the International Conference on Networking, Sensing and Control; Okayama, Japan. 26–29 March 2009; pp. 897–902.

[6] M. Patel, S.K.L. Lal, D. Kavanagh, P. Rossiter, “Applying neural network analysis on heart rate variability data to assess driver fatigue,” Exp. Syst. Appl. 2011;38:7235–7242

[7] C.L. Fu, W.K. Li, H.C. Chun, P.S. Tung, T.L. Chin, “Generalized EEG-based drowsiness prediction system by using a self-organizing neural fuzzy system,” IEEE Trans. Circ. Syst. 2012;59:2044–2055.

[8] E.C. Chua, W.Q. Tan, S.C. Yeo, et al, “Heart Rate Variability Can Be Used to Estimate Sleepiness-related Decrements in Psychomotor Vigilance during Total Sleep Deprivation,” Sleep. 2012;35(3):325-334. doi:10.5665/sleep.1688.

[9] E.J. Sung, B.C. Min, S.C. Kim, and C.J. Kim, “Effects of oxygen concentrations on driver fatigue during simulated driving,” Applied Ergonomics, vol. 36, pp. 25-31, 2005.

[10] Texas Instruments, “Integrated Analog Front End for Heart Rate Monitors and Low Cost Pulse Oximeters”, AFE4400 datasheet. [Online]. Available: http://www.ti.com/product/AFE4400/datasheet. [Accessed Dec.03 2015]

[11] “BioMon Sensor SFH 7050 | Sensor Family | OSRAM Opto Semiconductors.” [Online]. Available: http://www.osram-os.com/osram_os/en/products/product-promotions/infrared-product s/sensor-family/biomon-sensor-sfh-7050/index.jsp. [Accessed: 03-Dec-2015].

[12] “KiCad EDA.” [Online]. Available: http://kicad-pcb.org/. [Accessed: 03-Dec-2015]. [13] “Requesting AFE4400 Board Layout Review - Medical Forum - Medical - TI E2E

Community.” [Online]. Available: http://e2e.ti.com/support/applications/high_reliability/f/30/p/470097/1688790#168879 0. [Accessed: 03-Dec-2015].

[14] Sparkfun, “Single lead, heart rate monitor front end”, AD8232 datasheet. [Online]. Available: https://cdn.sparkfun.com/datasheets/Sensors/Biometric/AD8232.pdf. [Accessed Dec.03 2015]

[15] “AD8232 | datasheet and product info Single-Lead Heart Rate Monitor Analog Front End | Analog Devices.” [Online]. Available: http://www.analog.com/en/products/application-specific/medical/ecg/ad8232.html. [Accessed: 03-Dec-2015].

[16] Texas Instruments,“Overview for MSP432”. [Online]. Available: http://www.ti.com/lsds/ti/microcontrollers_16-bit_32-bit/msp/low_power_performanc

e/msp432p4x/overview.page [Accessed Dec.03 2015] [17] Physionet, “The WFDB software package”. [Online]. Available:

https://www.physionet.org/physiotools/wfdb.shtml [Accessed Dec.03 2015] [18] Physionet, “Heart rate variability analysis with the HRV toolkit”. [Online]. Available:

https://physionet.org/tutorials/hrv-toolkit/ [Accessed Dec.03 2015]