measuring eye-movement patterns on a smartphone for medical...

Hsin-Yu Lai, Gladynel Saavedra-Peña

Measuring Eye-Movement Patterns on a Smartphone for Medical Applications

Prof. Charlie Sodini, Prof. Thomas Heldt, Prof. Vivienne Sze

• Motivation & background

• Measurement system

• Eye movement data

• Conclusions

Outline1




• Conclusions

Outline2

Neurodegenerative Disorders Affect Millions3

• Neurodegenerative disorders affect millions of people worldwide

• 5 million Americans live with Alzheimer’s Disease [NIH]

• 500,000 Americans live with Parkinson’s Disease [NIH]

• No cure for neurodegenerative diseases

• No accurate way to measure disease stage

• Need for accessible tools that can accurately measure disease progression

• Objective measurements of disease progression lead to effective therapies[Newsweek, 2016]

Current Tests to Track Disease Progression

• Repeat medical assessments are sparse and suffer from high retest variability

• Tests are mostly qualitative, time consuming and require a trained specialist.

Mini-Mental State Examination (MMSE)

Q1. What is the year? Season? Date?

Q2. Where are you now? State? Floor?

Q3. Could you count backward from 100 by sevens? (93, 86, …)

Clock-drawing test

[Agrell et al. Age and Ageing, 1998]

Current cognitive tests are not ideal tools to measure disease progression.

4

Based on a physician’s assessment of patient performance.

We want to track neurodegenerative disease

progression by measuring quantitative

physiological data with a consumer-grade device.

Goal of our Research5

Why consumer-grade devices?• Non-obtrusive, low-cost, higher availability• Does not require trained specialist, less sparsity in data

Alzheimer’s Disease or Parkinson’s Disease

Smartphone or a tablet

Visual Reaction Time in Patients 6

Visual reaction time is larger and more variable in patient populations.

[Garbutt et al. Brain, 2008]

Patients with frontotemporal

degeneration & dementia

Mean v

isual re

action t

ime (

ms)

[Heuer et al. Neurology, 2013]

Mean v

isual re

action t

ime (

ms)

NE: Normal Elderly

MCI: Mild Cognitive Impairment

AD: Alzheimer’s Disease

Reaction time is the time delay between the presentation of a stimulus and when the eye starts to move towards said stimulus.

Visual Tasks to Measure Reaction Time7

Central Fixation PointStimulus

x

TimeEye P

osit

ion

Reaction time

No

rma

lize

d c

ou

nt

Reaction time (ms)

Measurements in Clinical Environments8

Substantial head support

SR EYELINK 1000 PLUS

IR illumination

[Reulen et al., Med. & Biol. Eng. & Comp, 1988.]

Clinical measurements are done in constrained environments that rely on specialized, costly equipment.

High-speed camera

PHANTOM V25-11

Measurements in Clinical Environments

Substantial head supportHigh-speed camera IR illumination

9

… to this!

GOAL: Go from this…




• Conclusions

Outline10

The Pipeline Consists of an Eye Tracking Algorithm and Saccade Onset Detection

11

Goal: Find a robust eye-tracking algorithm that is able to show clear saccade onsets.

Approach: Improve current state-of-the-art gaze estimation algorithms for our purpose and analyze their robustness.

[Lai et al. ICIP, 2018]

Reaction Time (ms)

iTracker is a CNN-based Gaze Estimation Algorithm Trained on 30-fps Recordings

12

[Krafka et al. CVPR, 2016]

To Apply iTracker for Our Purpose, We Manually Annotated the Facial Landmarks

13

Identify the face

location in the grid

We Propose iTracker-face for Clear Onset Presentation

14

• To measure saccade latency, we took recordings at 240 fps.

We propose iTracker-face: iTracker with only face-related layers.

30 fps 240 fps

Images become blurry.

• Our goal is to estimate when the gaze changes.i.e. Accuracy ⇒ Clear saccade onset

Eye Gaze (x,y)

• Test 1: Chinrest + manual face annotation on the first frame of each video

Compare iTracker and iTracker-face based on Whether a Clear Onset can be Seen

15

iTracker: Grey + Blue

iTracker-face: Blue

Video Recording Setup16

• Subject: Subjects were recorded while viewing a visual task that was displayed on a laptop.

• Recording device: iPhone 6 with a framerate of 240 fps and an image resolution of 1280 x 720 pixels

• Synchronized monitor: Duplicates the laptop’s display and allows for the calculation of reaction time.

iTracker-Face Produces Traces with Clearer Saccade Onsets Compared to iTracker

17

How about robustness?

We tested • under four lighting conditions (highest to lowest) • on five subjects • with and without glasses.⇒ For this experiment, we acquired 9,600 saccades in total.

Sample Frames from Four Lighting Conditions 18

278 Lux 220 Lux 54 Lux 26 Lux

0

20

40

60

80

100

High Light Room Light Medium Light Low Light

Without Glasses

iTracker-Face is more Robust than iTracker19

0

20

40

60

80

100

High Light Room Light Medium Light Low Light

With Glasses

• Two expert annotators visualized 9,600 traces and labeled them as good/bad saccades (clear/unclear onset).

• It took each annotator approximately 10-15 hours to do so.

iTracker-face is much more robust to lighting conditions and glasses.

Automatic Rejection of Bad Saccades is Crucial.

Ag

reed

Go

od

(%

)

iTracker

iTracker - face

[Lai et al. JBHI, 2019]

We Choose iTracker-face to be the Eye Tracking Algorithm

20

iTracker-face

Goal: Find a saccade onset detection method and a way to automatically detect bad saccades.

Approach: Model for saccade traces.

Reaction Time (ms)

Tanh Fitting can Measure Saccade Onset and Detect “Bad” Saccades

21

• The normalized root-mean-square error (NRMSE) between the model and the eye movement trace can be used to reject bad saccades.

Using an adequate NRMSE threshold, we can classify traces as “good” or “bad” with an average true-positive rate of 0.87 and an average false-positive rate of 0.18.

Recall that we have 9,600 annotated traces from the robustness test.

• Horizontal saccades are modeled using tanh.Saccade

latency

iTracker-face

Tanh model

3% of the

model

amplitude

Reaction

Time

We Choose Tanh Fitting to Measure Saccade Onset and Determine Good Saccades

22

iTracker-faceTanh fitting

with automated rejection

How do smartphone recordings compare to recordings made with a research-grade camera?

Reaction Time (ms)

High-Speed Video Recording Set-up23

• Subject: Subjects were recorded while viewing a visual task that was displayed on a laptop.

• High-speed video recordings: In a subset of recordings, subjects were simultaneously recorded with a high-speed video camera at 500 fps and an image resolution of 1280 x 720 pixels.

• Recording device: iPhone 6 with a framerate of 240 fps and an image resolution of 1280 x 720 pixels

• Synchronized monitor: Duplicates the laptop’s display and allows for the calculation of reaction time.

We Achieved Similar Statistics Compared with a Research Grade Camera

24

High-speed research-grade cameraPhantom v2511,cost $100k, 720p resolution, 500 fps28 µm pixel size, global shutter

Consumer grade cameraiPhone 6, cost <$1k,

720p resolution, 240 fps1.5 µm pixel size, rolling shutter

[Lai et al. ICIP, 2018]

Reaction Time (ms)

Our Pipeline is Almost Automated25

iTracker-faceTanh fitting

with automated rejection

Can we remove manual face annotation?

Reaction Time (ms)

• Test 1: Chinrest + manual face annotation

• Test 2: Chinrest + Viola-Jones face detector

Can Our Pipeline be Fully Automated?26

Approach: We compare the reaction time acquired from manual face annotation with the reaction time acquired

from the Viola-Jones face detector.

Manual Face Annotation and Face Detector Attained Almost Identical Reaction Time

27

The distribution shows the absolute difference in mean reaction time calculated with manual face annotation and the Viola-Jones face detector.

Automating face-detection does not affect the estimated reaction time.


Absolute Reaction Time Difference (ms)

Can We Remove the Chinrest?28

• Test 1: Chinrest + manual face annotation

• Test 2: Chinrest + Viola-Jones face detector

• Test 3: Without chinrest + Viola-Jones face detector

Recordings with Chinrest and without Have Comparable Signal-to-Noise Level

29

*The traces were intentionally chosen to have similar reaction time for easier comparison.


We Completely Automate the Pipeline for Reaction Time Measurement

30

iTracker-faceTanh-fitting

with automated rejectionViola-Jones face

detector

Reaction Time (ms)




• Conclusions

Outline31

Eye movement Data Breakdown32

We recorded 19,200 eye movements across 160 recording sessions in 29 subjects.

Average fraction of good data per session: 77%Average fraction of bad data due to noise: 13%

Data can be used

Data cannot be used


Reaction Time in Healthy Individuals33

• We recorded over 19,000 saccades in 29 healthy subjects.

• Among the 29 subjects, 10 subjects were recorded on multiple ocassions.


Saccade latency (ms)

No

rma

lize

d c

ou

nt

Reaction time (ms)

No

rma

lize

d C

ou

nt

Reaction time distributions have distinctive shapes and parameters. Reaction time variability is substantial within a subject and across subjects.




• Conclusions

Outline34

• We have enabled the measurement of visual reaction time outside of the clinical environment by proposing a robust, novel system that combines deep-learning and modeling techniques.

• We have eliminated the need for research-grade cameras, IR illumination, and chinrest by building algorithms that can measure reaction time using recordings from a smartphone camera.

• We collected over 19,000 measurements across 29 healthy individuals and observed that their reaction time distributions have distinctive shapes and parameters.

Conclusions35

Questions?

36

• Neurodegenerative Diseases

- “The new offensive on Alzheimer’s Disease: Stop it before it starts,”

https://www.newsweek.com/2017/02/24/stopping-alzheimers-disease-it-starts-557221.html,

Accessed on 04-26-2019.

- “Neurodegenerative diseases”,

https://www.niehs.nih.gov/research/supported/health/neurodegenerative/index.cfm, Accessed on

04-26-2019.

- B. Agrell and O. Dehlin, “The clock-drawing test,” Age and Ageing, vol. 27, pp. 399 – 403, 1998.

- H. Heuer, J. Mirsky, E. Kong, B. Dickerson, B. Miller, J. Kramer, and A. Boxer, “Antisaccade task

reflects cortical involvement in mild cognitive impairment,” Neurology, vol. 81, no. 14, pp. 1235 –

1243, 2013.

- S. Garbutt, A. Matlin, J. Hellmuth, A. Schenk, J. Johnson, H. Rosen, D. Dean, J. Kramer, J.

Neuhaus, B. Miller, S. Lisberger, and A. Boxer, “Oculomotor function in frontotemporal lobar

degeneration, related disorders and Alzheimer’s Disease,” Brain, vol. 131, pp. 1268 – 1281,

2008.

• Eye-Tracking Algorithm

- K. Krafka, A. Khosla, P. Kellnhofer, H. Kannan, S. Bhandarkar, W. Matusik, and A. Torralba,

“Eye tracking for everyone”, in Proceedings of the IEEE Conference on Computer Vision and

Pattern Recognition (CVPR), 2016, pp. 2176 – 2184.

References37

https://www.newsweek.com/2017/02/24/stopping-alzheimers-disease-it-starts-557221.html

https://www.niehs.nih.gov/research/supported/health/neurodegenerative/index.cfm

• Our Publications

- H.-Y. Lai, G. Saavedra-Peña, C. Sodini, T. Heldt, and V. Sze, “Enabling saccade latency

measurements with consumer-grade cameras”, in Proceedings of the IEEE International

Conference on Image Processing (ICIP), 2018, pp. 3169 – 3173.

- G. Saavedra-Peña, H.-Y. Lai, V. Sze, and T. Heldt, “Determination of saccade latency

distributions using video recordings from consumer-grade devices,” in Proceedings of the IEEE

Engineering in Medicine and Biology Conference (EMBC), 2018, pp. 953 – 956.

- H.-Y. Lai, G. Saavedra-Peña, C. Sodini, V. Sze, and T. Heldt, “Measuring saccade latency using

smartphone cameras,” Journal of Biomedical and Health Informatics (JBHI), 2019.

References38

measuring eye-movement patterns on a smartphone for medical...

Documents