TurkerGaze: Crowdsourcing Saliency with Webcam based Eye TrackingPingmei Xu Krista A. Ehinger Yinda Zhang Adam Finkelstein Sanjeev R. Kulkarni Jianxiong Xiao

Motivations

System Overview

Our Approach

http://isun.cs.princeton.eduVISI N GROUP

1. Traditional eye tracking requires special hardware. 2. Collecting data is expensive, tedious, slow. 3. Data cannot be collected from crowd sourcing platform, e.g. AMT.

Data Collection

Dataset & Benchmark1. Small datasets limit the potential for training data intensive algorithm2. Benchmark can be easily overfit.

1. We propose a webcam-based gaze tracking system, which can collect large-scale data from Amazon Mechanical Turk2. We apply a carefully designed gaming protocol to ensure the quality of the collected data.

1. Efficiency: Real-time performance in web browser.2. Uncontrolled environment: Head movement & Lack of attention.3. Privacy issue: Cannot streaming recorded video.

Challenging

Gaming Protocol

System Performance

Crowdsourcing vs. Lab Collected

0 256 512 768 1024

192

384

576

768

Screen WidthSc

reen

Hei

ght

Error in pixelError in pixel

0 100

Time (ms)0 500 1000 1500 2000 2500

X

200

300

400

500

600

700

800

Time (ms)0 500 1000 1500 2000 2500

Y

200

300

400

500

600EyelinkOurs

0

0.5

1

head fla

gtex

tclo

ck

paint

ingscr

eenpic

ture

signbo

ttle

flowers

desk

lamp

cush

ion glass

perso

nva

serai

ling

stree

tlight

pillow

hous

e

doub

le do

or

sconc

e

buildi

ngsmirro

rpa

ne door

book

window

baske

t

buildi

ng

armch

air

5%10%20%40%60%80%

head flag text clock painting screen picture sign bottle flowers desk lamp cushion glass person vase

railing streetlight pillow house double door sconce buildings mirror pane door book window basket building armchair

~20 21~30 31~40 41~50 51~50 60~0%

10%

20%

30%

40%

0%

47% male female

Age-Gender Distribution

Asian Hispanic White Native Black Other Unknown0%

10%

20%

30%

40%

50%

0%

56%

0.18

0.08

0.56

0.02 0.02 0.02

0.12

Race DistributionGeographic Distribution

Mechanical Turk Worker

Fixations from Judd et al. (2009) Fixations from AMTurk workers

Calibration Train Regression Model Prediction

1. Leverage the impact of these well known games to attract more workers.2. Encourage workers to work more carefully.3. Integrate the quality control to the experiment naturally.

Gaze prediction accuracy evaluated by a 33-point chart

Comparison with professional eye trackerRed: Eyelink Green: Our system

Fixation detection results. Our meanshift based algorithm on low sampling rate data provides satisfactory estimations. Red: ground truth, Yellow: estimation.

Statistics of basic information for workers from AMTurk

Distribution of fixations collected in the lab and on AMTurk

20 40 60 800.5

0.6

0.7

0.8

0.9

1

JuddAMTurk

20 40 60 80

20 40 60 80

Fixation prediction with 14 Judd subjects’ fixations

iSUN Database

Video Saliency Panoramic Saliency

Example of the saliency data obtained by our system. Object saliency statistics. Overlapping with top percents of saliency map.

Eye tracking saliency on SUN database: 9000 images (expected to label 20608 images)1. Saliency data on scene images. 2. Pixelwise semantic labelling + Scene category