Understanding Spatial Correlation in Eye-fixation

maps for Visual Attention in Videos

Tariq Alshawi*, Zhiling Long, and Ghassan AlRegib

Multimedia and Sensors Lab (MSL)

Center for Signal and Information Processing (CSIP)

School of Electrical and Computer Engineering

Georgia Institute of Technology

Outline

1. Introduction to Human Visual Attention• Motivation

• Applications

2. Data• Dataset

• Eye-fixations Maps

3. Spatial Correlation• Modeling

• Results and discussion

4. Conclusions

2

Introduction to Human Visual Attention:

Motivation

3

(Diagram from http://ivrgwww.epfl.ch/supplementary_material/RK_CVPR09)

Introduction to Human Visual Attention:

Applications

4

Auto-Cropping2Compression1

1. Chenlei Guo; Liming Zhang, "A Novel Multiresolution Spatiotemporal Saliency Detection Model and Its Applications in Image

and Video Compression," in Image Processing, IEEE Transactions on , vol.19, no.1, pp.185-198, Jan. 2010

2. F. W. M. Stentiford, “Attention based Auto Image Cropping,” Workshop on Computational Attention and Applications, ICVS,

Bielefeld, March 21-24, 2007.

Introduction to Human Visual Attention:

Uncertainty Framework

5

T. Alshawi, Z. Long, and G. AlRegib, "Unsupervised Uncertainty Analysis For Video Saliency Detection" the 49th Asilomar Conference

on Signals, Systems and Computers, Pacific Grove, CA, Nov. 8-11, 2015

Dataset

• CRCNS

• 50 video clips, 5-90 seconds

• Street scenes, TV sports, TV

news, TV talks, video games,

etc.

• Ground truth by human subjects

(eye tracking)

6

Preparing Eye-fixation maps

7

[#] [x,y] [t,N]

[#] [x,y] [t,N]

[#] [x,y] [t,N]

[#] [x,y] [t,N]

[#] [x,y] [t,N]

[#] [x,y] [t,N]

[#] [x,y] [t,N]

Eye-Fixation Data

240 Hz

Spatial Correlation:

Spatiotemporal neighbors

8

Frame# k Frame# k+1Frame# k–1

Pixel of

Interest

Spatial Correlation:

Modeling

9

Frame# k Frame# k+1Frame# k–1

Pixel of

Interest

Frame# k Frame# k+1Frame# k–1

Pixel of

Interest

Temporal NeighborsSpatial Neighbors

Spatial Correlation:

Results (Spatial)

10

gamecube_07

Spatial Correlation:

Results (Spatial)

11

sccadetest_01

Spatial Correlation:

Results (Spatial)

12

tv-news_03

Spatial Correlation:

Results (Temporal)

13

Conclusions

• Insights into visual attention mechanisms for videos can help improve saliency-dependent video processing applications

• Analysis of eye-fixation maps correlation, independent of video content

• Experiments show substantial correlation between saliency of a pixel and that of its direct neighbors

• Eye-fixation map correlation is significantly affected by the video’s content and complexity

• Eye-fixation correlation can be used as a measure of the reliability of detected saliency, thus, optimize saliency-based video processing applications

14

Questions?

15