a real time automatic eye tracking system for ophthalmology

58
A Real-Time Automatic Eye Tracking System for Ophthalmology Mr. Wattanit Hotrakool Mr. Prarinya Siritanawan Supervised by Dr.Toshiaki Kondo Sirindhorn International Institute of Technology 1

Upload: prarinya-siritanawan

Post on 14-Jun-2015

670 views

Category:

Technology


2 download

DESCRIPTION

Presentation of my senior Project about "A real time automatic eye tracking system for ophthalmology"In the presentation, it briefly explains about conventional object tracking method "template matching" based on Sum-of-Square difference. Therefore we also present the powerful matching technique called Gradient Orientation Pattern Matching (GOPM) proposed by T.Kondo and we proposed an improved version of GOPM called time-vary GOPM to solve a illumination and noise problem.

TRANSCRIPT

Page 1: A real time automatic eye tracking system for ophthalmology

A Real-Time Automatic Eye Tracking

System for Ophthalmology

Mr. Wattanit Hotrakool

Mr. Prarinya Siritanawan

Supervised by Dr.Toshiaki Kondo

Sirindhorn International Institute of Technology

1

Page 2: A real time automatic eye tracking system for ophthalmology

Outline

Project Background

Project Objective

A eye-tracking technique using Traditional Template

Matching

A eye-tracking technique using Gradient Orientation

Pattern Matching

A eye-tracking technique using Time-varying GOPM

Conclusion

Question and Answer

2

Page 3: A real time automatic eye tracking system for ophthalmology

Project Background

The conventional eye-surgery cameras are manual-

controlled and they reduce the efficiency of surgery.

In order to reduce the burden of oculist, the automated

camera control is required. The image processing is used

to locate and track the eye’s centroid.

3

Page 4: A real time automatic eye tracking system for ophthalmology

Project Background

Many real-time eye tracking techniques used intensity

data as an input; they are very sensitive to changing

lighting condition and result as miss-matching.

This project proposes new template matching based

technique which robust to changing lighting condition by

using Time-varying Gradient Orientation PatternMatching.

4

Page 5: A real time automatic eye tracking system for ophthalmology

Project Objectives

To implement matching-based techniques in real-time.

To verify the robustness to changing lighting condition of

gradient orientation pattern matching.

To develop new eye-tracking technique; a time-varying

gradient orientation pattern matching.

5

Page 6: A real time automatic eye tracking system for ophthalmology

Eye-tracking method using template

matching technique

Template matching is the intensity-based technique for

measuring the similarity between template and

corresponding block of image.

Template

Sample frame

Match

6

Page 7: A real time automatic eye tracking system for ophthalmology

Simulation Specification

Simulation System using the following hardware and

software specifications:

Software Specification

Operating System Windows/Linux

Programming Language C/C++

Primary library OpenCV 2.0

Hardware Specification

Processor Intel Core2 Duo

Processor Speed 1.66 GHz

Memory 4GB

7

Page 8: A real time automatic eye tracking system for ophthalmology

Video sequences used for simulation

Video sequences used in this simulation can be

categorized into 3 categories:

1. Test video in normal lighting condition

2. Test video in changing lighting condition

3. Actual surgery video from real camera

8

Page 9: A real time automatic eye tracking system for ophthalmology

Template Initialization

Template initialization is required before the eye-tracking

method.

The pupil is specified as a template in order to track the

eye.

The example of video and corresponding template for

each categories is shown in next page

9

Page 10: A real time automatic eye tracking system for ophthalmology

Test video in normal lighting condition

Video sequence

Template

10

Page 11: A real time automatic eye tracking system for ophthalmology

Test video in changing lighting condition

Video sequence

Template

11

Page 12: A real time automatic eye tracking system for ophthalmology

Actual surgery video from real camera

Video sequence

Template

12

Page 13: A real time automatic eye tracking system for ophthalmology

A eye-tracking method using

Traditional template matching

13

Page 14: A real time automatic eye tracking system for ophthalmology

A eye-tracking method using Traditional

template matching

There are many traditional techniques of template

matching such as sum-of-absolute difference (SAD), sum-

of-squared difference (SSD), or cross-correlation (CC)

technique.

In this step, we implement the sum-of-squared difference

technique (SSD) to be eye-tracking method.

14

Page 15: A real time automatic eye tracking system for ophthalmology

Sum-of-squared Difference technique

SSD is the template matching method done by finding the

lowest difference value between input and template. The

differences are squared in order to remove the sign.

where I1 is the intensity of input image and

I2 is the intensity of template

N is the size of the template

15

Page 16: A real time automatic eye tracking system for ophthalmology

Procedure

1. Convert template to gray scale image.

2. Convert input frame to gray scale image.

3. For every pixel, compute the SSD between input and

template.

4. Find the minimum difference pixel, which is the best

matching location.

5. For every frame, repeat step 2-4.

16

Page 17: A real time automatic eye tracking system for ophthalmology

Procedure

17

Page 18: A real time automatic eye tracking system for ophthalmology

Result

Input Average computation time

(ms)

Precision error

(%)

Normal light video

(resolution: 320x240 px)

78.35 1.33

Changing light video

(resolution: 320x240 px)

78.63 40.47

Actual surgery video

(resolution: 384x288 px)

103.33 0

Average computation time mostly depends on video

resolution

However, this method currently can process at 10 -12.5 frame/sec

18

Page 19: A real time automatic eye tracking system for ophthalmology

Result

Input Average computation time

(ms)

Precision error

(%)

Normal light video

(resolution: 320x240)

78.35 1.33

Changing light video

(resolution: 320x240)

78.63 40.47

Actual surgery video

(resolution: 384x288)

103.33 0

SSD technique can work very well in normal light video.

However, this technique give high error in changing light video

because it uses intensity data which are sensitive to light.

Therefore SSD cannot work in changing light condition

19

Page 20: A real time automatic eye tracking system for ophthalmology

Result

Properties of eye-tracking using SSD technique

Obstacle robustness Yes

Blur robustness Yes

Light condition robustness No

Scaling robustness No

Average computation time About 50-350 ms

20

Page 21: A real time automatic eye tracking system for ophthalmology

Presented at ICESIT2010, Chiang Mai, Thailand

A eye-tracking method using Gradient

Orientation Pattern Matching

21

Page 22: A real time automatic eye tracking system for ophthalmology

A eye-tracking technique using Gradient

Orientation Pattern Matching

In order to develop a method that can provide the

robustness to light condition, the new template matching

technique is used.

The gradient orientation pattern matching (GOPM) is a

new template matching technique proposed by Dr.

Toshiaki Kondo.

22

Page 23: A real time automatic eye tracking system for ophthalmology

Gradient Orientation Pattern Matching

GOPM is a template matching method which use the

normalized gradient of the image in place of intensity

data. Thus, it only consider about the shape of the

pattern but not light.

Gradient vector is the first derivative of intensity. The

gradient in x and y direction are defined as:

where I is a intensity of an image

23

Page 24: A real time automatic eye tracking system for ophthalmology

Gradient Orientation Pattern Matching

The gradient in x and y will then be normalized. This

step provides the robustness to light condition. The

normalized gradient in x and y direction are defined

as:

where

24

And is a small constant used to prevent zero-division.

Page 25: A real time automatic eye tracking system for ophthalmology

Gradient Orientation Pattern Matching

The normalized gradient in x and y direction of input

frame and template will be match by using SSD

where N1 is the normalized gradient of input image and

N2 is the normalized gradient of template

25

Page 26: A real time automatic eye tracking system for ophthalmology

Procedure

For every frame, we can divide the procedure into 2 main

steps;

1. Gradient orientation information (GOI) extraction

2. Gradient orientation pattern matching (GOPM)

26

Page 27: A real time automatic eye tracking system for ophthalmology

Gradient Orientation Information (GOI)

Extraction

Extract the gradient Images

(Template and sample frame) in

x and y direction.

27

Page 28: A real time automatic eye tracking system for ophthalmology

Gradient Orientation Pattern Matching

Apply template matching in x and y

direction. Then add the result of x and

y direction

28

Page 29: A real time automatic eye tracking system for ophthalmology

Result

Input SSD technique GOPM technique

Average

computation

time (ms)

Precision

error (%)

Average

computation

time (ms)

Precision

error (%)

Normal lighting condition

(resolution: 320x240px)

78.35 1.33 62.48 0

Change lighting condition

(resolution: 320x240px)

78.63 40.47 63.09 12.87

Actual surgery camera

(resolution: 384x288px)

103.33 0 77.7 0

Average computation time of GOPM slightly inprove from

SSD

Even though the method is more complex, but the computation time

is decrease due to the variable type and internal structure of OpenCV

However, this method still can process at only <15 frame/sec29

Page 30: A real time automatic eye tracking system for ophthalmology

Input SSD technique GOPM technique

Average

computation

time (ms)

Precision

error (%)

Average

computation

time (ms)

Precision

error (%)

Normal lighting condition

(resolution: 320x240px)

78.35 1.33 62.48 0

Change lighting condition

(resolution: 320x240px)

78.63 40.47 63.09 12.87

Actual surgery camera

(resolution: 384x288px)

103.33 0 77.7 0

Result

In changing light condition, GOPM error is dramatically decrease due to

the normalized process.

Therefore GOPM can provide the robustness to changing light

condition

30

Page 31: A real time automatic eye tracking system for ophthalmology

Result

Properties of eye-tracking using SSD and GOPM technique

SSD GOPM

Obstacle robustness Yes Yes

Blur robustness Yes Yes

Light condition robustness No Yes

Scaling robustness No No

Average computation time About 50-350 ms About 50-350 ms

31

Page 32: A real time automatic eye tracking system for ophthalmology

A eye-tracking method using a

Time-varying GOPM

Accepted by ECTI-CON 2010, Chiang Mai, Thailand

32

Page 33: A real time automatic eye tracking system for ophthalmology

Time-varying GOPM

Even though GOPM provides robustness to changing light

condition, however the static template will not

guaranteed that it yields the good result for all condition.

There are many uncontrolled factors such as skin and

noise.

Time-varying GOPM uses the dynamic template which

update itself automatically in place of static template. It

reduce the difference of template environment in various

period of time.

33

Page 34: A real time automatic eye tracking system for ophthalmology

Template Update Algorithm

Step 1 : Perform GOPM, get best matching coordinate

current Template

BEST MATCH

Sample frame

34

Page 35: A real time automatic eye tracking system for ophthalmology

Template Update Algorithm

Step 2 : Crop region with the same size of old template

for creating new template

new Template

Sample frame

35

Page 36: A real time automatic eye tracking system for ophthalmology

Correct-matching criterion

1st Criterion Equation

where Nxn+1 and Nyn+1 are the normalized gradient of the newly created template, Nxn and Nyn are the normalized gradient of the current template, i and j are the size of the template.

is a threshold value defined as

1st Criterion is used to check the correctness of the updated template and prevent the jumping coordinate.

36

Page 37: A real time automatic eye tracking system for ophthalmology

Correct-matching criterion

2nd Criterion Equation

where Xn and Yn are the location of current matching result,

Xc and Yc are the location of the last known correct result.

T is a threshold value define as

2nd Criterion is used to double check the jumping real

coordinate.

37

Page 38: A real time automatic eye tracking system for ophthalmology

1st Criterion

STEP 1 : Find the difference b/w gradient component of

old template and new template in X and Y direction

Old Template New Template

=-Template diff.

=-x

y

38

Page 39: A real time automatic eye tracking system for ophthalmology

1st Criterion

STEP 2 : Combine the difference of x and y

STEP 3 : Sum all elements and then thresholding

If summation is less than the threshold function, update

template.

If summation is more than the threshold function, discard the

new template.

Diff x Diff y Total Diff

=+

39

Page 40: A real time automatic eye tracking system for ophthalmology

2nd Criterion

Using the fact that It is impossible that the eye would

change its position suddenly in next frame.

Frame N Frame N+1

(300,500)

(400,100)

31

40

Page 41: A real time automatic eye tracking system for ophthalmology

2nd Criterion

STEP 1 : Find the best matching of the frame N.

Frame N

(x1,y1)

41

Page 42: A real time automatic eye tracking system for ophthalmology

2nd Criterion

STEP 2 : If location of N passed the criteria, the location is used as a latest known correct position C.

Frame N

42

Page 43: A real time automatic eye tracking system for ophthalmology

2nd Criterion

STEP 3 : Find best matching of the frame N+1.

Frame N+1

(x2,y2)

43

Page 44: A real time automatic eye tracking system for ophthalmology

2nd Criterion

STEP 4 : Find Euclidean distance between position C and position in frame N+1.

Frame N+1

44

Page 45: A real time automatic eye tracking system for ophthalmology

2nd Criterion

STEP 5.1 : If distance more than threshold function, discard the current location.

Frame N+1

Threshold fcn

45

Page 46: A real time automatic eye tracking system for ophthalmology

2nd Criterion

STEP 5.2 : If distance less than threshold function, mask the location as new corrected position C.

Frame N+1

Threshold fcn

46

Page 47: A real time automatic eye tracking system for ophthalmology

Correct-matching criteria with Template Update

47

Page 48: A real time automatic eye tracking system for ophthalmology

Procedure

48

Page 49: A real time automatic eye tracking system for ophthalmology

Downsampling

In here, we resizes input video sequence and template to

50 percent of the height and width. Hence the

downsampled image is reduced to ¼ of the original size.

Thus, computation time is 4 times faster.

No effect to the matching result since both video and

template are downsampled with the same ratio

49

Page 50: A real time automatic eye tracking system for ophthalmology

Result

Input GOPM technique Time-varying GOPM

Average

computation

time (ms)

Precision

error (%)

Average

computation

time (ms)

Precision

error (%)

Normal lighting condition

(resolution: 320x240 px)

62.48 0 13.81 0

Change lighting condition

(resolution: 320x240 px)

63.09 12.87 12.92 0

Actual surgery camera

(resolution: 384x288 px)

77.7 0 17.12 0

Average computation time is decreased by downsampling

Currently this method can process at > 50 frame/sec which is enough

for most of video capture device that run at 25 frame/secs.

50

Page 51: A real time automatic eye tracking system for ophthalmology

Result

Input GOPM technique Time-varying GOPM

Average

computation

time (ms)

Precision

error (%)

Average

computation

time (ms)

Precision

error (%)

Normal lighting condition

(resolution: 320x240 px)

62.48 0 13.81 0

Change lighting condition

(resolution: 320x240 px)

63.09 12.87 12.92 0

Actual surgery camera

(resolution: 384x288 px)

77.7 0 17.12 0

In all cases, time-varying GOPM provide the better result than normal

GOPM. Especially for the case of changing light condition, which error is

decrease to 0%

Therefore time-varying GOPM can provide the robustness to

changing light condition with more precision than normal GOPM

51

Page 52: A real time automatic eye tracking system for ophthalmology

Result

Properties of eye-tracking using SSD, GOPM , and time-varying GOPM

technique

SSD GOPM Time-varying

Obstacle robustness Yes Yes Yes

Blur robustness Yes Yes Yes

Lighting condition robustness No Yes Yes

Scaling robustness No No No

Average computation time About 50-350

ms

About 50-350

ms

About 10-90

ms

52

Page 53: A real time automatic eye tracking system for ophthalmology

Drawback of time-varying GOPM

In rarely case, when the 2nd criteria drop the frame

repeatedly, it causes the template slightly shifts from the

eye’s centroid.

However for real implementation prospective, if not be

too much, can be tolerated by the surgeon

Fail update

53

Page 54: A real time automatic eye tracking system for ophthalmology

Conclusion

54

Page 55: A real time automatic eye tracking system for ophthalmology

Conclusion

This work verify that the speed of template matching

technique with downsampling is able to implement in

real-time. (speed > 50 frame/secs).

In the changing light condition, the result clearly shows

that GOPM is more robust than SSD.

A time-varying GOPM reduce the difference of

template environment in various time and provides the

higher precision of tracking than normal GOPM.

55

Page 56: A real time automatic eye tracking system for ophthalmology

Future Work

Optimize the utilization of the threshold function in

corrected-matching criterion.

Due to difference in camera specification such as

resolution or sensitivity, it required other advance

method to supervised the threshold function such as

machine learning or neuron network.

56

Page 57: A real time automatic eye tracking system for ophthalmology

Acknowledgement

Assist. Prof. Dr. Toshiaki Kondo

Assoc.Prof. Dr. Waree Kongprawechnon

Dr. Itthisek Nilkhamhang

All faculty members and our beloved friends

57

Page 58: A real time automatic eye tracking system for ophthalmology

Question and Answer

58