An Extensive Empirical Evaluation of Focus Measures for Digital Photography

Hashim Mir, Peter Xu, and Peter van BeekUniversity of Waterloo

• Digital cameras are equipped with contrast-based autofocus

• Contrast-detection autofocus makes use of a focus measure• maps an image to a value that represents degree of focus

of image

• Speed and accuracy of autofocus crucial to user satisfaction• search for an image with maximal focus according to focus

measure• choice of focus measure is an important factor in

performance of autofocus

February 4, 2014 Focus Measures for Digital Photography 2

Introduction

• Previous studies on focus measures:• small number of benchmarks images in their evaluation,• directed at microscopy and not digital cameras, or• based on ad hoc evaluation criteria

• Here: • extensive empirical evaluation of focus measures for digital

photography• advocate using three standard statistical measures of

performance—precision, recall, and mean absolute error—as evaluation criteria

February 4, 2014 Focus Measures for Digital Photography 3

Introduction

• Introduction• Focus Measures• Related Work• Experimental Methodology• Experimental Results• Conclusion

February 4, 2014 Focus Measures for Digital Photography 4

Outline

• Many focus measures have been proposed in the literature

• Let f (x, y) be the luminance or grayscale at pixel (x, y) in an image of size M × N pixels

• Example: squared gradient focus measure for an image acquired when the lens is at position p is given by,

February 4, 2014 Focus Measures for Digital Photography 5

Focus measures

1 22

=0 =0

( ) = ( ( , 1) ( , ))M N

x y

p f x y f x y

(a) Focus measures of images at each lens position; (b) flower in focus

February 4, 2014 Focus Measures for Digital Photography 6

(a) (b)

(a) Focus measures of images at each lens position; (b) fern & grasses in focus

February 4, 2014 Focus Measures for Digital Photography 7

(a) (b)

• We evaluated more than 30 focus measures in our study

• The focus measures can be classified under six types:1. first-order differentiation2. second-order differentiation3. image histogram4. image statistics5. correlation6. data compression

February 4, 2014 Focus Measures for Digital Photography 8

Focus measures: Categorizing

• Example: squared gradient• Example: focus measure based on 3×3 Sobel

operators,

February 4, 2014 Focus Measures for Digital Photography 9

Focus measures: First-order differentiation

1 0 1 1 2 1

2 0 2 0 0 0

1 0 1 1 2 1

(horizontal) (vertical)

• Example: focus measure based on 3×3 Laplacian operator,

February 4, 2014 Focus Measures for Digital Photography 10

Focus measures: Second-order differentiation

1 1 1

1 8 1

1 1 1

• Example: variance focus measure is given by,

February 4, 2014 Focus Measures for Digital Photography 11

Focus measures: Image statistics

1 12

=0 =0

1( ) = ( ( , ) )

M N

x y

p f x yMN

where µ is the mean luminance of the image

• Example: Vollath’s F4 focus measure is given by,

February 4, 2014 Focus Measures for Digital Photography 12

Focus measures: Correlation

2 1

=0 =0

31 1

=0 =0

( ) = ( , ) ( 1, )

( , ) ( 2, )

M N

x y

M N

x y

p f x y f x y

f x y f x y

February 4, 2014 Focus Measures for Digital Photography 13

Related work: Evaluating focus measures

setting citation conclusions

Microscopy applications

Groen, Young & Ligthart, 1985

squared gradient

Firestone et al. 1991 vertical Brenner, variance

Santos et al. 1997 Vollath's F4, F5

Liu, Wang & Sun 2007 variance

Digital photography

Subbarao and Tyan 1998 Laplacian

Ng, Neow & Ang 2001 first-order 3×3 Sobel, variance

Shih 2007 first-order 3×3 Sobel & Prewitt

Yousefi et al. 2011 Vollath's F4

• Step 1: Implemented a camera remote control application

• Step 2: Using app, gathered 25 sets of benchmark images • cover a range of common photography settings • 4,303 jpeg images (23 × 167 + 2 × 231, one for each

focus position)• Canon EOS 550D/Rebel T2i camera

February 4, 2014 Focus Measures for Digital Photography 14

Experimental methodology: Comparative evaluation of focus measures for digital photography

• Step 3: Determined ground truth• true lens position (or image) where an object is in best

focus

• Step 4: Implemented more than 30 focus measures• extensive survey of the literature

• Step 5: Given a benchmark set of images and a focus measure• applied focus measure to each jpeg image in benchmark • focus measure predicts where an object will be in best

focus• given predictions of focus measure and ground truth,

calculated precision, recall, and mean absolute error

February 4, 2014 Focus Measures for Digital Photography 15

Experimental methodology: Comparative evaluation of focus measures for digital photography

ground truthtrue false

predicted

truetp

(true positive)

fp(false

positive)

falsefn

(false negative)

tn(true

negative)

February 4, 2014 Focus Measures for Digital Photography 16

Experimental methodology: Precision and recall

=tp

precisiontp fp

=tp

recalltp fn

February 4, 2014 Focus Measures for Digital Photography 17

squared gradient Laplacian

February 4, 2014 Focus Measures for Digital Photography 18

squared gradient Laplacian

3=

3 0precision

3=

3 1recall

4=

4 0precision

4=

4 0recall

• MAE chosen over mean squared error • MAE prefers errors that are occasionally large but small on

average• mean square error prefers errors that avoid large errors

while still possibly being quite unsatisfactory overall

February 4, 2014 Focus Measures for Digital Photography 19

Experimental methodology: Mean absolute error (MAE)

p p absolute error =

where p is lens position of image in maximal focus

is lens position that focus measure predictsp

Focus measure precision

recall MAE

Brenner 100.00 99.00 0.00squared gradient 100.00 99.00 0.003 × 3 difference 98.00 97.00 0.023 × 3 Sobel 98.00 97.00 0.023 × 3 Scharr 98.00 97.00 0.023 × 3 Roberts 98.00 97.00 0.023 × 3 Prewitt 98.00 97.00 0.02Gaussian 95.00 92.00 0.05vertical sq. gradient

91.33 92.00 0.55

vertical Brenner 91.33 90.00 0.23

February 4, 2014 Focus Measures for Digital Photography 20

Experimental results: First-order differentiation

Focus measure precision

recall MAE

Laplacian of Gaussian

98.00 99.00 0.32

5 × 5 vertical Sobel

96.67 99.00 0.65

5 × 5 Laplacian 96.67 99.00 0.465 × 5 cross Sobel 90.33 93.00 0.663 × 3 cross Sobel 89.67 93.00 1.183 × 3 vertical Sobel

88.73 96.00 3.94

3 × 3 Laplacian 85.00 91.00 1.415 × 5 horizontal Sobel

83.33 88.00 1.08

3 × 3 horizontal Sobel

73.27 78.00 2.95February 4, 2014 Focus Measures for Digital Photography 21

Experimental results : Second-order differentiation

Focus measure precision

recall MAE

normalized variance

43.57 53.00 26.36

variance 32.33 35.00 20.80threshold pixel count

4.24 18.00 52.38

threshold content 2.13 4.00 36.31power 0.53 2.00 43.80

February 4, 2014 Focus Measures for Digital Photography 22

Experimental results: Image statistics

Focus measure precision

recall MAE

autocorrelation 80.40 99.00 26.87Vollath's F4 61.74 87.00 24.60

Vollath's F5 33.00 35.00 19.00

February 4, 2014 Focus Measures for Digital Photography 23

Experimental results: Correlation

• Results:• some popular focus measures perform poorly when applied

to autofocusing in digital photography or to more representative sets of images• simple focus measures based on taking the first derivative

of an image perform exceedingly well in digital photography, although care must be taken in which direction derivative is taken

April 8th 2013Combinatorial Problems in Compiler

Optimization24

Experimental results: Discussion

• The speed and accuracy of a digital camera's contrast-based autofocus algorithm are crucial to user satisfaction

• We advocate using three standard statistical measures of performance—precision, recall, and mean absolute error—as accuracy criteria

• Our experimental results indicate that two simple focus measures based on taking first derivative of an image perform exceedingly well in digital photography

February 4, 2014 Focus Measures for Digital Photography 25

Conclusion