Image Deblurring with Optimizations

Qi ShanQi ShanLeo Jiaya Jia Leo Jiaya Jia Aseem AgarwalaAseem Agarwala

University of WashingtonThe Chinese University of Hong KongAdobe Systems, Inc.

2

The Problem

An Example

4

Previous Work (1)

Hardware solutions:

[Raskar et al. 2006]

[Ben-Ezra and Nayar 2004]

[Levin et al. 2008]

5

Previous Work (2)

Multi-frame solutions:

[Petschnigg et al. 2004]

[Jia et al. 2004] [Rav-Acha and Peleg 2005]

[Yuan et al. 2007]

6

Previous Work (3)

Single image solutions:

[Jia 2007][Fergus et al. 2006]

[Levin et al. 2007]

Most recent work on Single Image Deblurring

Qi Shan, Jiaya Jia, and Aseem AgarwalaHigh-Quality Motion Deblurring From a Single Image. SIGGRAPH 2008

Lu Yuan, Jian Sun, Long Quan and Heung-Yeung ShumProgressive Inter-scale and intra-scale Non-blind Image Deconvolution. SIGGRAPH 2008.

Joshi, N., Szeliski, R. and Kriegman, D. PSF Estimation using Sharp Edge Prediction, CVPR 2008.

A. Levin, Y. Weiss, F. Durand, W. T. Freeman Understanding and evaluating blind deconvolution algorithms. CVPR 2009

Sunghyun Cho and Seungyong Lee, Fast Motion Deblurring.SIGGRAPH ASIA 2009

And many more...

Some take home ideas

1. Using hierarchical approaches to estimate kernel in different scales

2. Realize the importance of strong edges

3. Bilateral filtering to suppress ringing artifacts

4. RL deconvolution is good, but we've got better chioces

5. Stronger prior does a better job

6. Deblurring by assuming spatially variant kernel is a good way to go

Today's topic

How to apply natural image statistics, image local smoothness constraints, and kernel sparsity prior in a MAP process

Short discussion on

1. the stability of a non-blind deconvolution process

2. noise resistant non-blind deconvolution and denoising

10

Image Global Statistics

…

11

…

Image Global Statistics

12

Image Global Statistics

13

LI

Image Local Constraint

14

LI

Image Local Constraint

15

LI

Image Local Constraint

16

LI

12 )( |) ( 0,i ii N dL dIp L

p2

Image Local Constraint

17

exponentially distributed

) ( jfj ep f

Kernel Statistics

18

Combining All constraints

1 2min ( , ) min log[ ( ) ( ) ( ) ( )]E L f p n p dL p L p f

L f n

Two-step iterative optimization• Optimize L• Optimize f

19

Idea: separate convolution

22'( ) \ (|| * || )E L Sum L f I

log ( )p n

Optimize L

1log ( )p dL 2log ( )p L

22 2(\ ( || || ))i i iSum m dL dI

Optimization Process

idLreplace with i

1 1 1|| log ( ) ||p dL

20

22'( ) \ (|| * || )E L Sum L f I

log ( )p n

Idea: separate convolution

Optimize L

1log ( )p dL 2log ( )p L

1 1 1|| lo g ( ) ||p 22 2(\ ( || || ))i iiSum m dI

Optimization Process

idLreplace with i

21

22'( ) \ (|| * || )E L Sum L f I

1 1 1|| log ( ) ||p 22(|| || )dL 2

2 2(\ (|| || ))i iSum dI

22arg min \ (|| * || )opt LL Sum dL f dI 2

2(|| || )dL

Adding a new constraint to makeRemoving terms that are not relevant to

~ dLL

Updating L

An easy quadratic optimization problem with a closed form solution in the frequency domain

22

Updating

Removing terms that are not relevant to

22'( ) \ (|| * || )E Sum L f I

1 1 1|| lo g ( ) ||p 22(|| || )dL 2

2 2(\ (|| || ))i iSum dI

21 1 1 2 2arg min || log ( ) || (\ (|| || ))opt i ip Sum dI

22(|| || )dL

23

each only contains a single variable Ψi'i

E

21 1 1 2 2arg m in || lo g ( ) || (\ ( || || ))op t i ip Sum dI

22(|| || )dL

arg min(\ ( ' ))i

Sum E

It is then a set of easy single variable optimization problems

24

Iteration 0 (initialization)

25

Time: about 30 seconds for an 800x600 image

Iteration 8 (converge)

26

A comparison

RL deconvolution

27

A comparison

Our deconvolution

28

Two-step iterative optimization• Optimize L• Optimize f

22 1( ) || * || || ||E f L f I f

1 2min ( , ) min log[ ( ) ( ) ( ) ( )]E L f p n p dL p L p f

Optimization with a total variation regularization

29

Results

30

Results

31

32

33

More results

34

More results

Today's topic

How to apply natural image statistics, image local smoothness constraints, and kernel sparsity prior in a MAP process

Short discussion on

1. the stability of a non-blind deconvolution process

2. noise resistant non-blind deconvolution and denoising

Stability

Considering the simplest case: Wiener Filtering

T

T

FX B

F F I

How about if *B B n

**

T

T T

F XX X n

F F I F F I

* *T

T

FX B

F FAnd

Stability

Thus

* 2 2 22 2|| || || ||

PX X C

PP

Pwhere is the frequency domain representation of

is the variance of the noise

Observation: the noise in the blur image is magnified in

the deconvolved image. And the Noise Magnification

Factor (NMF) is solely determined by the filter

F2

F

Some examples

Some examples

Dense kernels are less stable for deconvolution than sparse ones

40

Noise resistant deconvolution and denoising

With Jiaya Jia, Singbing Kang and Zenlu QinIn CVPR 2010

Blind and non-blind image deconvolution softwareis available online and will be updated soon!

See you in San Francisco!