Image Deblurring using Fast Best Kernel Retrieval

Hui-Yu Huang* Dep. of Computer Science and Information Engineering

National Formosa University, Taiwan Email:hyhuang@nfu.edu.tw

Wei-Chang Tsai Dept. of Computer Science and Information Engineering

National Formosa University, Taiwan Email:simplelife1988@gmail.com

Abstract—In this paper, we propose a kernel estimation algorithm using fast best kernel retrieval (FBKR) from spectral irregularities to achieve the deblurring. In daily life, when people take a photograph by any kinds of camera equipment, image motion blur caused by camera motion often happens. Camera motion during exposure will produce a blurred image; this kind of blur is often non-liner mode. Motion blur always causes the decline of image quality, as long as the users using the hand-held photography equipment often have a similar experience. Based on this reason, reconstructing a clear image from blurred image is the main objective in this paper. Reconstruction is an ill-pose problem. In current motion blur estimation algorithms, these algorithms usually use the recursive method to estimate motion blur kernel. However, recursive process is quite time-consuming. In order to enhance time-consuming, in this paper, based on iterative phase retrieval algorithm and normalized sparsity measure, we propose a fast and effective blur kernel retrieval algorithm, which can find the best kernel in a short time. Experiments verify that our method can effectively reduce the execution time and obtain the best motion blur kernel to deblur a deblurred image and keep the quality of image deblurring.

Keywords- deblurring, non-linear motion blur, blur kernel, image restoration.

I. INTRODUCTION In daily life, non-linear motion blur image is often

appearing in hand-held camera or the camera mounted on vehicle. In order to avoid motion blur situation from camera shake, the manufacturers of photographic equipment have put forward a variety of different principles of anti-shock technology. In general, anti-shock technology can be divided into three major technologies: optical anti-shake technology, electronic anti-shake technology, and mechanical anti-shake technology. No matter what kind of anti-shake technologies they used, they have the limitations. Therefore, the image blurring caused by excessive shock cannot be avoided. The motion blur image reconstruction is still a major problem in image processing.

Nonlinear motion blur is caused by the relative motion of the camera and shooting scene during image exposure. At present, most of the researches are regarded motion blur information is assumed to be a point spread function (PSF)[1-3]. The blurring process is modeled by

* ,B I K N� � (1)

B is blurred image, I is clear image, * is convolution, K is a point spread function and N is noise. Motion blur image reconstruction first step is to find what is causing image blur and remove it. Therefore, how to estimate blurred kernel K from blurred image B is the key point in motion blur image reconstruction.

Blurred kernel information is usually hidden in edges information as Fig. 1 shown. If the edge of an image suffered serious damage by blurred kernel, it will result in inaccurate blur kernel estimation. The previous work has been raised quite a lot of motion blur for reconstruction. Tai et al. [4] purposed a projective Richardson Lucy algorithm which is based on Richardson Lucy algorithm to deblur the image. This method needs to do 500 iterations of a blurred image, thereafter, a clear image can be obtained, and it is very time-consuming. Dobeš et al. [5] proposed a solution of a linear motion blur images. Authors use Fourier spectral characteristics dappling Radon projection to identify the blurred kernel angle and through the projection of the target function to calculate the blurred core length. Zhang et al. [6] proposed an algorithm for an image edge-based kernel estimation algorithm.

According to the definition of wavelet module value, three-order spline wavelet was chosen to implement multiscale wavelet edge detection and wavelet edge module values were obtained. Area and girth of the blurred edge of the objects in the blurred image were calculated according to pixel statistic principle. Blurred edge width of objects in the image was given by area and girth of the transition edge. Goldstein and Fattal [7] and Dobeš et al. [5] used the frequency domain methods to estimate blur kernel, but Dobeš only estimated the linear motion kernel. Goldstein proposed an algorithm that can estimate non-linear motion kernel. Goldstein and Fattal described a recovering the blur kernel method in motion-blurred images based on statistical irregularities their power spectrum exhibits. The blur kernel is then recovered using a phase retrieval algorithm with improved convergence and disambiguation capabilities. Unlike many existing methods, they don’t perform a maximum a posteriori (MAP) estimation, which involves the repeated reconstructions of the latent image, and hence offers attractive running times.

In this paper, based on blur kernel estimation from statistical [7] proposed by Goldstein and mean structural similarity (SSIM) [14], we propose a fast blur kernel retrieval algorithm that can accelerate the speed of blurred kernel search and obtain a clear deblurring image.

2013 Second IAPR Asian Conference on Pattern Recognition

978-1-4799-2190-4/13 $26.00 © 2013 IEEE

DOI 10.1109/ACPR.2013.110

611

2013 Second IAPR Asian Conference on Pattern Recognition

978-1-4799-2190-4/13 $31.00 © 2013 IEEE

DOI 10.1109/ACPR.2013.110

611

The reminder of this paper is organized as follows. The related work is introduced in Section II. The proposed method is presented in Section III. It includes kernel clustering algorithm, kernel integration, and fast better kernel retrieval. Experimental results are presented in Section IV. Finally, we give a brief conclusion in Section V.

Fig.1. The blurred information is always hidden in the edge component.

II. RELATED WORK In this section, we briefly introduce the related methods

used in our proposed algorithm in the following subsections.

A. Blur kernel spectral estimation [7] According to Burton and Moorhead [10] pointed out, the

Fourier spectral response and natural scene images between a power law have the following relationship.

� �

2ˆ ,I �� � � (2)

where I is a natural image, I is Fourier transform of I and� denotes the frequency coordinates. Various studies suggest that 2� . They use this power law as the theoretical basis to consist a 2-D blur kernel spectral estimation.

� �� � � � � �

� � � � � �

� �

2

22

2

ˆ ˆ

ˆ ˆˆ

ˆ ,

d P BR l B r

l I r k r

c k r

� �

� �

� �

� �

� � �

�

� �

�

(3)

where d = [3, -32, 168, -672, 0, 672, -168, 32, -3]/840 which is a nine-point 1D differentiation filter, c� is a constant, r� is a unit vector in 2D with the orientation of � , and l d d� � , the sign denotes as signal mirroring. Moreover, the noise effect on the blur kernel estimation can be reduced by using a factor of 1/ n in noise terms N within Eq. (1). Hence, the noise can ignore in Eq. (3), and Equation 3 can be rewritten by Wiener-Khinchin theorem, expressed as

� �� �� � � � � � ,d P B P kR x c R x� ��� (4)

The first step of Goldstein blur kernel power spectrum is to compute ( )

0( ) limd P B

xf R x

���

��

� for every � . Estimating

2ˆ( )k � in Eq. (3) introduces a set of unknowns c� , which

are common to all values of 2k along the same angle� . This method consists of iterative EM-like procedure, and starts from the initial guess � �arg minxs f x� �� . Then, given the retrieved kernel, these values can be updated by setting Eq. (5). This algorithm would summarize in Alg. 1.

� � � � � �� �� �arg max 0.1 max ,x P k P ks R x R� �� � � (5)

B. Phase-Retrieval

In order to recover blur kernel from kernel spectral 2k into the spatial domain K, it requires estimating a phase component � �k � by phase retrieval algorithm. However, this algorithm cannot guarantee the results of each execution can get the same solution [12]. So after repeated execution M times, it will produce M blur kernels. Thus, it has to find the best kernel in M kernels. Therefore, a normalized sparsity measure proposed by [8] is usually used to evaluate every kernel and find the best kernel.

C. Kernel Retrieval As the previous step, it will generate M blur kernels by

M-times iteration with the normalized sparsity measure proposed by [8]. This method can evaluate the blur kernel’s deconvolution score. However, this score is used to analyze the symmetry of the blur kernel. The symmetry characteristic of blur kernel is obtained by calculated the normalized sparsity measure. It needs to calculate the normalized sparsity measure score for two times. Thus, this step is very time-consuming. In this step, the symmetry characteristic of kernel is very important factor that can determine whether this kernel has the reflection situation or not, and then do to modify. This procedure here we called is kernel retrieval. Alg. 2 would describe in detail.

D. Mean Structural Similarity[13] Mean structural similarity (MSSIM) derived from SSIM

proposed by Wang et al. [13] is used to evaluate the

Algorithm 1: Iterative kernel recovery. Input: blurry image B ; calculate � �d P Bf R

�� �� ;

set initial � �arg min xs f x� �� ; for i=1...m do estimate 2

k given s� ;

recover kernel k using phase retrieval given 2k ;

update� � � � � �� �� �arg max 0.1 max ,x P k P ks R x R

� �� � �

end Output: recovered blur kernel

612612

similarity measure between two signals. For overall quality measure of the entire image, the MSSIM index is more suitable than SSIM index. Therefore, they use a mean SSIM (MSSIM) index to evaluate the overall image quality.

1

1( , ) ( , )M

j jj

MSSIM X Y SSIM x yM �

� � (6)

where X and Y are the reference and the distorted images, respectively; jx and jy are the image contents at the jth local window; and M is the number of local windows of the image.

MSSIM is a method which is suitable for measuring the similarity between two images. MSSIM value is in the range [0, 1]. If MSSIM = 1, it means that the current image is exactly the same as the original image. Hence, we use this MSSIM value to cluster the blur kernel in this paper.

III. PROPOSED METHOD

In this section, we will introduce our deblurring method which includes kernel clustering algorithm, kernel integration, and the fast better kernel retrieval (FBKR) mechanism. Details of procedures are depicted in the following.

A. Kernel clustering algorithm We use MSSIM to construct a kernel clustering

algorithm. The reason why we don’t use conventional PSNR for kernel clustering algorithm is because PSNR doesn’t estimate image structure property. Based on the previous iterative phase retrieval algorithm, we can obtain M blur kernels by M-times iteration. In order to efficiently extract the best blur kernel and reduce the iteration times, here, we are going to classify M kernels sequence into n kernel clusters. The kernel clustering algorithm is depicted as follows.

Step 1: putting all the kernels in a queue, and taking the first kernel to compute MSSIM with the rest of kernels.

Step 2: If the MSSIM value is greater than or equal to 0.9, we will classify into the same category and remove it from kernel queue.

Step3: until the entire blur kernels are classified. Figure 2 is shown an example about this algorithm in detail.

B. Kernel integration and refined After clustering, we can get n kernel clusters, and each

cluster contains at least two or more of blur kernels. In order to find which one of clusters is the best blur kernel cluster, so we integrate all of kernel clusters into one blur kernel. First step is to count an average kernel value in each of blur kernel clusters corresponding to the coordinates, the average value corresponding to each cluster is expressed as

� �, ,1

1( , ) , , 0 , kernel size,in

avg k i kii

N x y N x y x yn �

� � �� (7)

where k and in denote number of clusters and the number of kernels belonging to the ith cluster, respectively. ,avg kN is the average kernel of the kth cluster. Second step is to refine ,avg kN , we calculate the probability of non-zero values in the coordinates for each blur kernel belonging to the same cluster. If this probability value is less than 50%, we set zero in this corresponding coordinate. Thus, we can obtain the refined ,avg kN value corresponding to the kth cluster.

C. Fast Best Kernel Retrieval(FBKR) In spite of [7] estimated a blur kernel to achieve the

blurred image, it is very time-consuming in kernel retrieval procedure. Based on this reason, we propose a fast best kernel retrieval algorithm using kernel clustering algorithm, kernel integration and normalized sparsity measure to reduce the computational complexity and times, at the same time, the deblurring image can maintain a good quality. Firstly, based on the result of the kernel clustering process, we can obtain n kernel clusters and ,avg kN average kernels. Next, we will do the normalized sparsity measure to find the

Step 1 Queue K1 K2 K3 K4 K5 K6 K7

K1 1 0.99 0.8 0.89 0.9 0.5 0.77

Group 1 K1 K2 K5 Step 2

Queue K3 K4 K6 K7

K3 1 0.83 0.9 0.81

Group 2 K3 K6

Step 3 Queue K4 K7

K4 1 0.99

Group 3 K4 K7

Fig.2. An example for clustering algorithm

Algorithm 2: Kernel Retrieval Input: n set of kernel Random select a small window from blurred image W for 1i n� do

� �� �

1 , ;

2 90( ,2), ;

i

i

score n W

score

NormalizedSparsityMeasure

NormalizedSparsityMeasu rotre n W

�

�

if � �1 2score score�

BestKernelList(i)= in ;

BestScoreList(i)= 1score ; else BestKernelList(i)= in ;

BestScoreList(i)= 2score ; end end [ , ] min( );Finalscore Index BestScoreList� Output: BestKernelList(Index);

613613

best kernel group and record its symmetry corresponding to the average kernel. Finally, we apply the normalized sparsity measures again to find one best kernel from the best kernel group. The FBKR procedure describes in the following in detail.

IV. EXPERIMENTAL RESULTS

In our experiments, the platform OS is used the Window 7 64-bit, CPU is i5 dual core 3.2GHz and 4G RAM. And the implementation software is used Matlab2010b and Visual Studio 2010 C#. In addition, we use SSIM to evaluate the quality of the reconstructed image. The blurred image data include ten different clear images and five different blur kernels. We blurred each of these images by Eq. (1) with Gaussian noise with� = 0.5%. In the parameter setting, we refer to Goldstein’s setup, the outerN is set 3 using in Alg. 1 and the phase-retrieval step is running 30 times. We used the non-blind deconvolution [9] to generate all of the deblurred images. The experimental results are compared our method and Goldstein and Fattal [7] for the reconstruction quality and the estimation speed of blur kernel.

Figure 3 shows the time-consuming of blur kernel estimation compared our method and Foldstein and Fattal’s [7] method. It is clear that our method is spent less time even a half of them. It is because our proposed fast kernel retrieval algorithm can efficiently reduce the normalized sparsity measure time.

About the evaluation of the reconstruction quality, owing to the initial phase of our FBKR algorithm is randomly generated, therefore, it may cause the different results under the same parameter settings. In order to efficiently evaluate the quality, in this paper, we adopt the average MSSIM values (Ave_MMSIM) computing by ten times of deblurring processes corresponding to the kernels to present the performance. Compared our method and Goldstein and Fattal’s method [7], the Ave_MSSIM value shown in Fig. 4 is very close. Hence, it is clear that our method not only efficiently enhance computational speed superior to [7] but the quality is also kept. Figures 5 and 6 show the deblurred results compared with [7].

V. CONCLUSION In this paper, we have introduced a fast algorithm to find

the best blur kernel after iterative phase retrieval algorithm. Our method is mainly used to blur kernel integration to reduce unnecessary calculations which can efficiently enhance the speed. As the experimental results shown, our proposed algorithm not only speeds up execution but also the quality of image blurring is maintained comparing with Goldstein and Fattal’s method [7]. In the future, we will concentrate our attention on the kernel estimation of non-global blurred image and the improvement of kernel estimation.

ACKNOWLEDGEMENT This work was supported in part by the National Science Council of Republic of China under Grant No. NSC 100-2628-E-150-003-MY2.

REFERENCES [1] Y. W. Tai, H. Du, M. S. Brown, and S. Lin, “Correction of spatially

varying image and video motion blur using a hybrid camera,” IEEE Tran. on Pattern Analysis and Machine Intelligence, vol. 32, no. 6, pp.1012-1027, 2010.

[2] W. Hu, J. Xue, and N. Zheng, “PSF estimation via gradient domain correlation,” IEEE Trans. on Image Processing, vol. 21, no. 1, pp. 386-392, 2012.

[3] H. Takeda and P. Milanfar, “Removing motion blur with space–tme processing,” IEEE Trans. on Image Processing, vol. 20, no. 10, pp. 2990-3000, 2011.

[4] Y. W. Tai, P. Tan, and M. S. Brown, “Richardson-Lucy deblurring for scenes under a projective motion path” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 33, no, 8, pp. 1603-1618, 2011.

[5] M. Dobeša , L. Machalab, and T. Furstc, “Blurred image restoration: A fast method of finding the motion length and angle,” ELSEVIER Journal on Digital Signal Processing, vol. 20, no. 6, pp. 1677-1686, 2010

[6] F. Zhang, G. Wang, J. Ye, Q. Liu, and X. Xiao, “A new approach to estimate image blur extent based on wavelet module maximum,” in Proc. of IEEE Conf. on ICISS, pp. 193-196, 2010.

[7] A. Goldstein and R. Fattal, “Blur-kernel estimation from spectral iIrregularities,” Lecture Notes in Computer Science, vol. 7576, pp. 622-635, 2012.

FBKR Algorithm 3 : Fast Best Kernel Retrieval

Input: n set of kernel and avgN

Random select a small window from blurred image W for 1 avgi N� do

� �� �

1 ( ), ;

2 90( ( ),2), ;

avg

avg

score N i W

score

NormalizedSparsityMeasure

NormalizedSparsityMeasure rot N i W

�

�

if � �1 2score score�

BestSetScoreList(i)= 1score ; Symmetry(i) = 0;

else BestSetScoreList(i)= 2score ; Symmetry(i) = 1;

end end [ , ] min( );score Index BestSetScoreList�

S =size(N(Index)); for 1j S� do

if(Symmetry(i) == 0) � �1 ( )( ), ; NormalizedSparsityMeasurescore N i j W�

BestKernelList(i)= ( )n i ; BestScoreList(i)= 1score ;

else � �2 90( ( )( ),2), ; avgNormalizedSparsityMeascore rotsu N i jre W�

BestKernelList(i)= ( )n i ; BestScoreList(i)= 2score ; end

end [ , ] min( );Finalscore Index BestScoreList� Output: BestKernelList(Index);

614614

[8] D. Krishnan, T. Tay and R. Fergus, “Blind deconvolution using a normalized sparsity measure”, in Proc. of IEEE Conf. on CVPR, pp. 223-240, 2011.

[9] D. Krishnan and R. Fergus, “ Fast image deconvolution using Hyper-Laplacian priors,” Neural Information Pro-cessing Systems, pp.1033-1041, 2009.

[10] G. J. Burton and I. R. Moorhead, “Color and spatial structure in natural scenes,” Applied Optics, vol. 26, pp. 157-170, 1987.

[11] D. Russell Luke, “Relaxed averaged alternating reflections for diffraction imaging”, Inverse Problems, vol. 21, pp. 37-50, 2005.

[12] Y M. Bruck, L. G. Sodin, “On the ambiguity of the image reconstruction problem”, Optics Communications, vol. 30, pp. 304-308,1979.

[13] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Tran. on Image Processing, vol. 13, no. 4, pp. 600-612, 2004.

Fig. 3. The time-consuming of blur kernel estimation.

Fig. 4. Comparison of deconvolution quality with our method and Goldstein and Fattal’s method [7].

(a) (b)

(c) (d)

Fig. 5. (a) Clear image. (b) Blurred image by blur kernel as shown in the bottom of (b). (C) Deblurred image by using our proposed method. (D) Deblurred image by using Goldstein and Fattal’s [7] method.

Fig.6. (a) Real blurred image. (b) Our deblurred image. (c) Result by using Goldstein and Fattal’s [7] method.

615615