Adaptive Regularization of the NL-Means : Application to Image and Video Denoising陈金楚

IEEE TRANSACTION ON IMAGE PROCESSING ， VOL ， 23 ， NO,8 ， AUGUST 2014

Sutour, C. ; Deledalle, C.-A. ; Aujol, J.-F. “Adaptive regularization of the NL-means: Application to image and video denoising.“ IEEE Trans. Image Process., vol. 23, no. 8, pp. 3506 – 3521, Aug. 2014

Camille Sutour, Charles-Alban Deledalle, and Jean-François Aujol

A. ROF Model [1]

The usual model is the case of additive white Gaussian noise:

g f

The general problem in denoising is to recover the image based on the noised observation .

fg

2argmin ( )

N

TV

u

u u g TV u

ROF model [1] relies on the total variation (TV), hence forcing smoothness while preserving edges. The restored image is obtained by minimizing the following energy:

[1] L. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Phys. D, vol. 60, no. 1, pp. 259–268, 1992. (cited by 7418)

TVu

( ) ( )iiTV u u

(1)

(2)

(3)

Original image Horizontal Vertical

TV

0

90

45

135

[1, 1];

[1; 1];

[1,1; 1,1];

[1,1;1, 1];

o

o

o

o

h

h

h

h

Results

Sigma = 20Original ROF

Shortcoming:1 、 the textures tend to be overly smoothed ；2 、 the flat areas are approximated by a piecewise constant surface resulting in a staircasing effect ；3 、 the image suffers from losses of contrast.

B. Non-local Means [2-3]

[2]Buades, Antoni, Bartomeu Coll, and J-M. Morel. “A non-local algorithm for image denoising.” Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. Vol. 2. IEEE, 2005.(cited by 1792 )[3] Buades, Antoni, Bartomeu Coll, and Jean-Michel Morel. "A review of image denoising algorithms, with a new one." Multiscale Modeling & Simulation 4.2 (2005): 490-530. (cited by 1916)

Fig. Scheme of NL-means strategy. Similar pixel neighborhoods give a large weight, w(p,q1) and w(p,q2), while much different neighborhoods give a small weight w(p,q3).

( , ) ( )NLi

j

u i j g j

These weights are define as,2

22

( ) ( )1

( , ) ,( )

j ig N g N

hi j eZ i

where is the normalizing constant

2

22

( ) ( )

( )j ig N g N

h

j

Z i e

( )Z i

(1)

(2)

(3)

Shortcoming

Fig. Illustration of the defaults of the NL-means: the rare patch effect (red circle) can be observed around the head and the camera, while the patch jittering effect (blue circles) can be observed on the background.

1 、 On singular structures the algorithm might fail to find enough similar patches and thus performs insufficient denoising. This is referred to as the rare patch effect.2 、 False detections. It can result in averaging several pixel values that do not truly belong to the same underlying structure, creating an over-smoothing sometimes referred to as the patch jittering blur effect.

C. Non-local TV [4]

[4]Gilboa G, Osher S. Nonlocal operators with applications to image processing[J]. Multiscale Modeling & Simulation, 2008, 7(3): 1005-1028.(cited by 437)

Define a nonlocal gradient as follows:

, ,( ) ( )i j i j i ju u u

where is the weight that measures the similarity between pixels i and j . This leads to the definition of a nonlocal framework, including the nonlocal ROF model:

,i j

with2

,( ) ( )i i j i ji i j

u u u

(1)

(2)

(3)

Characteristics : free of the staircasing effect but it is still subject to the rare patch effect.

2argmin ( )NLTV

iu i

u u g u

Contributions

1 、 Adaptive Regularization of the NL-Means(combine TV with NL-means) reduce the patch jittering blur effect correct the rare patch effects without introducing over-smoothing, staircasing or contrast losses inherent to the non-adaptive TV minimization

2 、 Propose a model that adapts to different noise modelsGaussian CasePoisson CaseGamma Case

3 、 Application to Video Denoising with 3D patches

A. Dejittering of the NL-means(NLDJ)

With reference to the literature [5], [6], we locally perform a convex combination between the nonlocal estimation and the noisy data g, according to the following formula:

NLu

(1 )NLDJ NLi i i i iu u g

Where is a confidence index defined by:i

^2 2

2

2 2 ^2 2 2

( ) ( )( )

( ) ( )( ) ( ) ( )

NLnoise

i isignali

i signal noise NLi i noise noise

i i i

[5]Lee, Jong-Sen. "Refined filtering of image noise using local statistics." Computer graphics and image processing 15.4 (1981): 380-389. (cited by 585)[6] Kuan D T, Sawchuk A A, Strand T C, et al. Adaptive noise smoothing filter for images with signal-dependent noise[J]. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 1985 (2): 165-177. (cited by 1055)

^2 2 2

, ,( ) ( ) ( )NL

NL NLi i j j i j j

j j

g g

(1)

(2)

(3)with

The solution can be rewritten as the following weighted sum

NLDJu

,NLDJ NLDJi i j j

j

u g

where

, , ,(1 )NLDJ NLi j i i j i i j

where , 1i j if ,i j 0 otherwise.

The residual variance of the dejittered solution can be approached at pixel i by:

NLDJu

^2 2 2

,( ) [ ( ) ]( )residual

NLDJ noisei i j i

j

(1)

(2)

(3)

B. Regularization of the NL-Means(R-NL)

2,argmin ( ) ( )

N

R NLi i j i j

u i j

u u g TV u

2argmin ( ) ( )N

R NL NLi i i

u i

u u u TV u

The proposed model combines both the NL-means and the TV minimization:

(1)

(2)

2argmin ( ) ( )N

R NL NLi i i

u i

u u u TV u

2argmin ( )NLTV

iu i

u u g u

How to set the regularization parameter

Regularization parameter

^2 2 2

,( ) [ ( ) ]( )residual

NLDJ noisei i j i

j

the lower, the better!!!

What's the relationship between and

^ residual

2argmin ( ) ( )N

R NL NLi i i

u i

u u u TV u

^

1 2 1/2,( ) ( )

residual

i

i i jnoiseji

R-NL for the exponential family

Both TV and NL-means are robust to different kind of noises, the proposed model can then be extended to other types of (uncorrelated) noise with a weighted data fidelity of the form [8]

, log ( | )i j i ip g u

A probability law belongs to the exponential family [7] if it can be written under the following form:

( ( ) | ) ( ) exp( ( ) ( ) ( ))p T g u c g u T g A u

where c, T , η and A are known functions. The extended model is then the following:

,argmin [ ( ) ( ) ( )] ( )N

R NLi i j i i i

u i j

u A u u T g TV u

[7]Collins M, Dasgupta S, Schapire R E. A generalization of principal components analysis to the exponential family[C]//Advances in neural information processing systems. 2001: 617-624.(cited by 249)[8] Polzehl, Jörg, and Vladimir Spokoiny. "Propagation-separation approach for local likelihood estimation." Probability Theory and Related Fields 135.3 (2006): 335-362.(cited by 125)

(1)

(2)

(3)

A. Gaussian Case

As in the Gaussian case, it can be reformulated with a weighted NL-means based fidelity term:

argmin log ( | ) ( )N

R NL NLi i i

u i

u p u u TV u

2 2( )noisei

2

2

( )argmin ( )

2N

NLi i

iu i

u uTV u

where is the weighted log-likelihood

(1)

(2)

In this case, solving (1) is equivalent to solving

The expected nonlocal variance involved in the dejittering step is chosen as:

[9]Chambolle, Antonin, and Thomas Pock. "A first-order primal-dual algorithm for convex problems with applications to imaging." Journal of Mathematical Imaging and Vision 40.1 (2011): 120-145.(cited by 775)

B. Poisson Case

The negative log-likelihood of 0u for an observed intensity g is given by

The expected nonlocal variance involved in the dejittering step is chosen as:

where Q is the non-negative integer.

The solution of the NL-means and the adaptive regularization parameters can then be computed accordingly and the variational problem becomes:

i

(1)

(2)

C. Gamma Case

The negative log-likelihood of 0u for an observed intensity g is given by

where L is the “number of looks” that sets the level of the noise.

The expected nonlocal variance involved in the dejittering step is chosen as:

2 2( ) ( ) /noise NLi iu L

The solution of the NL-means and the adaptive regularization parameters can then be computed accordingly and the variational problem becomes:

i

(1)

(2)

Results

A. Gaussian Case

B. Poisson Case

C. Gamma Case

References [1] L. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Phys. D, vol. 60, no. 1, pp. 259–268, 1992. (cited by 7418)[2]Buades, Antoni, Bartomeu Coll, and J-M. Morel. “A non-local algorithm for image denoising.” Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. Vol. 2. IEEE, 2005.(cited by 1792 )[3] Buades, Antoni, Bartomeu Coll, and Jean-Michel Morel. "A review of image denoising algorithms, with a new one." Multiscale Modeling & Simulation 4.2 (2005): 490-530. (cited by 1916)[4]Gilboa G, Osher S. Nonlocal operators with applications to image processing[J]. Multiscale Modeling & Simulation, 2008, 7(3): 1005-1028.(cited by 437)[5]Lee, Jong-Sen. "Refined filtering of image noise using local statistics." Computer graphics and image processing 15.4 (1981): 380-389. (cited by 585)[6] Kuan D T, Sawchuk A A, Strand T C, et al. Adaptive noise smoothing filter for images with signal-dependent noise[J]. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 1985 (2): 165-177. (cited by 1055)[7]Collins M, Dasgupta S, Schapire R E. A generalization of principal components analysis to the exponential family[C]//Advances in neural information processing systems. 2001: 617-624.(cited by 249)[8] Polzehl, Jörg, and Vladimir Spokoiny. "Propagation-separation approach for local likelihood estimation." Probability Theory and Related Fields 135.3 (2006): 335-362.(cited by 125)[9]Chambolle, Antonin, and Thomas Pock. "A first-order primal-dual algorithm for convex problems with applications to imaging." Journal of Mathematical Imaging and Vision 40.1 (2011): 120-145.(cited by 775)[10] Combettes, Patrick L., and Jean-Christophe Pesquet. "Proximal splitting methods in signal processing." Fixed-point algorithms for inverse problems in science and engineering. Springer New York, 2011. 185-212.(cited by 539)

Thank you