200081003 friday food@ibbt
DESCRIPTION
Ghent UniversityImage Processing and Interpretation GroupAleksandra PizuricaAdvances and challenges in image andvideo restorationTRANSCRIPT
Ghent University
Image Processing and Interpretation Group
Aleksandra Pizurica
Advances and challenges in image and video restoration
2Image and video restorationIBBT Friday food talk, October 3, 2008
Median filter: Reduction of impulse noise
median over 3x3
Median filter removes isolated noise peaks, without blurring the image
impulse noise
3Image and video restorationIBBT Friday food talk, October 3, 2008
Median filter and reduction of white noise
median over 3x3
For not-isolated noise peaks (e.g., white Gaussian noise) median filter is not very efficient.
original
4Image and video restorationIBBT Friday food talk, October 3, 2008
Why is denoising important
Not only visual enhancement, but also: automatic processing is facilitated!
original denoised
Example: edge detection
5Image and video restorationIBBT Friday food talk, October 3, 2008
Image restoration example
State of the art image restoration methods iterate wavelet domain denoising and
Fourier domain deconvolution. From now on we focus on the denoising step
©Max Planck Institute for Biophysical Chemistry
[F. Rooms et al; Journal of Microscopy 2005]
6Image and video restorationIBBT Friday food talk, October 3, 2008
Overview
• Wavelet domain image restoration
• Gain from using other wavelet-like representations
• Medical applications: MRI, CT, OCT
• On noise and blur estimation
• Video denoising and advances in 3D video
7Image and video restorationIBBT Friday food talk, October 3, 2008
highpass
lowpass
wavelet coefficients
scaling coefficients
s j
2
2
w j+1
h
g
s j+1
2w j+2
h
g
s j+1
2
2
2
w j+3
h
g
s j+3
DWT algorithm: a filter bank iterated on the lowpass output
Discrete Wavelet Transform (DWT)
8Image and video restorationIBBT Friday food talk, October 3, 2008
Choosing a wavelet: Nv, support size K, symmetry
Nv - number of vanishing moments: ,0)( =ψ∫∞
∞−
k10 −≤≤ vNkdttt
Daubechies wavelets dbNv:
K 2Nv-1≥A tradeoff:
0 5 10 15-0.5
0
0.5
1
1.5
-5 0 5-1
-0.5
0
0.5
1
1.5Symmlets (Daubechies)
ϕ ψsym8
1.5 2
-0.5 0 0.5 1 1.50
0.5
1
1.5
-1.5 -1 -0.5 0 0.5-1.5
-1
-0.5
0
0.5
1
1.5
db1
ϕ ψ
0 1 2 3-0.5
0
0.5
1
-1 0 1 2-2
-1
0
1
db2
ϕ ψ
0 5 10 15-0.5
0
0.5
1
1.5
-5 0 5
-1
-0.5
0
0.5
1
db8
ϕ ψ-4 -2 0 2 4
-1
-0.5
0
0.5
1
-5 00
0.2
0.4
0.6
0.8
5
-5 0 5
-0.5
0
0.5
1
1.5
2
-4 -2 0 2 4
-2
-1
0
1
2
Biorthogonal wavelets
ϕ ψ
ϕ~ ψ~K=15
K=3
K=1
9Image and video restorationIBBT Friday food talk, October 3, 2008
DETAIL IMAGESwavelet coefficients
APPROXIMATIONscaling coefficients
Wavelet coefficient values
0
Peaks indicate image edges
Two dimensional DWT
10Image and video restorationIBBT Friday food talk, October 3, 2008
Noise in the wavelet domain
Wavelet coefficient values
Peaks indicate image edges
0
Noise-free reference
11Image and video restorationIBBT Friday food talk, October 3, 2008
Generalized Laplacian (generalized Gaussian) distribution
noise-free histogram f(y)=Aexp(- |y /s |ν )
s: scale parameter
ν: shape parameter
)10( ≤≤ν
�Parameters accurately estimated from a signal corrupted by additive white Gaussian noise
noisy
Marginal priors: Generalized Laplacian
Often yields complicated expressions
Extension to higher dimensions (joint histograms) difficult
12Image and video restorationIBBT Friday food talk, October 3, 2008
Gaussian Scale Mixture (GSM) models
z: mixture variable, random multiplier
= uzy
wavelet coefficient Gaussian random variable
u
f(u)f(y)
y
Efficient for modelling joint histograms of the neighboring wavelet coefficients
A state-of the art denoiser for many years BLS-GSM: Bayesian Least Squares estimator using GSM prior [Portilla et al, IEEE TIP’03]
∫∞
∞−= dzzfzzEE )(),|()|( xyxy
xCC
Cxy
nu
u
+=
z
zzE ),|(
nunyx +=+= z
noise:, nu CC signal and noise
covariances
13Image and video restorationIBBT Friday food talk, October 3, 2008
Locally adaptive denoising ProbShrink
yl
LSAI zl
LSAI
OBSERVATION
ESTIMATE
yyzyHPηξµ
ηξµβ
+==
1),|(ˆ
1
)|(
)|(
0
1
Hyf
Hyf=η
)|(
)|(
0
1
Hzf
Hzf=ξ )(
)(
0
1
HP
HP=µ
zy
f(y|H1)
f(y|H0)f(z|H0)
f(z|H1) P(H1)
P(H0)
LSAInoisy coefficient subband statistics
,ny += β
[Pizurica&Philips, IEEE TIP 2006]LSAI – Local Spatial Activity Indicator
H1 signal of interest present
H0 signal of interest absent
14Image and video restorationIBBT Friday food talk, October 3, 2008
Locally adaptive denoising: ProbShrink…
[Pizurica&Philips, IEEE TIP 2006]
15Image and video restorationIBBT Friday food talk, October 3, 2008
ProbShrink for correlated noise…
50 100 150
20
40
60
80
100
120
140
160
180
local window
23
22
X
X
vector of coefficients
X22 X23
X22
X23
X22
H1
H0
H1
[B. Goosens, A. Pizurica, W. Philips; IEEE TIP 2008, in press]
16Image and video restorationIBBT Friday food talk, October 3, 2008
… ProbShrink for correlated noise
[B. Goosens, A. Pizurica, W. Philips; IEEE TIP 2008, in press]
17Image and video restorationIBBT Friday food talk, October 3, 2008
Denoising by singularity detection
w1 wavelet coefficients
w2
w3
w1
Input signal Rate of increase of the
modulus of the wavelet
transform across scales is
proportional to thelocal Lipschitz regularity
[Mallat&Zhong, IEEE IT 1992]
18Image and video restorationIBBT Friday food talk, October 3, 2008
Statistics: magnitude and the rate of increase
ACR-4 -2 0 2 4 6
0
0.2
0.4
0.6
0.8
1
scale
cone of influence
edgesnoise
-50 0 50 100 150 200 250 3000
0.01
0.02
0.03
0.04
0.05
noisy magnitude
edges
noise
ACR
Noise standard deviation=25.5
Magnitude
-3 -2 -1 0 1 2 30
50
100
150
xl= 1
xl= 0noise
edges
Average Cone Ratio – an estimate
of the local Lipschitz exponent –measures the rate of increase of
the coefficients across the scales
[A. Pizurica et al; IEEE TIP 2002]
19Image and video restorationIBBT Friday food talk, October 3, 2008
Inter- and intrascale dependencies
• Bivariate models• Hidden Markov Tree models• Markov Random Field models
20Image and video restorationIBBT Friday food talk, October 3, 2008
Statistical modeling: MRF models
positive potential
negative potential
Example: penalize isolated peaks
−= ∑
∈ςCCV
ZP )(exp
1)( xx
clique potentials
cliquesneighborhoodx0
Prior model
P(x)
xMAP
21Image and video restorationIBBT Friday food talk, October 3, 2008
Statistical modeling: MRF models
cliquesneighborhoodx0
Prior model
P(x)
xMAP
Initial edges Iteration 3Iteration 2Iteration 1
22Image and video restorationIBBT Friday food talk, October 3, 2008
OriginalGamma MAP filter
MRF based wavelet denoising
wavelet filter
[A. Pizurica et al; ICIP 2001]
23Image and video restorationIBBT Friday food talk, October 3, 2008
GSM in non-overlapping blocks • Ignores non-local correlations
• Block size?
[Guerrero-Colon et al,
IEEE TIP,08]
Spatially variant GSM
(SVGSM)
Computationally expensive
[Portilla et al, Spie2008]
Mixture of GSM
(MGSM)
MRF models
Fields of GSM[Liu and Simoncelli,
PAMI’08]
Field of Experts (FoE)
[Roth and Black, CVPR’05]
[Tappen, Adelson, Freeman,
CVPR’07, CVPR’08]
Mixture of projected GSM
(MPGSM)
[Goossens, in review TIP 2008]Dimension reduction
in MGSM
Current trends in Bayesian wavelet denoising
?
Gaussian Scale Mixture (GSM) model
[Portilla et al, IEEE TIP 2003]
Shortcoming: assumes the same but scaled covariance for the whole subband
24Image and video restorationIBBT Friday food talk, October 3, 2008
Overview
• Wavelet domain image restoration
• Gain from using other wavelet-like representations
• Medical applications: MRI, CT, OCT
• On noise and blur estimation
• Video denoising and advances in 3D video
25Image and video restorationIBBT Friday food talk, October 3, 2008
Why other multiresolution representations
Classical wavelets are well suited for point-like singularities,
but not for curvilinear singularities in images
• Poor orientation selectivity; no difference between 45 and -45o
• Checkerboard pattern � appears also as an artifact in denoising
Many wavelet-like representations with a better orientation selectivity: complex wavelets [Kingsbury, Selesnick] , steerable pyramids [Freeman, Adelson], curvelets [Donoho, Candes], contourlets [Do, Vetterli], …
An example of wavelet base functions
26Image and video restorationIBBT Friday food talk, October 3, 2008
Curvelets: specific tiling of the frequency plane:
localized + directional
Curvelet-domain image denoising…
27Image and video restorationIBBT Friday food talk, October 3, 2008
Curvelet Hard Thresholding
PSNR=29.02dBPSNR=22.16dB
Noisy ImageWavelet ProbShrink
PSNR=29.50dB
…Curvelet domain image denoising…
PSNR=30.43dB
Curvelet ProbShrink
[L. Tessens, A. Pizurica, W. Philips, J Electr Imag 2008 in press]
28Image and video restorationIBBT Friday food talk, October 3, 2008
…Curvelet domain image denoising…
Results
Wavelet ProbShrink Curvelet ProbShrink
29Image and video restorationIBBT Friday food talk, October 3, 2008
Wavelet ProbShrink
…Curvelet domain image denoisingCurvelet ProbShrink
30Image and video restorationIBBT Friday food talk, October 3, 2008
Overview
• Wavelet domain image restoration
• Gain from using other wavelet-like representations
• Medical applications: MRI, CT, OCT
• On noise and blur estimation
• Video denoising and advances in 3D video
31Image and video restorationIBBT Friday food talk, October 3, 2008
MRI denoising: signal dependent noise
high SNR ( f =f1 )
f1
low SNR (f=0)
m
p(m)
noise-free f
noisy m
ma
gn
itud
e
co
ntr
ast
SNR
32Image and video restorationIBBT Friday food talk, October 3, 2008
Step 1: Bias removal
Square magnitude MRI image – after squaring constant bias, proportional
to noise standard deviation.
For better results: square root the result before denoising!
MRI denosing: algorithm
[Pizurica et al IEEE TMI 2003]
T?
Coarser, processed detail
A noisy detailMask
p(z|H1)
p(z|H0) histograms( )log
p(z|H0)
p(z|H1)
Step 2: Denoising (coarse-to-fine, empirical density estimation)
33Image and video restorationIBBT Friday food talk, October 3, 2008
Noisy image Denoised image Ground truth
MRI denosing: some results
34Image and video restorationIBBT Friday food talk, October 3, 2008
3D MRI volume denoising
using 3D dual-tree complex wavelet transform
[J. Aelterman et al, EUSIPCO 2008]
35Image and video restorationIBBT Friday food talk, October 3, 2008
Denoising low-dose CT images
• Reducing radiation dose increases noise level
• Can we use denoising on low dose CT to obtain the same diagnostic quality as in a higher dose CT image? [IBBT Ica4dt project]
• Difficulties:
- non-stationary correlated noise
- Streak artefacts
- How to estimate noise
36Image and video restorationIBBT Friday food talk, October 3, 2008
Denoising algorithm
wavelet
transform
(WT)
Vector
ProbShrink
Inverse wavelet
transform
(IWT)
(Dual-tree
complex)
segmentation
H1
H0
H1
[B. Goossens et al, EMBS 2007]
37Image and video restorationIBBT Friday food talk, October 3, 2008
Watershed segmentation
…Results
denoised
38Image and video restorationIBBT Friday food talk, October 3, 2008
Qualitative Validation CT: psycho-visual experiment
…
+++
--
++
++
+
++
+
+
+
+++
++
---
+
+
--
+
-
-
-
+++
++ --+ Versatile Probshrink
+++
++
+
+++
Structure
---+++ -Wavelet GSM Filter
--++ --Curvelet Filter
-+ --Versatile Probshrink2
+++---+++ Low-dose CT
QualityNoiseBlurAbdomen/Lung
The abdomen image judged better than the original by radiologists!
39Image and video restorationIBBT Friday food talk, October 3, 2008
Optical Coherence Tomography (OCT) images
2D signal
[IBBT Ica4dt project, with AGFA Healthcare]
OCT – “echography with light”
Noise: speckle similar to that in radar and ultrasound images
3D OCT data
40Image and video restorationIBBT Friday food talk, October 3, 2008
Denoising OCT images
A developed 3D OCT denoiser combines• wavelet domain speckle filter• motion compensated video denoising method
3D OCT data
Video denoising
coarse-to-fine
processing
Locally adaptive denoising
0 50 100 150 200 2500
0.005
0.01
0.015
0.02
0.025
0.03
0.035
Image:g120406breastseconformolnogelLR0050 Detail:Dx1 Parameter:b=10.3752
likelih
ood p
(m|1
)
magnitude m
Gamma, b=10.3752
0 50 100 1500
0.05
0.1
0.15
0.2
Image:g120406breastseconformolnogelLR0050 Detail:Dx2 Parameter:a=4.8348
likelih
ood p
(m|0
)
magnitude m
Laplace, a=4.8348
Signal and noise statistics
[IBBT Ica4dt project]
41Image and video restorationIBBT Friday food talk, October 3, 2008
LeeRKTOur methodNoisy Image SATGTF
Results and evaluation of OCT denoising
42Image and video restorationIBBT Friday food talk, October 3, 2008
Results and evaluation of OCT denoising
Noisy LeeSATOur method
[Pizurica et al; CMIR 2008 in press]
43Image and video restorationIBBT Friday food talk, October 3, 2008
Results and evaluation of OCT denoising
input Our method (2D version)BLS-GSM
[Pizurica et al; CMIR 2008 in press]
44Image and video restorationIBBT Friday food talk, October 3, 2008
Overview
• Wavelet domain image restoration
• Gain from using other wavelet-like representations
• Medical applications: MRI, CT, OCT
• On noise and blur estimation
• Video denoising and advances in 3D video
45Image and video restorationIBBT Friday food talk, October 3, 2008
Block-based
Search for blocks of nearly uniform intensity
Wavelet basedGradient distribution based
HH1
HL1
LH1usethis part
Noise variance estimation
or compensate forthe peak shift
noise (Rayleigh distr.)
σσσσ
Smoothing based
signal+noise
smooth
estimateσ
Median{|HH1|}
0.6745=σ̂
46Image and video restorationIBBT Friday food talk, October 3, 2008
Blur estimation using wavelet coefficients
Original image Blurred image-4 -2 0 2 4 6 8
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
original
blurred
ACR 1-2
ACR - Average Cone Ratio – an estimate of the local Lipschitz exponent –
measures the rate of increase of the coefficients across the scales
• A well known approach: kurtosis of the wavelet coefficient histogram
• An alternative: examine the propagation of the wavelet coefficients across
the scales
47Image and video restorationIBBT Friday food talk, October 3, 2008
blur
-1 0 1 2 3 4 50
1000
2000
3000
4000
5000
6000
ACR 2-4
reference
blur 3
blur 5
blur 7
blur 0 + noise
blur 3 + noise
blur 5 + noise
blur 7 + noise
Different colors -
different levels of blur
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 23000
3500
4000
4500
5000
5500
6000
ACR 2-4
referenceblur 3blur 5blur 7blur 0 + noiseblur 3 + noiseblur 5 + noiseblur 7 + noise
solid – noise free
dashed – noisy
Blur estimation using wavelet coefficients
48Image and video restorationIBBT Friday food talk, October 3, 2008
Overview
• Wavelet domain image restoration
• Gain from using other wavelet-like representations
• Medical applications: MRI, CT, OCT
• On noise and blur estimation
• Video denoising and advances in 3D video
49Image and video restorationIBBT Friday food talk, October 3, 2008
[V. Zlokolica, A. Pizurica, W. Philips; IEEE TCSVT 2006]
Noise Estimation
Time delay
Adaptive SpatialFiltering
2D WaveletTransform
Inverse 2D WaveletTransform
Input NoisyFrame
DenoisedFrame
Adaptive SpatialFiltering
Motion Estimation
Recursive Temporal Filtering
Time delay
Video denoising
50Image and video restorationIBBT Friday food talk, October 3, 2008
center ofthe motion block
motion direction(smaller amplitude)
motion direction(larger amplitude)
Video denoising: motion estimation…
Accurate motion estimation is essential for video denoising.Also important: reliability of the estimated motion at each point
51Image and video restorationIBBT Friday food talk, October 3, 2008
…Motion compensated video denoising
[V. Zlokolica, A. Pizurica, W. Philips; IEEE TCSVT 2006]
Further development currently within IBBT project ISYSS
52Image and video restorationIBBT Friday food talk, October 3, 2008
Reusing motion estimator from video codecs
• Motion estimators from video codecs tolerate errors � cannot be directly used in denoising
• Can we still use them with some postprocessing? The core of our approach:
• Motion field refinement step
• Reliability to motion estimates controls the recursive filter
• Competitive with state-of-the art video denoisers
[LJ. Jovanov et al; IEEE TCSVT 2008, in press]
53Image and video restorationIBBT Friday food talk, October 3, 2008
[Balster; TCSVT 2006] [Jovanov; TCSVT 2008]
Reusing motion estimator from video codecs
noise-free input
54Image and video restorationIBBT Friday food talk, October 3, 2008
Denoising and outlier removal in 3D video
Time-of-flight camera �records simultaneously luminance and depth information
The biggest errors in the depth measurement are induced by strong ambient light �� The measured distance is much smaller than the true distance)
Degradations in the depth image: noise, and outliers (similar to impulse noise but in bursts)
3D reconstructions using “surf” in Matlab
55Image and video restorationIBBT Friday food talk, October 3, 2008
Noisy 3D video sequence (luminance and depth)
Luminance image contains much less noise
Luminance and depth images are correlated
� Use the luminance information for denoising depth data
56Image and video restorationIBBT Friday food talk, October 3, 2008
Denoised luminance and depth
57Image and video restorationIBBT Friday food talk, October 3, 2008
Acknowledgements
Thanks to my colleagues for their contributions
• Vladimir Zlokolica (video denoising)
• Bart Goossens (removal of correlated noise)
• Ljubomir Jovanov (video, 3D video, OCT)
• Linda Tessens (curvelets)
• Jan Aelterman (MRI denoising)
• Filip Rooms (deblurring)
• Ewout Vansteenkiste (quality evaluation CT)
Related material available at: http://telin.ugent.be/~sanja