high-quality computational imaging through simple lens
TRANSCRIPT
High-Quality Computational Imaging Through Simple Lenses
F. Heide1, M. Rouf1, M. Hullin1, B. Labitzke2, W. Heidrich1, A. Kolb2
1University of British Columbia, 2University of Siegen
(ACM Transactions on Graphics, 2013)
Presented by Monica Drăgan
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Correct for :● Geometric distortion● Spherical aberation● Chromatic aberation● Coma
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Correct for :● Geometric distortion● Spherical aberation● Chromatic aberation● Coma
expensive, large, heavy
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Correct for :● Geometric distortion● Spherical aberation● Chromatic aberation● Coma
expensive, large, heavy
Alternative
approach
to high-quality
photography
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Correct for :● Geometric distortion● Spherical aberation● Chromatic aberation● ComaCOM
PUTATIONALLY
Simple lenses:● Plano-convex● Biconvex● Achromatic doublets
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Blurred captured image
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Blurred captured image
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Corrected image
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Outline
● What are the challenges?
● Previous work
● Paper contribution
● Current approach
● Performance
● Applications
● Future work
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Outline
● What are the challenges?
● Previous work
● Paper contribution
● Current approach
● Performance
● Applications
● Future work
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Point spread function (PSF)
f/2.0 f/4.5
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Point spread function (PSF)
f/2.0 f/4.5
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Point spread function (PSF)
● Spatially large (50x50px)
● Spatial variation
● Wavelength dependent
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Image convolution
Blur
kernel
=Observed
blurred imageUnderlying
sharp image
*
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Deconvolution
Blur
kernel
=Observed
blurred imageUnderlying
sharp image
*-1
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Deconvolution
*-1 =
*-1 =
*-1 =
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Outline
● What are the challenges?
● Previous work
● Paper contribution
● Current approach
● Performance
● Applications
● Future work
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Previous work
Levin et al. '07
Idependent
deconvolution
on each
color channel
Schuler et al. '11
Deconvolution in YUV space
Cossairt & Nayar '10
Use chromatic
aberations to
increase DOF
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Outline
● What are the challenges?
● Previous work
● Paper contribution
● Current approach
● Performance
● Limitations
● Future work
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Paper contribution
● Cross-channel instead of channel-independent deconvolution
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Paper contribution
● Cross-channel instead of channel-independent deconvolution
Observations: ● hue changes are sparse and occure near the edges● edges appear in the same place in all channels
Blurred image Sharp image
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Paper contribution
● Cross-channel instead of channel-independent deconvolution
Severe ringing
Residual blur
Levins '07 Current approachBlurred image
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Paper contribution
● Cross-channel instead of channel-independent deconvolution
● Efficient convex optimization solver [Chambolle & Pock '11]
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Paper contribution
● Cross-channel instead of channel-independent deconvolution
● Efficient convex optimization solver [Chambolle & Pock '11]
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Paper contribution
● Cross-channel instead of channel-independent deconvolution
● Efficient convex optimization solver [Chambolle & Pock '11]
● Robust approach for per-channel PSF estimation
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Paper contribution
● Cross-channel instead of channel-independent deconvolution
● Efficient convex optimization solver [Chambolle & Pock '11]
● Robust approach for per-channel PSF estimation
White noise calibration pattern
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Outline
● What are the challenges?
● Previous work
● Paper contribution
● Current approach
● Performance
● Applications
● Future work
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Current approach
*-1 =
*-1 =
*-1 =
estimate
PSF
(lens specific)
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Current approach
*-1 =
*-1 =
*-1 =
formulate
the optimization
problem
estimate
PSF
(lens specific)
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
-*
Optimization problemLeast squares
data fitting
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Optimization problemSparse
image gradient
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Optimization problem
Cross-channel prior
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Least squares data fitting
Cross-channel priorSparse
image gradient
Optimization problem
Weighted contributions
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Least squares data fitting
Cross-channel priorSparse
image gradient
PROBLEM IS CONVEX
Optimization problem
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Least squares data fitting
Cross-channel priorSparse
image gradient
PROBLEM IS CONVEX
Optimization problem
Efficiently solvable by standard forward – backward splitting methods
[Chambolle & Pock '11]
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Optimization problem
Original Resulted
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Regularization for
low intensity areas
needed
Original
Optimization problem
Resulted
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Optimization problem
Original Resulted
Unscaled gradients
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Improved result
Optimization problem
Original Initial result
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Least squares
data fittingCross-channel prior
Regularization for
low intensity pixels
Optimization problemSparse
image gradient
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
PSF estimation
● PSF calibration – with white noise pattern● for a certain aperture
I J
f/2.0 f/4.5
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
PSF estimation
● PSF calibration – with white noise pattern● for a certain aperture
● for each tileI J
f/2.0 f/4.5
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
PSF estimation
● PSF calibration – with wite noise pattern● for a certain aperture● for each tile
I JB
* ? =
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
PSF estimation
● PSF calibration – with wite noise pattern● for a certain aperture● for each tile
I JB
* ? =
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
PSF estimation
● PSF calibration – with wite noise pattern● for a certain aperture● for each tile
● Non-blind deconvolution
Least squares
data fitting
I JB
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
PSF estimation
● PSF calibration – with wite noise pattern● for a certain aperture● for each tile
● Non-blind deconvolution
Least squares
data fitting
Energy
conservation
Gradient
total variation
I JB
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
PSF estimation
● PSF calibration – with wite noise pattern● for a certain aperture● for each tile
● Non-blind deconvolution
Efficiently solvable by standard
forward – backward splitting methods
[Chambolle & Pock '11]I JB
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
PSF estimation
● PSF calibration – with wite noise pattern● for a certain aperture● for each tile
● Non-blind deconvolution
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
PSF estimation
● Once per lens
● Accurate (two consecutive shots with different apertures)
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Outline
● What are the challenges?
● Previous work
● Tool's magic
● Current approach
● Performance
● Applications
● Future work
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Performance
● Outperforms other state-of-art methods
● Postprocessing image quality comparable to that of a compact camera (at f/4.5)
● Improves also images taken with compact cameras
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
OriginalDeblurred
LevinDeblurred Schuler
Current approach
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Outline
● What are the challenges?
● Previous work
● Tool's magic
● Current approach
● Performance
● Applications
● Future work
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Applications
● Deconvolution for multispectral cameras
● Remove residul blur in regular cameras
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Outline
● What are the challenges?
● Previous work
● Tool's magic
● Current approach
● Performance
● Applications
● Future work
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Future work
● Use drastically simpler lens design
● Optimize lenses to generate blur that is easier to remove
● Calibrate full depth-dependent PSFs
● Speed up the computation (distributed computing)● Running time: ~10-20 s for a 8MP image
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Thank you!
& special thanks to Marios Papas
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Regularization for
low intensity pixels
Efficiently solvable by standard forward – backward splittin methods
[Chambolle & Pock '11]
Least squares
data fittingCross-channel prior
1. Optimization problemSparse
image gradient
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Original
Original
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Deblurred
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014
Running time