high-quality computational imaging through simple lens

High-Quality Computational Imaging Through Simple Lenses

F. Heide1, M. Rouf1, M. Hullin1, B. Labitzke2, W. Heidrich1, A. Kolb2

1University of British Columbia, 2University of Siegen

(ACM Transactions on Graphics, 2013)

Presented by Monica Drăgan

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Correct for :● Geometric distortion● Spherical aberation● Chromatic aberation● Coma

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Correct for :● Geometric distortion● Spherical aberation● Chromatic aberation● Coma

expensive, large, heavy

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Correct for :● Geometric distortion● Spherical aberation● Chromatic aberation● Coma

expensive, large, heavy

Alternative

approach

to high-quality

photography

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Correct for :● Geometric distortion● Spherical aberation● Chromatic aberation● ComaCOM

PUTATIONALLY

Simple lenses:● Plano-convex● Biconvex● Achromatic doublets

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Blurred captured image

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Blurred captured image

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Corrected image

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Outline

● What are the challenges?

● Previous work

● Paper contribution

● Current approach

● Performance

● Applications

● Future work

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Outline

● What are the challenges?

● Previous work

● Paper contribution

● Current approach

● Performance

● Applications

● Future work

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Point spread function (PSF)

f/2.0 f/4.5

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Point spread function (PSF)

f/2.0 f/4.5

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Point spread function (PSF)

● Spatially large (50x50px)

● Spatial variation

● Wavelength dependent

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Image convolution

Blur

kernel

=Observed

blurred imageUnderlying

sharp image

*

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Deconvolution

Blur

kernel

=Observed

blurred imageUnderlying

sharp image

*-1

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Deconvolution

*-1 =

*-1 =

*-1 =

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Outline

● What are the challenges?

● Previous work

● Paper contribution

● Current approach

● Performance

● Applications

● Future work

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Previous work

Levin et al. '07

Idependent

deconvolution

on each

color channel

Schuler et al. '11

Deconvolution in YUV space

Cossairt & Nayar '10

Use chromatic

aberations to

increase DOF

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Outline

● What are the challenges?

● Previous work

● Paper contribution

● Current approach

● Performance

● Limitations

● Future work

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Paper contribution

● Cross-channel instead of channel-independent deconvolution

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Paper contribution

● Cross-channel instead of channel-independent deconvolution

Observations: ● hue changes are sparse and occure near the edges● edges appear in the same place in all channels

Blurred image Sharp image

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Paper contribution

● Cross-channel instead of channel-independent deconvolution

Severe ringing

Residual blur

Levins '07 Current approachBlurred image

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Paper contribution

● Cross-channel instead of channel-independent deconvolution

● Efficient convex optimization solver [Chambolle & Pock '11]

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Paper contribution

● Cross-channel instead of channel-independent deconvolution

● Efficient convex optimization solver [Chambolle & Pock '11]

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Paper contribution

● Cross-channel instead of channel-independent deconvolution

● Efficient convex optimization solver [Chambolle & Pock '11]

● Robust approach for per-channel PSF estimation

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Paper contribution

● Cross-channel instead of channel-independent deconvolution

● Efficient convex optimization solver [Chambolle & Pock '11]

● Robust approach for per-channel PSF estimation

White noise calibration pattern

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Outline

● What are the challenges?

● Previous work

● Paper contribution

● Current approach

● Performance

● Applications

● Future work

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Current approach

*-1 =

*-1 =

*-1 =

estimate

PSF

(lens specific)

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Current approach

*-1 =

*-1 =

*-1 =

formulate

the optimization

problem

estimate

PSF

(lens specific)

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

-*

Optimization problemLeast squares

data fitting

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Optimization problemSparse

image gradient

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Optimization problem

Cross-channel prior

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Least squares data fitting

Cross-channel priorSparse

image gradient

Optimization problem

Weighted contributions

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Least squares data fitting

Cross-channel priorSparse

image gradient

PROBLEM IS CONVEX

Optimization problem

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Least squares data fitting

Cross-channel priorSparse

image gradient

PROBLEM IS CONVEX

Optimization problem

Efficiently solvable by standard forward – backward splitting methods

[Chambolle & Pock '11]

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Optimization problem

Original Resulted

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Regularization for

low intensity areas

needed

Original

Optimization problem

Resulted

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Optimization problem

Original Resulted

Unscaled gradients

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Improved result

Optimization problem

Original Initial result

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Least squares

data fittingCross-channel prior

Regularization for

low intensity pixels

Optimization problemSparse

image gradient

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

PSF estimation

● PSF calibration – with white noise pattern● for a certain aperture

I J

f/2.0 f/4.5

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

PSF estimation

● PSF calibration – with white noise pattern● for a certain aperture

● for each tileI J

f/2.0 f/4.5

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

PSF estimation

● PSF calibration – with wite noise pattern● for a certain aperture● for each tile

I JB

* ? =

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

PSF estimation

● PSF calibration – with wite noise pattern● for a certain aperture● for each tile

I JB

* ? =

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

PSF estimation

● PSF calibration – with wite noise pattern● for a certain aperture● for each tile

● Non-blind deconvolution

Least squares

data fitting

I JB

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

PSF estimation

● PSF calibration – with wite noise pattern● for a certain aperture● for each tile

● Non-blind deconvolution

Least squares

data fitting

Energy

conservation

Gradient

total variation

I JB

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

PSF estimation

● PSF calibration – with wite noise pattern● for a certain aperture● for each tile

● Non-blind deconvolution

Efficiently solvable by standard

forward – backward splitting methods

[Chambolle & Pock '11]I JB

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

PSF estimation

● PSF calibration – with wite noise pattern● for a certain aperture● for each tile

● Non-blind deconvolution

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

PSF estimation

● Once per lens

● Accurate (two consecutive shots with different apertures)

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Outline

● What are the challenges?

● Previous work

● Tool's magic

● Current approach

● Performance

● Applications

● Future work

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Performance

● Outperforms other state-of-art methods

● Postprocessing image quality comparable to that of a compact camera (at f/4.5)

● Improves also images taken with compact cameras

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

OriginalDeblurred

LevinDeblurred Schuler

Current approach

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Outline

● What are the challenges?

● Previous work

● Tool's magic

● Current approach

● Performance

● Applications

● Future work

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Applications

● Deconvolution for multispectral cameras

● Remove residul blur in regular cameras

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Outline

● What are the challenges?

● Previous work

● Tool's magic

● Current approach

● Performance

● Applications

● Future work

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Future work

● Use drastically simpler lens design

● Optimize lenses to generate blur that is easier to remove

● Calibrate full depth-dependent PSFs

● Speed up the computation (distributed computing)● Running time: ~10-20 s for a 8MP image

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Thank you!

& special thanks to Marios Papas

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Regularization for

low intensity pixels

Efficiently solvable by standard forward – backward splittin methods

[Chambolle & Pock '11]

Least squares

data fittingCross-channel prior

1. Optimization problemSparse

image gradient

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Original

Original

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Deblurred

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Advanced Methods in Computer Graphics, SS2014Freitag, 4. April 2014

Running time