study dappled photography
DESCRIPTION
study of "Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing"TRANSCRIPT
![Page 1: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/1.jpg)
Dappled Photography: Mask Enhanced Cameras forHeterodyned Light Fields and Coded Aperture Refocusing
Ashok Veeraraghavan, Ramesh Raskar, Amit AgrawalMitsubishi Electric Research Labs (MERL), Cambridge, MA
Ankit Mohan, Jack TumblinNorthwestern University, Evanston, IL
study
![Page 2: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/2.jpg)
Abstract
• Heterodynes Light Field camera
• Coded Aperture camera
A theoretical framework for modulating 4D light fields using a mask between lens and sensors
![Page 3: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/3.jpg)
Abstract
• Heterodynes Light Field camera
1. Add a high-frequency mask between Lens and Sensors
2. 4D light field - sense different rays from lens (u,v) in a sensor position (x,y)
3. Re-arrange light field
• Low resolution with different focus settings
• Full resolution in-focus
A theoretical framework for modulating 4D light fields using a mask between lens and sensors
![Page 4: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/4.jpg)
Abstract
• Heterodynes Light Field camera
1. Add a high-frequency mask between Lens and Sensors
2. 4D light field - sense different rays from lens (u,v) in a sensor position (x,y)
3. Re-arrange light field
• Full resolution in-focus
• Low resolution with different focus settings
• Coded Aperture camera 1. Replace the Aperture
with a coded mask
2. A broadband mask enhance refocus at full resolution for Lambertian scene
A theoretical framework for modulating 4D light fields using a mask between lens and sensors
4D light field L(u,v,x,y)
![Page 5: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/5.jpg)
Abstract
• Heterodynes Light Field camera
1. Add a high-frequency mask between Lens and Sensors
2. 4D light field - sense different rays from lens (u,v) in a sensor position (x,y)
3. Rearrange light field
• Low resolution with different focus settings
• Full resolution in-focus
A theoretical framework for modulating 4D light fields using a mask between lens and sensors
![Page 6: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/6.jpg)
Abstract
• Heterodynes Light Field camera
1. Add a high-frequency mask between Lens and Sensors
2. 4D light field - sense different rays from lens (u,v) in a sensor position (x,y)
3. Re-arrange light field
• Low resolution with different focus settings
• Full resolution in-focus
• Coded Aperture camera 1. Replace the Aperture
with a coded mask
2. A broadband mask enhance refocus at full resolution for Lambertian scene
3. Refocusing partial
A theoretical framework for modulating 4D light fields using a mask between lens and sensors
![Page 7: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/7.jpg)
Heterodynes Light Field camera
![Page 8: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/8.jpg)
Heterodynes Light Field camera
• Based on modulation theorem in 4D frequency domain – mask carries rays
= rays x mask
F MaskRays
![Page 9: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/9.jpg)
Modulation Theorem
• [Oppenheim et al. 99]http://en.wikipedia.org/wiki/Amplitude_modulation
![Page 10: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/10.jpg)
Heterodynes Light Field camera
• Based on modulation theorem in 4D frequency domain – mask carries rays
= rays x mask
α depends on (d,v)
F MaskRays
good mask !
A poor mask blends the rays ! A good mask carriers the rays !
rays
![Page 11: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/11.jpg)
Heterodynes Light Field camera
• Based on modulation theorem in 4D frequency domain – mask carries rays
= rays x mask
F MaskRays
![Page 12: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/12.jpg)
Heterodynes Light Field camera
• Based on modulation theorem in 4D frequency domain – mask carries rays
• recover the light field by rearranging the tiles of 2D Fourier transfer into 4D plane to get the full resolution image information for the in-focus parts of the scene
F-1
= rays x mask
Rearrange
F MaskRays
![Page 13: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/13.jpg)
Heterodynes Light Field camera
• Based on modulation theorem in 4D frequency domain – mask carries rays
• recover the light field by rearranging the tiles of 2D Fourier transfer into 4D plane to get the full resolution image information for the in-focus parts of the scene
• A raw sensor holds a modulated 4D light filed
= rays x mask
Raw sensor (modulate 4D light field
data)
In-focus at full resolution (demodulated)
![Page 14: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/14.jpg)
Optical Heterodyning
Baseband Audio Signal
Receiver: DemodulationHigh Freq Carrier 100.1 MHz
ReferenceCarrier
Incoming Signal
Photographic Signal
(Light Field)
Carrier Incident Modulated
SignalReference
Carrier
Main LensObject Mask Sensor
RecoveredLight Field
Software Demodulation
99 MHz
![Page 15: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/15.jpg)
Coded Aperture camera
• Base on Convolution• Aperture as a Modulator
▫ sinc function depends on θ
▫ Pinhole camera has a very very broadband modulator
▫ Design broadband mask
= rays x mask
![Page 16: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/16.jpg)
Outline
•Introduction•Related Work•Theory & Framework•Heterodyne Light Field Camera•Encoded Blur Camera•Implements & Analysis•Contributions & Future Work
![Page 17: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/17.jpg)
Introduction
![Page 18: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/18.jpg)
Light Field
![Page 19: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/19.jpg)
http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt
SensorLens
θ x
Imaginary film
x
θ
Sensed image (in-focus)
object
object
red : the in-focus lineyellow : sample
![Page 20: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/20.jpg)
http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt
Lens Sensor
θ x
Imaginary film
x
θ
Sensed image (in-focus)
![Page 21: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/21.jpg)
http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt
Lens Sensor
u x
Imaginary film
x
θ
Sensed image (in-focus)red : the in-focus lineyellow : sample
![Page 22: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/22.jpg)
http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt
Lens Sensor
θ x
x
θ
Imaginary film
Sensed image (out of focus, far)red : the in-focus lineyellow : sample
![Page 23: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/23.jpg)
http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt
Lens Sensor
θ x
x
θ
Imaginary film
Sensed image (out of focus, far)red : the in-focus lineyellow : sample
![Page 24: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/24.jpg)
http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt
Lens Sensor
θ x
x
θ
Imaginary film
Sensed image (out of focus, near)red : the in-focus lineyellow : sample
![Page 25: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/25.jpg)
http://graphics.stanford.edu/papers/fourierphoto/fourierphoto.ppt
Lens Sensor
θ x
x
θ
Imaginary film
Sensed image (out of focus, near)red : the in-focus lineyellow : sample
![Page 26: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/26.jpg)
Light Field Acquisition
Integral Photography• Interal Photograpy
[Lippmann 1908]
• Integral camera [Okano et al. 99; Martnez-Corral et al. 04; Javidi and Okano 02]
• Light field Camera▫ Virtual viewpoint
[Levoy and Hanrahan 96]
[Gertler et al 96]
▫ Virtual aperture [Levoy and Hanrahan
96] [Isaksen et al. 00]
▫ Synthetic appearture photography (similar virtual aperture) [Levoy et al. 04] [Vaish et al. 04]
![Page 27: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/27.jpg)
Light field Camera
Plenoptic cameraLight field rendering Dapped Photography
• [Levoy and Hanrahan Siggraph 96]
• [Gortler et al 96, 06]
• [Adelson et al, IEEE95]
• [Levoy and Hanrahan Siggraph 96]
• [Gortler et al 96]
• Hand-held light field camera [R Ng et al 05]
• Fourier slice photography [R Ng, SIGGRAPH05]
• The mask weights the rays
![Page 28: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/28.jpg)
Related work
• Coded Imaging▫ Coded aperture imaging
Overcome the limit of pinhole camera [Skinner 98]
▫ Coded Exposure Camera [Raskar et al. 06]
• Wavefront Coding [Dowski and Cathey 95; Dowski and Johnson 99; van der
Gracht et al. 96] Traditional lens based [Farid and Simoncelli 98] Both wavefront and coded aperture [Jahnson et al. 00]
• Deblurring & deconvolution Include extended DOF images by refocusing a light field at
multiple depth and applying the digital photomontage tech. [Agarwala et al. 04]
Fusion of multiple blurred images [ Jaeberli 94]
![Page 29: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/29.jpg)
Theory & Framework
![Page 30: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/30.jpg)
For different focus settings, the obtained images correspond to slices at different angles, “Fourier Slice Photography ” [Ng, R. 05]
![Page 31: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/31.jpg)
Assumption : simulate the aperture as mask placed at lens
Open Aperture
![Page 32: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/32.jpg)
Assumption : a planar Lambertian object at the focus plane
Because no angular variations in the irradiance of rays from a Lambertian object, the content of light field is restricted to be along the fx axis
The sensed image is a slice of the modulated light field
Open Aperture
![Page 33: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/33.jpg)
• In-focus sensor
▫ The in-focus image corresponds to a slice of LA(fx, f θ) along fx (f θ =0)
▫ No information lost
• Out of focus sensor▫ The sensor image is a slanted slice
▫ The slant angle depends on the degree of mis-focus
Open Aperture
![Page 34: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/34.jpg)
Heterodyne Light Field Camera
![Page 35: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/35.jpg)
Mask as Modulator
![Page 36: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/36.jpg)
Mask as Modulator
• d = v (at aperture stop, θ plane)▫ Mask affects the all rays at an angle θ in a similar way !▫ m(x, θ) = c (y = θ)▫ α = 900
• d = 0 (at sensor, conjugate plane)▫ Mask attenuates all rays for the same x equally !▫ m(x, θ) = c (y = x)▫ α = 00
![Page 37: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/37.jpg)
Mask as Modulator
• Optimal Mask Position
▫ In practice, since the spatial resolution is much larger than the angular resolution, is very small, and therefore the mask needs to be placed close to the sensor
• Optimal Mask Pattern
•Harmonic sine wave
•Boost
![Page 38: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/38.jpg)
Notes
• 4D light field
• Aliasing▫ When band-limit assumption is not valid in the spatial
dimension, the energy in the higher spatial frequencies of the light field masquerade as energy in the lower angular dimension.
▫ Post-filter the recovered light field using a Kaiser-Bessel filter with a filer width of 1.5 [Ng 05]
![Page 39: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/39.jpg)
Encoded Blur Camera
![Page 40: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/40.jpg)
Mask as modulator
Assumption : layered Lambertian scene
Because no angular variations in the irradiance of rays from a Lambertian scene, the content of light field is restricted to be along the fx axis
∵
![Page 41: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/41.jpg)
Optimal Mask for Encoding Defocus Blur
•Blurred image is linear convolution (circularly convolution with zero padded)▫Defocus by PSF (point spread function)
•Coded aperture remove SNR only special cases
+ Star , -Natural photography
- Optimal mask – continuous valued code by gradient decent optimization (Matlab, fmincon)- 7x7 Binary mask as initial guess
- 10 hours of search
![Page 42: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/42.jpg)
Implementation & Analysis
![Page 43: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/43.jpg)
Light Filed Camera
Mask & Sensor
Heterodyne Light Field Camera210 mm f/5.6Nikkor-W Lens
CanoScan LiDE 70scanner sensor
Mask80 dots/mm
![Page 44: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/44.jpg)
Raw sensor image
![Page 45: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/45.jpg)
Scene parts which are in-focus can be recovered at full resolution
![Page 46: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/46.jpg)
Far Focused
Near Focused
![Page 47: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/47.jpg)
In-focus – full resolution Out of focus Low resolution refocused image
![Page 48: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/48.jpg)
![Page 49: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/49.jpg)
Analysis
- Scanner sensor leading to pattern noise (horizontal /vertical lines)
+ Easy to cover over in a conventional digital camera with a finer mask placed inside in the future
• Computation
+ Computation burden is low because of computing light field and refocusing is done in Fourier domain
- Calibration of in-plane rotation and shift of the mask with respect to sensor
![Page 50: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/50.jpg)
Failure Cases
• If Assumption of a band-limited light field is invalid, the aliasing artifacts in recovered light field
• 2D cosine mask needs to be moved away from the sensor because it results in diffraction
![Page 51: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/51.jpg)
Encoded Blur Camera
Low resolution Mask
Encoded Blur Camera100 mm f/2.8 USM Macro Lens Mask Sensor
Canon Rebel XT SLR camera
![Page 52: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/52.jpg)
ISO-12233 Chart
![Page 53: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/53.jpg)
Modulation Transfer Function (MTF) of ISO-12233
MTF: low MTF: high
![Page 54: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/54.jpg)
Full resolution digital refocusing using encoded blur camera
Captured photo
Refocused photo
![Page 55: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/55.jpg)
In-focus fence + blurred person
Deblurring without taking the occluders
into account
Weighted deconvolution Eq.
Binary mask for the occluders
• In this case, we can recover the sharp image if the blur size is larger than the occluder size
• b is the vectorized blurred image
• A is the block-Toeplitz matrix representing 2D blur
• W is a weighting matrix which sets the weights corresponding to the occluded pixels in the blurred image to zero
![Page 56: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/56.jpg)
Failure Cases
• Scenes with large variation in depths and those with view dependencies can not be handle
▫ Practice value
7x7 mask : blur size of about 20 pixels
• Finer resolution mask can handle large defocus blur but lead to diffraction blur
![Page 57: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/57.jpg)
Contributions
• A theoretical framework of modulating 4D light fields camera working on frequency domain
• A new class of 4D light filed camera holds full resolution modulated 4D light field
• Don’t require additional optical elements such as lens arrays
• Analyze defocus blur as a special case of the frequency domain re-mapping and demonstrate that a broadband mask at aperture can preserve high spatial frequencies in defocused image
= rays x mask
![Page 58: study Dappled Photography](https://reader033.vdocuments.net/reader033/viewer/2022061106/5410a8ae8d7f721e338b45a7/html5/thumbnails/58.jpg)
Future Work
• Light Fields for Dynamic Scenes▫ Changing masks with time▫ Coding in time and space
• General Ray Modulators▫ Tilted/curved/multiple masks▫ Wavelength dependent masks▫ Angular/Spatial Resolution Tradeoff
• Applications▫ Estimating lens aberration▫ Microscopy▫ Light Field Applications