raskar mar09 nesosa
DESCRIPTION
TRANSCRIPT
Raskar, Camera Culture, MIT Media Lab
C t ti l Ph t hComputational Photography:Epsilon to Coded Imaging
Camera Culture
p g g
Camera Culture
Ramesh Raskar
C C ltCamera CultureAssociate Professor, MIT Media Lab http://raskar.info
Tools
for
Visual Computing
Shadow
Refractive
Reflective
Fernald, Science [Sept 2006]
How can we create an entirely new class of imaging platforms
that have an understanding of the world that far exceeds human ability
and produce meaningful abstractions that are well ithi h h ibilit ?within human comprehensibility ?
Ramesh Raskar http://raskar.info
Mitsubishi Electric Research Laboratories Raskar 2006Spatial Augmented Reality
CurvedPlanar Non-planar Pocket-ProjObjects
Computational IlluminationComputational Illumination
CurvedPlanar Non planar
SingleProjector
?
Pocket ProjObjects1998 2002 20021997
Projector
jUser : T
?
1998 2002 1999 20031998
MultipleProjectors
Computational Camera and PhotographyComputational Camera and Photography
Motion Blurred Photo
Sh t T diti l MURAShort Exposure
Traditional MURAShutter
Captured SinglePh tPhoto
Deblurred Result
Banding Artifacts and some spatial frequencies
Dark d i some spatial frequencies
are lostand noisy
Blurring == Convolution
Sh Bl d
Fourier Transform
PSF == Sinc Function
Sharp Photo
Blurred Photo
Traditional Camera: Shutter is OPEN: Box Filter
ω
Sh Bl d
Fourier Transform
Sharp Photo
Blurred PhotoPSF == Broadband Function
Preserves High Spatial Frequencies
Flutter Shutter: Shutter is OPEN and CLOSED
Flutter Shutter CameraFlutter Shutter CameraRaskar, Agrawal, Tumblin [Siggraph2006]
LCD opacity switched in coded sequence
Traditional
Coded Exposu
rere
Deblurred I
Deblurred I ImageImage
Image of Static Object
Coded Exposure Coded Aperture
Temporal 1-D broadband code: Motion Deblurring
Spatial 2-D broadband mask: Focus Deblurring
Coded Aperture CameraCoded Aperture Camera
The aperture of a 100 mm lens is modified
Rest of the camera is unmodifiedInsert a coded mask with chosen binary pattern
LED
In Focus Photo
Out of Focus Photo: Open Aperture
Out of Focus Photo: Coded Aperture
Captured Blurred Photo
Refocused on Person
Raskar, Camera Culture, MIT Media Lab
Computational Photography
1. Epsilon Photography– Low-level Vision: Pixels– Multiphotos by bracketing (HDR, panorama)– ‘Ultimate camera’
2. Coded Photography– Mid-Level Cues:
• Regions, Edges, Motion, Direct/globalg , g , , g– Single/few snapshot
• Reversible encoding of data– Additional sensors/optics/illum
3. Essence Photography– Not mimic human eyeNot mimic human eye– Beyond single view/illum– ‘New artform’
Raskar, Camera Culture, MIT Media Lab
• Ramesh Raskar and J k T bliJack Tumblin
• Book Publishers: A K Peters
Less is MoreLess is More
Blocking Light == More InformationBlocking Light == More Information
Coding in Time Coding in Time Coding in SpaceCoding in Space
Larval Trematode WormLarval Trematode Worm Coded Aperture CameraCoded Aperture Camera
Shielding Light …Shielding Light …g gg g
Larval Trematode WormLarval Trematode Worm Turbellarian WormTurbellarian Worm
Mask?
Sensor
MaskSensorMask
?
SensorMask?
Sensor
Sensor
Mask
4D Light Field from 2D Photo:
d h ld
Full Resolution Digital Refocusing:
Heterodyne Light Field Camera
Coded Aperture Camera
Light Field Inside a CameraLight Field Inside a Camera
Light Field Inside a CameraLight Field Inside a Camera
LensletLenslet--based Light Field camerabased Light Field camera
[Adelson and Wang, 1992, Ng et al. 2005 ]
Stanford Plenoptic Camera Stanford Plenoptic Camera [Ng et al 2005][Ng et al 2005]
Contax medium format camera Kodak 16-megapixel sensor
4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens
Adaptive Optics microlens array 125μ square-sided microlenses
Digital RefocusingDigital Refocusingg gg g
[Ng et al 2005][Ng et al 2005]
Can we achieve this with a Can we achieve this with a MaskMask alone?alone?Can we achieve this with a Can we achieve this with a MaskMask alone?alone?
Mask based Light Field CameraSensor
MaskSensor
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]
How to Capture How to Capture 4D Light Field with g
2D Sensor ?
Wh t h ld b th What should be the pattern of the mask ?pattern of the mask ?
Mask Tile
Cosine Mask Used
Mask Tile
1/f1/f0
Captured 2D Photo
Encoding due to Mask
Sensor Slice captures entire Light Field
fθfθ0
fxfx0
M d l iModulated Light Field
Modulation Function
Computing 4D Light Field
2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800
2D FFT
9*9=81 spectral copies
Rearrange 2D tiles into 4D planes200*200*9*94D IFFT
4D Light Field200*200*9*9
Full resolution 2D image of Focused Scene Parts
Captured 2D Photo
divide
Image of White Lambertian Plane
Wavefront Sensing in Any Wavelength !
MaskSensor
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]
Lens Flare Reduction/Enhancement using Lens Flare Reduction/Enhancement using 4D Ray Sampling4D Ray Sampling4D Ray Sampling4D Ray Sampling
Captured Glare Glare Captured Glare Reduced
Glare Enhanced
Glare = low frequency noise in 2D
•But is high frequency noise in 4D
•Remove via simple outlier rejection
i
Sensor
j
xu xu
Rays = Waves for Propagation and Interface
Fresnel propagation Chirp (Lens) Fourier transform Fractional Fourier transform
x2 x3 x4x1x1
x0
u2u1 u3b
¡ ba x0
u4
x2x0
- x0
a
x1x0x0- a
b- a x0
x3x4- b
a x0
a
x4
I
Imaging via volume hologram (Depth-specific Imaging)
KVH (x4=0, u4=θs/λ; x3, u3) -20
-15
-100.6
0.8
1 u4u3
u 3 [mm
-1] -5
0
5
10
150
0.2
0.4
0.6
x3 x4L
ZZ ½ µ ¶ ¾
x3 [mm]
-0.4 -0.2 0 0.2 0.4
15
20 -0.2
K V H (x4; u4; x3; u3) =ZZ
dx03dx0
4e¡ i 2¼(u 04 x 4 ¡ u 0
3 x 3 ) exp½
¡ i2¼̧ zf (u03 + u0
4)µ
¡ u3 + u4 ¡µs
¸
¶ ¾
£ sinc½
L¸µ
¡ u3 + u4 +u0
3 + u04
2
¶ µu4 +
u04
2¡
µs
¸
¶ ¾sinc
½L¸
µ¡ u3 + u4 ¡
u03 + u0
42
¶ µu4 ¡
u04
2¡
µs
¸
¶ ¾
K V H I (x2; u2; x1; u1)
Derivation: h(x2; x1) = exp½
¡ i¼¸
zf
f 2 (x1 + x2 ¡ f µs)2¾
sinc½
L¸ f 2 (x1 + x2) (x2 ¡ f µs)
¾
Parameters:0 5 ¹
K V H (x4; u4; x3; u3)¸ = 0.5 ¹m
µs= 30°L = 1 mm
zf = 50 mm
Raskar, Camera Culture, MIT Media Lab
Computational Photography
Camera Culture Group Ramesh Raskar http://raskar.info
Computational Photography1. Epsilon Photography
– Low-level Vision: PixelsMask
Sensor
– Multiphotos by bracketing (HDR, panorama)– ‘Ultimate camera’
2. Coded Photography– Mid-Level Cues:Mid Level Cues:
• Regions, Edges, Motion, Direct/global
• Coded Exposure– Flutter Shutter Motion Deblurring
• Coded Aperture– Defocus
• Optical Heterodyning• Optical Heterodyning– Lightfield or Wavefront sensing
• Coded Glare• 6D Display u 3 [m
m-1
]
-20
-15
-10
-5
0
5
100.2
0.4
0.6
0.8
1
p y• Femto-second Imaging• Rays = Waves x3 [mm]
-0.4 -0.2 0 0.2 0.4
15
20 -0.2
0
1 2
11
2D 2D 2D
How can we create an entirely new class of imaging platforms
that have an understanding of the world that far exceeds human ability
and produce meaningful abstractions that are well ithi h h ibilit ?within human comprehensibility ?
Ramesh Raskar http://raskar.info