lecture 11, part 1: coded imaging - mit opencourseware · mitsubishi electric research labs (merl)...
TRANSCRIPT
![Page 1: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/1.jpg)
Camera CultureCamera Culture
Ramesh RaskarRamesh Raskar
MIT Media LabMIT Media Lab
http://cameraculture.media.mit.edu/
Computational Camera & Photography: Coded Imaging
Computational Camera & Computational Camera & Photography: Coded ImagingPhotography: Coded Imaging
![Page 2: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/2.jpg)
Image removed due to copyright restrictions.See Fig. 1, “Eight major types of optics in animal eyes.”In Fernald, R. D. “Casting a Genetic Light on the Evolution of Eyes.”Science 313, no. 5795 (September 29, 2006): 1914-1918.http://dx.doi.org/10.1126/science.1127889
![Page 3: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/3.jpg)
Traditional Camera
Shutter is OPEN
![Page 4: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/4.jpg)
Blurring
==
Convolution
Traditional Camera: Box Filter
Sinc Function
f
|H(f)|
Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
![Page 5: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/5.jpg)
Blurring Process for Linear Motion
*
T
=
T
Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
![Page 6: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/6.jpg)
Deblurring Process
A \ bSmearMatrix
BlurredImage
\
T
UnknownImage=?
- Condition number for the smearing matrix is large- Thus invertibility is bad
Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
![Page 7: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/7.jpg)
Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
![Page 8: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/8.jpg)
Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
![Page 9: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/9.jpg)
Input ImageSource: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
![Page 10: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/10.jpg)
Rectified Crop
Deblurred Result
Source: Raskar, Agrawal and Tumblin. “Coded ExposurePhotography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
![Page 11: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/11.jpg)
Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
![Page 12: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/12.jpg)
Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
![Page 13: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/13.jpg)
Deblurred ResultInput Photo
Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
![Page 14: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/14.jpg)
Traditional Camera
Shutter is OPEN
![Page 15: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/15.jpg)
Our Camera
Flutter Shutter
![Page 16: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/16.jpg)
Shutter is OPEN and CLOSED
![Page 17: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/17.jpg)
Comparison of Blurred Images
![Page 18: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/18.jpg)
Implementation
Completely Portable
![Page 19: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/19.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras
Flutter Shutter On PointGrey CameraFlutter Shutter On PointGrey Camera
• Dragonfly2 camera
• External Trigger – Mode 5– Multiple Exposure Pulse Width Mode
• On Chip Fluttered Integration, no extra cost
Courtesy of MERL. Used with permission.
![Page 20: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/20.jpg)
Lab Setup
![Page 21: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/21.jpg)
Blurring
==
Convolution
Traditional Camera: Box Filter
Sinc Function
f
|H(f)|
Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
![Page 22: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/22.jpg)
Flutter Shutter: Coded Filter
Preserves High Spatial Frequencies
f
|H(f)|
Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
![Page 23: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/23.jpg)
Comparison
f
|H(f)|
f
|H(f)|
Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
![Page 24: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/24.jpg)
Inverse Filter Unstablef
|1/H(f)|
Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
Inverse Filter stable
f
|1/H(f)|
![Page 25: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/25.jpg)
Short Exposure Long Exposure
Matlab Lucy
Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
Coded Exposure
Our result
Ground Truth
![Page 26: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/26.jpg)
Motion Blur as Convolution
Source: Raskar, Agrawal and Tumblin. “Coded Exposure Photography: Motion Deblurring via Fluttered Shutter.”Proceedings of SIGGRAPH 2006.
![Page 27: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/27.jpg)
?
??
Convolution == Linear System
![Page 28: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/28.jpg)
Solving
![Page 29: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/29.jpg)
Are all codes “good”?
Alternate
All ones
Random
Our Code
f
f
f
f
|H(f)|
|H(f)|
|H(f)|
![Page 30: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/30.jpg)
Need to consider zero padded codes
f
f
f
|H(f)|
|H(f)|
|H(f)|
![Page 31: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/31.jpg)
License Plate Retrieval
![Page 32: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/32.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras
Camera Limitations and TradeoffsCamera Limitations and Tradeoffs
• Low dynamic range
• Resolution vs Noise
• Motion blur– Reduce shutter speed, but lose light
• Limited depth of field– Reduce aperture, but lose light– Increase shutter time, but motion blur
Courtesy of MERL. Used with permission.
![Page 33: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/33.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras
Changing Aperture SizeChanging Aperture Size
• Samuel Hasinoff and Kiriakos Kutulakos, ‘A Layer-Based Restoration Framework for Variable-Aperture Photography’ ICCV 2007
Images removed due to copyright restrictions.
![Page 34: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/34.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in CamerasHow to handle How to handle focus blur?focus blur?
Source: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007.
![Page 35: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/35.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in CamerasCodedCoded ExposureExposure (Flutter Shutter)(Flutter Shutter)
Raskar, Agrawal, TumblinRaskar, Agrawal, TumblinSIGGRAPH 2006SIGGRAPH 2006
CodedCoded ApertureAperturewith Veeraraghavan, Raskar, Tumblin, & Mohanwith Veeraraghavan, Raskar, Tumblin, & Mohan,,
SIGGRAPH 2007SIGGRAPH 2007
Temporal 1Temporal 1--D broadband code: D broadband code: MotionMotion DeblurringDeblurring
Spatial 2Spatial 2--D broadband code:D broadband code:FocusFocus DeblurringDeblurring
![Page 36: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/36.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras
In Focus Photo
LED
![Page 37: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/37.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras
Out of Focus Photo: Open Aperture
![Page 38: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/38.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras
Out of Focus Photo: Coded Aperture
![Page 39: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/39.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras
Out of Focus Photo: Coded Aperture
![Page 40: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/40.jpg)
Blurred Photos
Coded Aperture, 7 * 7 MaskOpen ApertureSource: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007.
![Page 41: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/41.jpg)
Deblurred Photos
Coded Aperture, 7 * 7 MaskOpen ApertureSource: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007.
![Page 42: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/42.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in CamerasCaptured Blurred Captured Blurred
PhotoPhoto
Source: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007.
![Page 43: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/43.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras
Refocused on Person
Source: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007.
![Page 44: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/44.jpg)
MERL,Northwestern Univ.
Mask-Enhanced Cameras: Heterodyned Light Fields & Coded Aperture Veeraraghavan, Raskar, Agrawal, Mohan & Tumblin
Blur Estimation & SegmentationBlur Estimation & Segmentation
Captured Blurred Photo
Deblur at different blur
sizes k
k = 10
k = 1
•Defocus blur dependent on depth
•Assumptions
•Layered Lambertian Scene
•Constant blur within each layer
![Page 45: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/45.jpg)
MERL,Northwestern Univ.
Mask-Enhanced Cameras: Heterodyned Light Fields & Coded Aperture Veeraraghavan, Raskar, Agrawal, Mohan & Tumblin
k = 10
k = 1
DeblurredImages
Define Cost FunctionDefine Cost Function
Cost Function Images
Likelihood Error: (Blurred image - Sharp Image * PSFk)2
Gradient Error: Natural Image Statistics, Gradient Kurtosis
k = 1
k = 10
![Page 46: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/46.jpg)
MERL,Northwestern Univ.
Mask-Enhanced Cameras: Heterodyned Light Fields & Coded Aperture Veeraraghavan, Raskar, Agrawal, Mohan & Tumblin
Blur Estimation & Segmentation == LabelingBlur Estimation & Segmentation == Labeling
• Graph cuts for labeling
Error Images
K = 7
K = 1k = 10
k = 1
![Page 47: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/47.jpg)
CapturedPhoto
Deblur, k = 7
Reblur
Fusion
Source: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007.
![Page 48: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/48.jpg)
![Page 49: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/49.jpg)
MERL,Northwestern Univ.
Mask-Enhanced Cameras: Heterodyned Light Fields & Coded Aperture Veeraraghavan, Raskar, Agrawal, Mohan & Tumblin
Weighted Deconvolution
Source: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007.
![Page 50: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/50.jpg)
MERL,Northwestern Univ.
Mask-Enhanced Cameras: Heterodyned Light Fields & Coded Aperture Veeraraghavan, Raskar, Agrawal, Mohan & Tumblin
Source: Veeraraghavan, Raskar, Agarwal, Mohan, and Tumblin. “Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing.” Proceedings of SIGGRAPH 2007.
![Page 51: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/51.jpg)
Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras
Blocking Light == More InformationBlocking Light == More Information
Coded ExposureCoded ExposureCoding in Time Coding in Time
Coded ApertureCoded ApertureCoding in SpaceCoding in Space
![Page 52: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/52.jpg)
Mask? Sensor
MaskSensorMask? Sensor
MaskSensor
Mask? Sensor
4D Light Field from 2D Photo:
Heterodyne Light Field Camera
Full Resolution Digital Refocusing:
Coded Aperture Camera
![Page 53: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/53.jpg)
Conventional Lens: Limited Depth of FieldConventional Lens: Limited Depth of Field
SmallerAperture
OpenAperture
Slide by Shree NayarCourtesy of Shree Nayar. Used with permission.
![Page 54: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/54.jpg)
Wavefront Coding using Cubic Phase PlateWavefront Coding using Cubic Phase Plate
"Wavefront Coding: jointly optimized optical and digital imaging systems“,E. Dowski, R. H. Cormack and S. D. Sarama , Aerosense Conference, April 25, 2000
Slide by Shree NayarCourtesy of Shree Nayar. Used with permission.
![Page 55: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/55.jpg)
Depth Invariant BlurDepth Invariant Blur
Conventional System Wavefront Coded System
Slide by Shree NayarCourtesy of Shree Nayar. Used with permission.
![Page 56: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/56.jpg)
Typical PSF changes slowly Designed PSF changes fast
Decoding depth via defocus blur
• Design PSF that changes quickly through focus so that defocus can be easily estimated• Implementation using phase diffractive mask
(Sig 2008, Levin et al used amplitude mask)
Phase mask
R. Piestun, Y. Schechner, J. Shamir, “Propagation-Invariant Wave Fields with Finite Energy,” JOSA A 17, 294-303 (2000)R. Piestun, J. Shamir, “Generalized propagation invariant wave-fields,” JOSA A 15, 3039 (1998)
Images removed due to copyright restrictions.
Images removed due to copyright restrictions.
Image removed due to copyright restrictions.
![Page 57: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/57.jpg)
Rotational PSFRotational PSF
R. Piestun, Y. Schechner, J. Shamir, “Propagation-Invariant Wave Fields with Finite Energy,” JOSA A 17, 294-303 (2000)R. Piestun, J. Shamir, “Generalized propagation invariant wave-fields,” JOSA A 15, 3039 (1998)
Images removed due to copyright restrictions.Two sequences showing rotating and standard PSF.
![Page 58: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/58.jpg)
Single Pixel Camera
Slides by Shree Nayar
Several slides removed due to copyright restrictions.
![Page 59: Lecture 11, Part 1: Coded Imaging - MIT OpenCourseWare · Mitsubishi Electric Research Labs (MERL) Coding and Modulation in Cameras Flutter Shutter On PointGrey Camera † Dragonfly2](https://reader034.vdocuments.net/reader034/viewer/2022042215/5ebd93b7f2ab1f689877b636/html5/thumbnails/59.jpg)
MIT OpenCourseWarehttp://ocw.mit.edu
MAS.531 / MAS.131 Computational Camera and PhotographyFall 2009
For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.