camera culturecamera cultureweb.media.mit.edu/~raskar/talks/08mitilp/raskarilpoct08web.pdftemporal...
TRANSCRIPT
![Page 1: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/1.jpg)
Raskar, Camera Culture, MIT Media Lab
Camera CultureCamera Culture
Ramesh Raskar
C C ltCamera CultureAssociate Professor, MIT Media Lab
![Page 2: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/2.jpg)
![Page 3: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/3.jpg)
Where are the ‘camera’s?Where are the ‘camera’s?
![Page 4: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/4.jpg)
Where are the ‘camera’s?Where are the ‘camera’s?
![Page 5: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/5.jpg)
We focus on creating tools to better capture and share visual information
Th l i t t ti l The goal is to create an entirely new class of imaging platforms
that have an understanding of the world that far exceeds human ability
and produce meaningful abstractions that are well within human comprehensibilitywithin human comprehensibility
Ramesh Raskar
![Page 6: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/6.jpg)
Raskar, Camera Culture, MIT Media Lab
QuestionsQuestions• What will a camera look like in 10,20 years?
![Page 7: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/7.jpg)
Raskar, Camera Culture, MIT Media Lab
Cameras of TomorrowCameras of Tomorrow
![Page 8: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/8.jpg)
MIT Media Lab
ApproachApproach
N t j t USE b t CHANGE• Not just USE but CHANGE camera– Optics, illumination, sensor, movement– Exploit wavelength, speed, depth, polarization etcp g p p p– Probes, actuators
W h h t d bit i i l• We have exhausted bits in pixels– Scene understanding is challenging– Build feature-revealing camerasg– Process photons
• Technology, Applications, Society– We study impact of Imaging on all fronts
![Page 9: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/9.jpg)
Mitsubishi Electric Research Laboratories Raskar 2006Spatial Augmented Reality
CurvedPlanar Non-planar Pocket-ProjObjects
Computational IlluminationComputational Illumination
CurvedPlanar Non planar
SingleProjector
?
Pocket ProjObjects1998 2002 20021997
Projector
jUser : T
?
1998 2002 1999 20031998
MultipleProjectors
Computational Camera and PhotographyComputational Camera and Photography
![Page 10: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/10.jpg)
Motion Blurred Photo
![Page 11: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/11.jpg)
![Page 12: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/12.jpg)
Flutter Shutter CameraFlutter Shutter CameraRaskar, Agrawal, Tumblin [Siggraph2006]
LCD opacity switched in coded sequence
![Page 13: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/13.jpg)
Sh t T diti l MURAShort Exposure
Traditional MURAShutter
Captured SinglePh tPhoto
Deblurred Result
Banding Artifacts and some spatial frequencies
Dark d i some spatial frequencies
are lostand noisy
![Page 14: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/14.jpg)
Blurring == Convolution
Sh Bl d
Fourier Transform
PSF == Sinc Function
Sharp Photo
Blurred Photo
Traditional Camera: Shutter is OPEN: Box Filter
ω
![Page 15: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/15.jpg)
Sh Bl d
Fourier Transform
Sharp Photo
Blurred PhotoPSF == Broadband Function
Preserves High Spatial Frequencies
Flutter Shutter: Shutter is OPEN and CLOSED
![Page 16: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/16.jpg)
Traditional
Coded Exposu
rere
Deblurred I
Deblurred I ImageImage
Image of Static Object
![Page 17: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/17.jpg)
![Page 18: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/18.jpg)
Coded Exposure Coded Aperture
Temporal 1-D broadband code: Motion Deblurring
Spatial 2-D broadband mask: Focus Deblurring
![Page 19: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/19.jpg)
Coded Aperture CameraCoded Aperture Camera
The aperture of a 100 mm lens is modified
Rest of the camera is unmodifiedInsert a coded mask with chosen binary pattern
![Page 20: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/20.jpg)
LED
In Focus Photo
![Page 21: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/21.jpg)
Out of Focus Photo: Open Aperture
![Page 22: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/22.jpg)
Out of Focus Photo: Coded Aperture
![Page 23: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/23.jpg)
Captured Blurred Photo
![Page 24: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/24.jpg)
Refocused on Person
![Page 25: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/25.jpg)
Less is MoreLess is More
Blocking Light == More InformationBlocking Light == More Information
Coding in Time Coding in Time Coding in SpaceCoding in Space
![Page 26: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/26.jpg)
Larval Trematode WormLarval Trematode Worm Coded Aperture CameraCoded Aperture Camera
![Page 27: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/27.jpg)
Shielding Light …Shielding Light …g gg g
Larval Trematode WormLarval Trematode Worm Turbellarian WormTurbellarian Worm
![Page 28: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/28.jpg)
Coded Computational Photography• Coded Exposure• Coded Exposure
– Motion Deblurring [2006]
C d d A t• Coded Aperture– Focus Deblurring [2007]– Glare reduction [2008]
O i l H d i• Optical Heterodyning– Light Field Capture [2007]
C d d Ill i ti• Coded Illumination– Motion Capture [2007]– Multi-flash: Shape Contours [2004]
• Coded Spectrum– Agile Wavelength Profile [2008]
• Epsilon->Coded->Essence Photography
http://raskar.info
![Page 29: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/29.jpg)
Raskar, Camera Culture, MIT Media Lab
Computational Photography
1. Epsilon Photography– Low-level Vision: Pixels– Multiphotos by bracketing (HDR, panorama)– ‘Ultimate camera’
2. Coded Photography– Mid-Level Cues:
• Regions, Edges, Motion, Direct/globalg , g , , g– Single/few snapshot
• Reversible encoding of data– Additional sensors/optics/illum
3. Essence Photography– Not mimic human eyeNot mimic human eye– Beyond single view/illum– ‘New artform’
![Page 30: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/30.jpg)
Raskar, Camera Culture, MIT Media Lab
Epsilon PhotographyEpsilon Photography
• Dynamic rangeDynamic range– Exposure braketing [Mann-Picard, Debevec]
• Wider FoV – Stitching a panorama
• Depth of field Fusion of photos with limited DoF [Agrawala04]– Fusion of photos with limited DoF [Agrawala04]
• Noise– Flash/no-flash image pairs [Petschnigg04, Eisemann04]
• Frame rate– Triggering multiple cameras [Wilburn05, Shechtman02]
![Page 31: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/31.jpg)
Raskar, Camera Culture, MIT Media Lab
Computational Photography
1. Epsilon Photography– Low-level Vision: Pixels– Multiphotos by bracketing (HDR, panorama)– ‘Ultimate camera’
2. Coded Photography– Mid-Level Cues:
• Regions, Edges, Motion, Direct/globalg , g , , g– Single/few snapshot
• Reversible encoding of data– Additional sensors/optics/illum
3. Essence Photography– Not mimic human eyeNot mimic human eye– Beyond single view/illum– ‘New artform’
![Page 32: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/32.jpg)
Raskar, Camera Culture, MIT Media Lab
• 3D– Stereo of multiple cameras
• Higher dimensional LF– Light Field Capture
• lenslet array [Adelson92, Ng05], ‘3D lens’ [Georgiev05], heterodyne masks [Veeraraghavan07]
• Boundaries and RegionsMulti flash camera with shadows [R k 08]– Multi-flash camera with shadows [Raskar08]
– Fg/bg matting [Chuang01,Sun06]
• DeblurringE i d PSF– Engineered PSF
– Motion: Flutter shutter[Raskar06], Camera Motion [Levin08]
– Defocus: Coded aperture, Wavefront coding [Cathey95]
Gl b l di t ill i ti• Global vs direct illumination– High frequency illumination [Nayar06]
– Glare decomposition [Talvala07, Raskar08]
• Coded Sensor– Gradient camera [Tumblin05]
![Page 33: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/33.jpg)
Raskar, Camera Culture, MIT Media Lab
Computational Photography
1. Epsilon Photography– Low-level Vision: Pixels– Multiphotos by bracketing (HDR, panorama)– ‘Ultimate camera’
2. Coded Photography– Mid-Level Cues:
• Regions, Edges, Motion, Direct/globalg , g , , g– Single/few snapshot
• Reversible encoding of data– Additional sensors/optics/illum
3. Essence Photography– Not mimic human eyeNot mimic human eye– Beyond single view/illum– ‘New artform’
![Page 34: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/34.jpg)
Raskar, Camera Culture, MIT Media Lab
Capturing the Essence of Visual ExperienceCapturing the Essence of Visual Experience
– Exploiting online collections• Photo tourism [Snavely2006]• Photo-tourism [Snavely2006]• Scene Completion [Hays2007]
– Multi-perspective Images• Multi-linear Perspective [Jingyi Yu, McMillan 2004]• Unwrap Mosaics [Rav-Acha et al 2008]• Video texture panoramas [Agrawal et al 2005]
– Non-photorealistic synthesis• Motion magnification [Liu05]
Image Priors– Image Priors• Learned features and natural statistics• Face Swapping: [Bitouk et al 2008]• Data-driven enhancement of facial attractiveness [Leyvand et al 2008]• Data-driven enhancement of facial attractiveness [Leyvand et al 2008]• Deblurring [Fergus et al 2006, Jia et al 2008]
![Page 35: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/35.jpg)
Raskar, Camera Culture, MIT Media Lab
Computational Photography
1. Epsilon Photography– Low-level Vision: Pixels– Multiphotos by bracketing (HDR, panorama)– ‘Ultimate camera’
2. Coded Photography– Mid-Level Cues:
• Regions, Edges, Motion, Direct/globalg , g , , g– Single/few snapshot
• Reversible encoding of data– Additional sensors/optics/illum
3. Essence Photography– Not mimic human eyeNot mimic human eye– Beyond single view/illum– ‘New artform’
![Page 36: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/36.jpg)
Raskar, Camera Culture, MIT Media Lab
• Ramesh Raskar and J k T bliJack Tumblin
• Book Publishers: A K Peters
![Page 37: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/37.jpg)
Mask?
Sensor
MaskSensorMask
?
SensorMask?
Sensor
Sensor
Mask
4D Light Field from 2D Photo:
d h ld
Full Resolution Digital Refocusing:
Heterodyne Light Field Camera
Coded Aperture Camera
![Page 38: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/38.jpg)
Light Field Inside a CameraLight Field Inside a Camera
![Page 39: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/39.jpg)
Light Field Inside a CameraLight Field Inside a Camera
LensletLenslet--based Light Field camerabased Light Field camera
[Adelson and Wang, 1992, Ng et al. 2005 ]
![Page 40: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/40.jpg)
Stanford Plenoptic Camera Stanford Plenoptic Camera [Ng et al 2005][Ng et al 2005]
Contax medium format camera Kodak 16-megapixel sensor
4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens
Adaptive Optics microlens array 125μ square-sided microlenses
![Page 41: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/41.jpg)
Digital RefocusingDigital Refocusingg gg g
[Ng et al 2005][Ng et al 2005]
Can we achieve this with a Can we achieve this with a MaskMask alone?alone?Can we achieve this with a Can we achieve this with a MaskMask alone?alone?
![Page 42: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/42.jpg)
Mask based Light Field CameraSensor
MaskSensor
[Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]
![Page 43: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/43.jpg)
How to Capture How to Capture 4D Light Field with g
2D Sensor ?
Wh t h ld b th What should be the pattern of the mask ?pattern of the mask ?
![Page 44: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/44.jpg)
Optical HeterodyningOptical HeterodyningReceiver:High Freq Carrier Receiver:
DemodulationHigh Freq Carrier
100 MHz
Incoming
Baseband Audio R f
Incoming Signal
Software Demodulation
Signal ReferenceCarrier99 MHz
Main LensObject Mask Sensor
RecoveredLi ht
Software Demodulation
Light Field
Photographic Signal
(Light Field)
Carrier Incident Modulat
ed
ReferenceCarrier
![Page 45: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/45.jpg)
Mask Tile
Cosine Mask Used
Mask Tile
1/f1/f0
![Page 46: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/46.jpg)
Captured 2D Photo
Encoding due to Mask
![Page 47: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/47.jpg)
2D FFT
Traditional Camera Photo
Magnitude of 2D FFTPhoto FFT
2D 2D FFT
Heterodyne Camera Photo
Magnitude of 2D FFT
![Page 48: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/48.jpg)
Computing 4D Light Field
2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800
2D FFT
9*9=81 spectral copies
Rearrange 2D tiles into 4D planes200*200*9*94D IFFT
4D Light Field200*200*9*9
![Page 49: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/49.jpg)
How to Capture 2D Light Field with 1D Sensor ?
fθ Band-limited fθfθ0
Light Field
fxfx0
Sensor Slice
Fourier Light Field Space
![Page 50: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/50.jpg)
Extra sensor bandwidth cannot capture extra dimension of the light fieldg
fθ Extra sensorfθfθ0
Extra sensor bandwidth
fxfx0
Sensor Slice
Fourier Light Field Space
![Page 51: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/51.jpg)
Solution: Modulation Theorem
Make spectral copies of 2D light fieldMake spectral copies of 2D light field
fθθfθ0
fxfx0
ModulationModulation Function
![Page 52: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/52.jpg)
Sensor Slice captures entire Light Field
fθfθ0
fxfx0
M d l iModulated Light Field
Modulation Function
![Page 53: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/53.jpg)
Mask Tile
Cosine Mask Used
Mask Tile
1/f1/f0
![Page 54: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/54.jpg)
Demodulation to recover Light Field1D Fourier Transform of Sensor Signal
fθ
fx
Reshape 1D Fourier Transform into 2D
![Page 55: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/55.jpg)
Where to place the Mask?
Mask Sensor
fθMask
Modulation
fθ
Modulation Functionfx
![Page 56: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/56.jpg)
Full resolution 2D image of Focused Scene Parts
Captured 2D Photo
divide
Image of White Lambertian Plane
![Page 57: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/57.jpg)
Coding and Modulation in Camera Using MasksCoding and Modulation in Camera Using Masks
Mask SensorMask?
MasSensorSensor
Mask
Mask
Coded Aperture for
Full Resolution
Heterodyne Light Field
![Page 58: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/58.jpg)
Agile Spectrum Imaging
Programmable Color Gamut for Sensor
With Ankit Mohan, Jack Tumblin [Eurographics 2008]
![Page 59: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/59.jpg)
Traditional Fixed Color Gamut
R ≈ 0.0G
G ≈ 0.2
B ≈ 0.8R B 0.8
B
R
B
![Page 60: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/60.jpg)
Adaptive Color PrimariesPrimaries
![Page 61: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/61.jpg)
Rainbow Plane inside Camera
Pinhole Sensor
A’
PinholeLens L1
Lens L2Sensor
C’Scene
C A’
B’
C’
B B
C’
B’’
A C
Rainbow Pl
A’’
Prism orDiffraction Grating
Plane
![Page 62: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/62.jpg)
Lens Flare Reduction/Enhancement using Lens Flare Reduction/Enhancement using 4D Ray Sampling4D Ray Sampling4D Ray Sampling4D Ray Sampling
Captured Glare Glare Captured Glare Reduced
Glare Enhanced
![Page 63: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/63.jpg)
Glare = low frequency noise in 2D
•But is high frequency noise in 4D
•Remove via simple outlier rejection
i
Sensor
j
xu xu
![Page 64: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/64.jpg)
Mitsubishi Electric Research Laboratories Raskar 2006Spatial Augmented Reality
Computational Camera and PhotographyComputational Camera and Photography
![Page 65: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/65.jpg)
Dependence on incident angle
![Page 66: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/66.jpg)
Dependence on incident angle
![Page 67: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/67.jpg)
Towards a 6D DisplayPassive Reflectance Field Display
1 2
11
2D 2D 2D
Martin Fuchs, Ramesh Raskar,Hans Peter Seidel Hendrik P A Lensch
11
Hans-Peter Seidel, Hendrik P. A. Lensch
Siggraph 2008
1 MPI Informatik, Germany 2 MIT
![Page 68: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/68.jpg)
View dependent 4D Display
![Page 69: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/69.jpg)
![Page 70: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/70.jpg)
![Page 71: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/71.jpg)
6D = light sensitive 4D display
![Page 72: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/72.jpg)
One Pixel of a 6D Display = 4D Display
![Page 73: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/73.jpg)
MIT Media Lab
Single shot visual hullg
Lanman, Raskar, Agrawal, Taubin [Siggraph Asia 2008]
![Page 74: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/74.jpg)
MIT Media Lab
Single shot 3D reconstruction: Si lt P j ti i M kSimultaneous Projections using Masks
![Page 75: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/75.jpg)
Long Distance BarLong Distance Bar--codescodes
• Barcode size : 3mm x 3mmDi f 5
gg
• Distance from camera : 5 meter
Woo, Mohan, Raskar, [2008]
![Page 76: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/76.jpg)
Raskar, Camera Culture, MIT Media Lab
ProjectsLight eight Medical Imaging• Lightweight Medical Imaging
• High speed Tomography• Muscle, blood flow activity with wearable devices for patients
• Femto-second Analysis of Light TransportFemto second Analysis of Light Transport• Building and modeling future ultra-high speed cameras• Avoid car-crashes, analyze complex scenes
• Programmable Wavelength in Thermal Range• Facial expressions, Healthcare• Human-emotion aware computing, Fast diagnosis
• Second skin• Wearable fabric for bio-I/O via high speed optical motion capture• Record and mimic any human motion, Care for elderly, Teach a robot
![Page 77: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/77.jpg)
Vicon Motion Capture
Medical Rehabilitation Athlete Analysis
Body-worn markers
High-speed IR Camera
Performance Capture Biomechanical Analysis
![Page 78: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/78.jpg)
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006Inverse Optical Mo Cap
TraditionalMo-Cap
Device High Speed Projector + Photosensing Markers
High Speed Camera + Reflecting/Emitting Markersg g
Params Location, Orientation, Illum Location
Settings
Natural SettingsAmbient Light
Outdoors, Stage lighting
Imperceptible tags
Controlled LightingVisible, High contrast Markers
Hidden under wardrobe
#of TagsUnlimited
Space LabelingUnique Id
LimitedNo Unique Id Marker swappingUnique Id Marker swapping
SpeedVirtually unlimited
Optical comm compsLimited
Special high fps camera
CostLow
Open-loop projectorsCurrent: Projector/Tag=$100
HighHigh bandwidth cameraCurrent Camera: $10K
![Page 79: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/79.jpg)
Mitsubishi Electric Research Laboratories Special Effects in the Real World Raskar 2006
Inside of ProjectorInside of Projector
Focusing OpticsCondensing Optics Light Source
Gray code Slide
The Gray code pattern
![Page 80: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/80.jpg)
Imperceptible Tags under clothing, tracked under ambient light
![Page 81: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/81.jpg)
Towards Second SkinTowards Second SkinCoded IlluminationCoded Illumination Motion Capture Clothing
• 500 Hz with Id for each Marker Tag
• Capture in Natural Environment– Visually imperceptible tags
– Photosensing Tag can be hidden under clothesPhotosensing Tag can be hidden under clothes
– Ambient lighting is ok
• Unlimited Number of TagsLight sensitive fabric for dense sampling– Light sensitive fabric for dense sampling
• Non‐imaging, complete privacy
• Base station and tags only a few 10’s $
• Full body scan + actions– Elderly, patients, athletes, performersy, p , , p
– Breathing, small twists, multiple segments or people
– Animation Analysis
![Page 82: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/82.jpg)
Coded ImagingCoded Imaging
Coding in Time Coding in Space (Optical Path) Coded Illumination Coded Wavelength Coded SensingCoding in Time Coding in Space (Optical Path) Coded Illumination Coded Wavelength Coded Sensing
Coded Exposure for Motion Deblurring
Coded Aperture for Extended Depth of Field
Mask‐based Optical Heterodyning for Light Field Capture
Multi‐flash Imaging for Depth Edge Detection
Agile Spectrum Imaging
Gradient Encoding Sensor for HDR
![Page 83: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/83.jpg)
Forerunners ..Forerunners ..
MaskSensor
MaskSensor
Mask
![Page 84: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/84.jpg)
Tools
for
Visual Computing
Shadow Refractive Reflective
Fernald, Science [Sept 2006]
![Page 85: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/85.jpg)
![Page 86: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/86.jpg)
Blind CameraBlind Camera
Sascha Pohflepp, U of the Art, Berlin, 2006
![Page 87: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/87.jpg)
Raskar, Camera Culture, MIT Media Lab
Cameras of TomorrowCameras of Tomorrow
![Page 88: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/88.jpg)
Cameras of Tomorrow• Coded Exposure• Coded Exposure
– Motion Deblurring [2006]
C d d A t• Coded Aperture– Focus Deblurring [2007]– Glare reduction [2008]
O i l H d i• Optical Heterodyning– Light Field Capture [2007]
C d d Ill i ti• Coded Illumination– Motion Capture [2007]– Multi-flash: Shape Contours [2004]
• Coded Spectrum– Agile Wavelength Profile [2008]
• Epsilon->Coded->Essence Photography
http://raskar.info
![Page 89: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/89.jpg)
• Capture– Cameras Everywhere
D i i– Deep pervasive sensing
• AnalysisC Vi i– Computer Vision
– Mo-cap • Personalized services, tracking in real world• Animation
Simulation of bio/chemical/physical processes at all scales– Simulation of bio/chemical/physical processes at all scales
• Synthesis– Virtual Human, Digital Actors, Tele-avatarsVirtual Human, Digital Actors, Tele avatars– Exa and Zeta-scale computing:
• simulate every neural activity, predict weather for weeks, simulate impact of global warming
• Display– Real world AR– Realistic Displays: 6D or 8D
![Page 90: Camera CultureCamera Cultureweb.media.mit.edu/~Raskar/Talks/08MitILP/RaskarILPOct08Web.pdfTemporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring](https://reader034.vdocuments.net/reader034/viewer/2022042216/5ebf3f1f8e4eca44394b9822/html5/thumbnails/90.jpg)
We focus on creating tools to better capture and share visual information
Th l i t t ti l The goal is to create an entirely new class of imaging platforms
that have an understanding of the world that far exceeds human ability
and produce meaningful abstractions that are well within human comprehensibilitywithin human comprehensibility
Ramesh Raskar http://raskar.info