mit media lab camera culture ramesh raskar jack tumblin computational photography: advanced topics...

MIT Media LabMIT Media Lab

Camera CultureCamera Culture

Ramesh RaskarRamesh Raskar

Jack TumblinJack Tumblin

Computational Computational Photography:Photography:

Advanced TopicsAdvanced Topics

Computational Computational Photography:Photography:

Advanced TopicsAdvanced Topics

Paul DebevecPaul Debevec

Speaker: Jack TumblinSpeaker: Jack Tumblin

Associate Professor of Computer Science at Northwestern Univ.

His “Look Lab” group pursues research on new methods to capture and manipulate the appearance of objects and surroundings, in the hope that hybrid optical/computer methods may give us new ways to see, explore, and interact with objects and people anywhere in the world. During his doctoral studies at Georgia Tech and post-doc at Cornell, he investigated tone-mapping methods to depict high-contrast scenes. His MS in Electrical Engineering (December 1990) and BSEE (1978), also from Georgia Tech, bracketed his work as co-founder of IVEX Corp., (>45 people as of 1990) where his flight simulator design work was granted 5 US Patents. He was an Associate Editor of ACM Transactions on Graphics (2000-2006), a member of the SIGGRAPH Papers Committee (2003, 2004), and in 2001 was a Guest Editor of IEEE Computer Graphics and Applications.

http://www.cs.northwestern.edu/~jet

Speaker: Paul DebevecSpeaker: Paul Debevec

Research Associate Professor ,University of Southern California and the

Associate director of Graphics Research,USC's Institute for Creative Technologies.

Debevec's Ph.D. thesis (UC Berkeley, 1996) presented Façade, an image-based modeling and rendering system for creating photoreal architectural models from photographs. Pioneer in high dynamic range photography, he demonstrated new image-based lighting techniques in his films Rendering with Natural Light (1998), Fiat Lux (1999), and The Parthenon (2004); he also led the design of HDR Shop, the first high dynamic range image editing program. At USC ICT, Debevec has led the development of a series of Light Stage devices used in Spider Man 2 and Superman Returns. He is the recipient of ACM SIGGRAPH's first Significant New Researcher Award and a co-author of the 2005 book High Dynamic Range Imaging from Morgan Kaufmann.

http://www.debevec.org

Speaker: Ramesh RaskarSpeaker: Ramesh Raskar

Associate Professor, MIT Media Lab.

Previously at MERL as a Senior Research Scientist.His research interests include projector-based graphics, computational photography and non-photorealistic rendering. He has published several articles on imaging and photography including multi-flash photography for depth edge detection, image fusion, gradient-domain imaging and projector-camera systems. His papers have appeared in SIGGRAPH, EuroGraphics, IEEE Visualization, CVPR and many other graphics and vision conferences. He was a course organizer at Siggraph 2002 through 2005. He was the panel organizer at the Symposium on Computational Photography and Video in Cambridge, MA in May 2005 and taught a graduate level class on Computational Photography at Northeastern University, Fall 2005. He is a member of the ACM and IEEE.

http://raskar.infohttp://www.media.mit.edu/~raskar

OverviewOverview

– Unlocking PhotographyUnlocking Photography• Not about the equipment but about the goalNot about the equipment but about the goal• Capturing ‘machine readable’ visual experienceCapturing ‘machine readable’ visual experience• Goes beyond what you can see through the viewfinderGoes beyond what you can see through the viewfinder• Push the envelope with seemingly peripheral techniques and advancesPush the envelope with seemingly peripheral techniques and advances

– Think beyond post-capture image processingThink beyond post-capture image processing• ‘‘Computation’ well before image processing and editing Computation’ well before image processing and editing • Learn how to build your own camera-toysLearn how to build your own camera-toys

– EmphasisEmphasis• Most recent work in graphics/vision (2006 and later)Most recent work in graphics/vision (2006 and later)• Research in other fields: Applied optics, novel sensors, materialsResearch in other fields: Applied optics, novel sensors, materials• Review of 50+ recent papers and projectsReview of 50+ recent papers and projects

– What we will not coverWhat we will not cover• Minimum discussion of graphics/vision papers before 2006Minimum discussion of graphics/vision papers before 2006• Epsilon photography (improving camera performance by bracketing)Epsilon photography (improving camera performance by bracketing)• Film Cameras, Novel view rendering (IBR), Color issues, Traditional image Film Cameras, Novel view rendering (IBR), Color issues, Traditional image

processing/editingprocessing/editing

Traditional PhotographyTraditional Photography

Courtesy: Shree Nayar

Lens

Detector

Pixels

Image

Traditional PhotographyTraditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot:

Single View, Single Instant, Fixed Dynamic range and Depth of field for given Illumination in a Static world

Computational PhotographyComputational Photography: :

Optics, Sensors and ComputationsOptics, Sensors and ComputationsGeneralized

Sensor

Generalized Optics

Computations

Picture

4D Ray Bender

Upto 4D Ray Sampler

Ray Reconstruction

Merged braketed photos, Coded sensing

Computational Computational PhotographyPhotography

Novel CamerasGeneralized

Sensor

Generalized Optics

Processing


Novel Illumination

Novel CamerasGeneralized

Sensor

Generalized Optics

Processing

Light Sources


Novel Illumination

Novel Cameras

Scene: 8D Ray Modulator

GeneralizedSensor

Generalized Optics

Processing

Light Sources


Novel Illumination

Novel Cameras


Display

GeneralizedSensor

Generalized Optics

Processing

Recreate 4D Lightfield

Light Sources


Novel Illumination

Novel Cameras


Display

GeneralizedSensor

Generalized Optics

Processing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

What is Computational Photography?What is Computational Photography?

• Create photo that could not have been taken by a traditional Camera (?)Create photo that could not have been taken by a traditional Camera (?)

Goal: Record a richer, multi-layered visual experienceGoal: Record a richer, multi-layered visual experience

1.1. Overcome limitations of today’s camerasOvercome limitations of today’s cameras

2.2. Support better post-capture processingSupport better post-capture processing

– Relightable photos, Focus/Depth of field, Fg/Bg, Shape boundariesRelightable photos, Focus/Depth of field, Fg/Bg, Shape boundaries

3.3. Enables new classes of recording the visual signal Enables new classes of recording the visual signal

– Moment [Cohen05], Time-lapse, Unwrap mosaics, Cut-viewsMoment [Cohen05], Time-lapse, Unwrap mosaics, Cut-views

4.4. Synthesize “impossible” photosSynthesize “impossible” photos

– Wrap-around views [Rademacher and Bishop 1998]), fusion of time-lapsed events [Raskar et al 2004], motion Wrap-around views [Rademacher and Bishop 1998]), fusion of time-lapsed events [Raskar et al 2004], motion magnification [Liu et al 2005]), video textures and panoramas [Agarwala et al 2005]. magnification [Liu et al 2005]), video textures and panoramas [Agarwala et al 2005].

5.5. Exploit previously exotic forms of scientific imaging Exploit previously exotic forms of scientific imaging

– Coded aperture [Veeraraghavan 2007, Levin 2007], confocal imaging [Levoy 2004], tomography [Trifonov Coded aperture [Veeraraghavan 2007, Levin 2007], confocal imaging [Levoy 2004], tomography [Trifonov 2006]2006]

Computational PhotographyComputational Photography

1. Epsilon Photography– Low-level vision: Pixels– Multi-photos by perturbing camera parameters– HDR, panorama, …– ‘Ultimate camera’

2. Coded Photography– Single/few snapshot– Reversible encoding of data– Additional sensors/optics/illum– ‘Scene analysis’ : (Consumer software?)

3. Essence Photography– Beyond single view/illum– Not mimic human eye– ‘New art form’

Epsilon PhotographyEpsilon Photography

• Dynamic range– Exposure bracketing [Mann-Picard, Debevec]

• Wider FoV – Stitching a panorama

• Depth of field – Fusion of photos with limited DoF [Agrawala04]

• Noise– Flash/no-flash image pairs

• Frame rate– Triggering multiple cameras [Wilburn04]

Goal: High Dynamic RangeGoal: High Dynamic Range

Short ExposureShort Exposure

Long ExposureLong Exposure

Dynamic Range

Epsilon PhotographyEpsilon Photography• Dynamic range

– Exposure braketing [Mann-Picard, Debevec]

• Wider FoV – Stitching a panorama

• Depth of field – Fusion of photos with limited DoF [Agrawala04]

• Noise– Flash/no-flash image pairs [Petschnigg04, Eisemann04]

• Frame rate– Triggering multiple cameras [Wilburn05, Shechtman02]


1. Epsilon Photography– Low-level Vision: Pixels– Multiphotos by perturbing camera parameters– HDR, panorama– ‘Ultimate camera’

2. Coded Photography– Mid-Level Cues:

• Regions, Edges, Motion, Direct/global– Single/few snapshot

• Reversible encoding of data– Additional sensors/optics/illum– ‘Scene analysis’

3. Essence Photography– Not mimic human eye– Beyond single view/illum– ‘New artform’

• 3D– Stereo of multiple cameras

• Higher dimensional LF– Light Field Capture

• lenslet array [Adelson92, Ng05], ‘3D lens’ [Georgiev05], heterodyne masks [Veeraraghavan07]

• Boundaries and Regions– Multi-flash camera with shadows [Raskar08]

– Fg/bg matting [Chuang01,Sun06]

• Deblurring– Engineered PSF– Motion: Flutter shutter[Raskar06], Camera Motion [Levin08]

– Defocus: Coded aperture [Veeraraghavan07,Levin07], Wavefront coding [Cathey95]

• Global vs direct illumination– High frequency illumination [Nayar06]

– Glare decomposition [Talvala07, Raskar08]

• Coded Sensor– Gradient camera [Tumblin05]

Marc Levoy

Digital Refocusing using Light Field Camera

125μ square-sided microlenses[Ng et al 2005]

Depth Depth EdgesEdges

LeftLeft TopTop RightRight BottomBottom

Depth EdgesDepth EdgesCanny EdgesCanny Edges

Flutter Shutter CameraFlutter Shutter CameraRaskar, Agrawal, Tumblin Raskar, Agrawal, Tumblin

[Siggraph2006][Siggraph2006]

LCD opacity switched LCD opacity switched in coded sequencein coded sequence

TraditioTraditionalnal

Coded Coded ExposuExposu

rere

Image of Image of Static Static ObjectObject

Deblurred Deblurred ImageImage

Deblurred Deblurred ImageImage








• Decomposition Problems– High frequency illumination, Global/direct illumination [Nayar06]



"Fast Separation of Direct and Global Components of a Scene using High Frequency Illumination," S.K. Nayar, G. Krishnan, M. D. Grossberg, R. Raskar, ACM Trans. on Graphics (also Proc. of ACM SIGGRAPH), Jul, 2006.

"Fast Separation of Direct and Global Components of a Scene using High Frequency Illumination," S.K. Nayar, G. Krishnan, M. D. Grossberg, R. Raskar, ACM Trans. on Graphics (also Proc. of ACM SIGGRAPH), Jul, 2006.

normal imagenormal imagenormal imagenormal imagecross-polarizedcross-polarizedsubsurface subsurface componentcomponent

cross-polarizedcross-polarizedsubsurface subsurface componentcomponent

polarization differencepolarization difference(primarily)(primarily)

specular componentspecular component

polarization differencepolarization difference(primarily)(primarily)

specular componentspecular component

Separating Reflectance Components withSeparating Reflectance Components withPolarization-Difference ImagingPolarization-Difference Imaging

Separating Reflectance Components withSeparating Reflectance Components withPolarization-Difference ImagingPolarization-Difference Imaging


1. Epsilon Photography– Multiphotos by varying camera parameters– HDR, panorama– ‘Ultimate camera’: (Photo-editor)

2. Coded Photography– Single/few snapshot– Reversible encoding of data– Additional sensors/optics/illum– ‘Scene analysis’ : (Next software?)

3. Essence Photography– High-level understanding

• Not mimic human eye• Beyond single view/illum

– ‘New artform’

Blind CameraBlind Camera

Sascha Pohflepp, Sascha Pohflepp, U of the Art, Berlin, 2006U of the Art, Berlin, 2006

Capturing the Essence of Visual Experience

– Exploiting online collections• Photo-tourism [Snavely2006]• Scene Completion [Hays2007]

– Multi-perspective Images• Multi-linear Perspective [Jingyi Yu, McMillan 2004]• Unwrap Mosaics [Rav-Acha et al 2008]• Video texture panoramas [Agrawal et al 2005]

– Non-photorealistic synthesis• Motion magnification [Liu05]

– Image Priors• Learned features and natural statistics• Face Swapping: [Bitouk et al 2008]• Data-driven enhancement of facial attractiveness [Leyvand et al 2008]• Deblurring [Fergus et al 2006, 2008 papers

Scene Completion Using Millions of PhotographsHays and Efros, Siggraph 2007

Andrew Davidhazy

Unwrap Mosaics + Video Editing

Rav-Acha et al Siggraph 2008

Motion Magnification

Liu, Torralba, Freeman, Durand, Adelson Siggraph 2005





– Image Priors• Learned features and natural statistics• Face Swapping: [Bitouk et al 2008]• Data-driven enhancement of facial attractiveness [Leyvand et al 2008]• Deblurring [Fergus et al 2006, 2007-2008 papers]

Face Swapping

• Find Candidate face in DB and align

• Tune pose, lighting, color and blend

• Keep result with optimized matching cost

[Bitouk et al 2008]


1. Epsilon Photography– Low-level vision: Pixels– Multi-photos by perturbing camera parameters– HDR, panorama, …– ‘Ultimate camera’

2. Coded Photography– Mid-Level Cues:

• Regions, Edges, Motion, Direct/global– Single/few snapshot

• Reversible encoding of data– Additional sensors/optics/illum– ‘Scene analysis’

3. Essence Photography– High-level understanding

• Not mimic human eye• Beyond single view/illum

– ‘New artform’

Submit your questions ..Submit your questions ..

1. Today

What makes photography hard? What moments you are not able to capture?

2. Future

What do you expect in a camera or photo-software you ‘buy’ in 2020?

Please submit by break at 3:30pmPanel Discussion at 5:10pm

Siggraph 2006 16 Computational Photography Papers

Hybrid Images• Oliva et al (MIT)

Drag-and-Drop Pasting• Jia et al (MSRA)

Two-scale Tone Management for Photographic Look

• Bae et al (MIT)

Interactive Local Adjustment of Tonal Values• Lischinski et al (Tel Aviv)

Image-Based Material Editing• Khan et al (Florida)

Flash Matting• Sun et al (Microsoft Research Asia)

Natural Video Matting using Camera Arrays• Joshi et al (UCSD / MERL)

Removing Camera Shake From a Single Photograph• Fergus (MIT)

Coded Exposure Photography: Motion Deblurring • Raskar et al (MERL)

Photo Tourism: Exploring Photo Collections in 3D• Snavely et al (Washington)

AutoCollage• Rother et al (Microsoft Research Cambridge)

Photographing Long Scenes With Multi-Viewpoint Panoramas

• Agarwala et al (University of Washington)

Projection Defocus Analysis for Scene Capture and Image Display

• Zhang et al (Columbia University)

Multiview Radial Catadioptric Imaging for Scene Capture• Kuthirummal et al (Columbia University)

Light Field Microscopy (Project)• Levoy et al (Stanford University)

Fast Separation of Direct and Global Components of a Scene Using High Frequency Illumination

• Nayar et al (Columbia University)


• Image Analysis & Enhancement• Image Deblurring with Blurred/Noisy Image Pairs • Photo Clip Art • Scene Completion Using Millions of Photographs

• Image Slicing & Stretching• Soft Scissors: An Interactive Tool for Realtime High Quality Matting • Seam Carving for Content-Aware Image Resizing • Image Vectorization Using Optimized Gradient Meshes • Detail-Preserving Shape Deformation in Image Editing

• Light Field & High-Dynamic-Range Imaging• Veiling Glare in High-Dynamic-Range Imaging • Ldr2Hdr: On-the-Fly Reverse Tone Mapping of Legacy Video and Photographs

• Appearance Capture & Editing• Multiscale Shape and Detail Enhancement from Multi-light Image Collections

• Computational Cameras• Active Refocusing of Images and Videos • Multi-Aperture Photography • Dappled Photography: Mask-Enhanced Cameras for Heterodyned Light Fields and Coded Aperture

Refocusing • Image and Depth from a Conventional Camera with a Coded Aperture

• Big Images• Capturing and Viewing Gigapixel Images • Efficient Gradient-Domain Compositing Using Quadtrees

• Video Processing• Factored Time-Lapse Video • Computational Time-Lapse Video (project page) • Real-Time Edge-Aware Image Processing With the Bilateral Grid


• Computational Photography & Display– Programmable Aperture Photography: Multiplexed Light Field Acquisition – Glare Aware Photography: 4D Ray Sampling for Reducing Glare Effects of Camera Lenses – Light-Field Transfer: Global Illumination Between Real and Synthetic Objects

• Deblurring & Dehazing– Motion Invariant Photography – Single Image Dehazing – High-Quality Motion Deblurring From a Single Image – Progressive Inter-scale and intra-scale Non-blind Image Deconvolution

• Faces & Reflectance– Data-driven enhancement of facial attractiveness – Face Swapping: Automatic Face Replacement in Photographs (Project) – AppProp: All-Pairs Appearance-Space Edit Propagation

• Image Collections & Video– Factoring Repeated Content Within and Among Images – Finding Paths through the World's Photos – Improved Seam Carving for Video Retargeting (Project) – Unwrap Mosaics: A new representation for video editing (Project)

• Perception & Hallucination– A Perceptually Validated Model for Surface Depth Hallucination – A Perception-based Color Space for Illumination-invariant Image Processing – Self-Animating Images: Illusory Motion Using Repeated Asymmetric Patterns

• Tone & Color– Edge-preserving decompositions for multi-scale tone and detail manipulation – Light Mixture Estimation for Spatially Varying White Balance

• Ramesh Raskar and Jack Tumblin

• Book Publishers: A K Peters• Siggraph 2008 booth: 20% off • Booth #821

More ..More ..• ArticlesArticles• IEEE Computer, IEEE Computer,

– August 2006 Special IssueAugust 2006 Special Issue– Bimber, Nayar, Levoy, Debevec, Cohen/SzeliskiBimber, Nayar, Levoy, Debevec, Cohen/Szeliski

• IEEE CG&A, IEEE CG&A, – March 2007 Special issueMarch 2007 Special issue– Durand and SzeliskiDurand and Szeliski

• Science News cover story Science News cover story – April 2007April 2007– Featuring : Levoy, Nayar, Georgiev, DebevecFeaturing : Levoy, Nayar, Georgiev, Debevec

• American ScientistAmerican Scientist– February 2008February 2008

• Siggraph 2008Siggraph 2008– 19 papers19 papers– HDRI, Mon/Tue 8:30amHDRI, Mon/Tue 8:30am– Principles of Appearance Acquisition and RepresentationPrinciples of Appearance Acquisition and Representation– Bilateral Filter course, Fri 8:30amBilateral Filter course, Fri 8:30am– Other courses .. (Citizen Journalism, Wedn 1:45pm)Other courses .. (Citizen Journalism, Wedn 1:45pm)

• First International Conf on Comp Photo, April 2009First International Conf on Comp Photo, April 2009– Athale, Durand, Nayar (Papers due Oct 3nd)Athale, Durand, Nayar (Papers due Oct 3nd)

Module 1: 105 minutesModule 1: 105 minutes

1:45: A.1 Introduction and Overview 1:45: A.1 Introduction and Overview (Raskar, 15 minutes)(Raskar, 15 minutes)

2:00: A.2 Concepts in Computational Photography 2:00: A.2 Concepts in Computational Photography (Tumblin, 15 minutes) (Tumblin, 15 minutes)

2:15: A.3 Optics: Computable Extensions 2:15: A.3 Optics: Computable Extensions (Raskar, 30 minutes) (Raskar, 30 minutes)

2:45: A.4 Sensor Innovations2:45: A.4 Sensor Innovations (Tumblin, 30 minutes)(Tumblin, 30 minutes)

3:15: Q & A3:15: Q & A (15 minutes)(15 minutes)

3:30: Break: 15 minutes3:30: Break: 15 minutes

Module 2: 105 minutesModule 2: 105 minutes

3:45: B.1 Illumination As Computing3:45: B.1 Illumination As Computing (Debevec, 25 minutes) (Debevec, 25 minutes)

4:10: B.2 Scene and Performance Capture4:10: B.2 Scene and Performance Capture (Debevec, 20 minutes)(Debevec, 20 minutes)

4:30: B.3 Image Aggregation & Sensible Extensions4:30: B.3 Image Aggregation & Sensible Extensions (Tumblin, 20 minutes)(Tumblin, 20 minutes)

4:50: B.4 Community and Social Impact 4:50: B.4 Community and Social Impact (Raskar, 20 minutes)(Raskar, 20 minutes)

5:10: B.4 Panel discussion 5:10: B.4 Panel discussion (All, 20 minutes) (All, 20 minutes)

Class Page : Class Page : http://ComputationalPhotography.orghttp://ComputationalPhotography.org

Class: Class: Computational Photography, Advanced TopicsComputational Photography, Advanced TopicsDebevec, Raskar and TumblinDebevec, Raskar and Tumblin

mit media lab camera culture ramesh raskar jack tumblin computational photography: advanced topics...

Documents

computational photography

high dynamic range photography

image fusion

new methods

projectorbased graphics

imagebased modeling

multiflash photography

research interests