cs 395: adv. computer graphics light fields and their approximations jack tumblin...

CS 395: Adv. Computer GraphicsCS 395: Adv. Computer Graphics

Light Fields and Light Fields and

their Approximationstheir Approximations

Jack TumblinJack Tumblin

[email protected]@cs.northwestern.edu

• for a given scene, describe:for a given scene, describe:– ALLALL rays through rays through – ALLALL pixels, of pixels, of – ALL cameras, at ALL cameras, at – ALL wavelengths, for ALL wavelengths, for – ALL timeALL time

F(F(x,y,zx,y,z,, ,,,, , , tt))

Plenoptic Function Plenoptic Function (Adelson, Bergen `91)(Adelson, Bergen `91)


• for a given scene, describe:for a given scene, describe:– ALLALL rays through rays through – ALLALL pixels, of pixels, of – ALLALL cameras, at cameras, at – ALL wavelengths, for ALL wavelengths, for – ALL timeALL time

F(F(x,y,zx,y,z,, ,,,, , , tt))


• for a given scene, describe:for a given scene, describe:– ALLALL rays through rays through – ALLALL pixels, of pixels, of – ALLALL cameras, at cameras, at – ALLALL wavelengths, wavelengths, – ALLALL time time

F(F(x,y,zx,y,z,, ,,,, , , tt))


• ! Complete !! Complete !– Geometry, Lighting, BRDF,… Geometry, Lighting, BRDF,… ! NOT REQUIRED !! NOT REQUIRED !

• Preposterously Huge: Preposterously Huge: (?!?! (?!?! 7-D7-D function sampled at image resolution !?!?), function sampled at image resolution !?!?), but but

• Hugely Redundant:Hugely Redundant:– WavelengthWavelengthRGB triplets, ignore Time, and Restrict RGB triplets, ignore Time, and Restrict

eyepoint movement: maybe ~3 or 4D ?eyepoint movement: maybe ~3 or 4D ?– Very Similar images—use Warping rules?Very Similar images—use Warping rules?– Exploit Movie Storage/Compression MethodsExploit Movie Storage/Compression Methods

The Big Idea: The Big Idea: Image-Based Modeling & RenderingImage-Based Modeling & Rendering

• Computer Graphics:Computer Graphics: given scene description, given scene description, synthesizesynthesize images images

• Digital Photography:Digital Photography: given scene, given scene, capturecapture images images

• Computer Vision:Computer Vision:

given images, assumptions,… given images, assumptions,… (& hope)(& hope) describedescribe scenescene

• Image-Based Modeling & Rendering:Image-Based Modeling & Rendering: Collect images to estimate Plenoptic Function Collect images to estimate Plenoptic Function

QUICKLY interpolate / recallQUICKLY interpolate / recall images images(more storage, but less computing…)(more storage, but less computing…)

OLDEST IBR: Shadow Maps OLDEST IBR: Shadow Maps (1984)(1984)

Fast Shadows from Z-buffer hardware:Fast Shadows from Z-buffer hardware:1) Make the “Shadow Map”:1) Make the “Shadow Map”:

– Render image seen at light source, Render image seen at light source, BUTBUT– Keep ONLY the Z-buffer values (depth)Keep ONLY the Z-buffer values (depth)

2) Render Scene from Eyepoint:2) Render Scene from Eyepoint:– Pixel + Z depth gives 3D position of surface;Pixel + Z depth gives 3D position of surface;– Project 3D position into Shadow mapProject 3D position into Shadow map– If Shadow Map depth < 3D depth, SHADOW!If Shadow Map depth < 3D depth, SHADOW!

MANY sampling hacks (See Eric Haines) MANY sampling hacks (See Eric Haines) nVidia / OpenGL implementations available nVidia / OpenGL implementations available

http://developer.nvidia.com/view.asp?IO=Shadow_Maphttp://developer.nvidia.com/view.asp?IO=Shadow_Map

Early IBR: QuickTime VR Early IBR: QuickTime VR (Chen, Williams ’93)(Chen, Williams ’93)

1) Four Planar Images 1) Four Planar Images 1 Cylindrical Panorama: 1 Cylindrical Panorama:Re-sampling Required!Re-sampling Required!

Planar Pixels:Planar Pixels: equal distance on x,y plane (tan equal distance on x,y plane (tan -1-1)) Cyl. Pixels: Cyl. Pixels: horiz: equal angle on cylinder (horiz: equal angle on cylinder ())

vert: equal distance on y (tanvert: equal distance on y (tan-1-1))


1) Four Planar Images 1) Four Planar Images 1 Cylindrical Panorama: 1 Cylindrical Panorama:

IN:IN:

OUT:OUT:


2) Windowing, Horizontal-only Reprojection:2) Windowing, Horizontal-only Reprojection:

IN:IN:

OUT:OUT:

Plenoptic Array: ‘The Matrix Effect’Plenoptic Array: ‘The Matrix Effect’

• Simple arc, line, or ring array of camerasSimple arc, line, or ring array of cameras• Synchronized shutterSynchronized shutter• Warp/blend between images to Warp/blend between images to

change viewpoint on ‘time-frozen’ scene:change viewpoint on ‘time-frozen’ scene:• http://www.http://www.ruffyruffy.com/.com/firinglinefiringline.html.html

View Interpolation: How?View Interpolation: How?

• Traditional Stereo Disparity Map: Traditional Stereo Disparity Map: pixel-by-pixel search for pixel-by-pixel search for

correspondencecorrespondence

(Lighter: more(Lighter: moreHoriz. Separation)Horiz. Separation)

View Interpolation: How?View Interpolation: How?

• Store Depth at each pixel: reprojectStore Depth at each pixel: reproject

• Coarse or Simple 3D model:Coarse or Simple 3D model:

View InterpolationView Interpolation

Problems:Problems:• Visibility ChangesVisibility Changes

• Disocclusion, or Disocclusion, or ‘holes in the picture’‘holes in the picture’

Partial Answers:Partial Answers:

• More PicturesMore Pictures

• Interpolate Nearby PixelsInterpolate Nearby Pixels

• Better: Multiple Samples (LDI Trees Better: Multiple Samples (LDI Trees SIGG99SIGG99))

LLviewview RRviewview

Plenoptic Modeling Plenoptic Modeling (McMillan`95)(McMillan`95)

• Alerted CGI community to notion of Alerted CGI community to notion of Plenoptic Function Plenoptic Function

• Solved ‘depth order’ problem: read images Solved ‘depth order’ problem: read images in in occlusion-compatible orderocclusion-compatible order

• Demonstrated reprojection from two Demonstrated reprojection from two spherical projectionsspherical projections

Seitz: ‘View Morphing’ SIGG`96Seitz: ‘View Morphing’ SIGG`96

• http://www.cs. http://www.cs.washingtonwashington.edu/homes/.edu/homes/seitzseitz//vmorphvmorph//vmorphvmorph..htmhtm

1)Manually set some 1)Manually set some

corresp.points corresp.points

(eye corners, etc.)(eye corners, etc.)

2) pre-warp and2) pre-warp and

post-warp to matchpost-warp to match

points in 3D,points in 3D,

3) Reproject for3) Reproject for

Virtual camerasVirtual cameras

Seitz: ‘View Morphing’ SIGG`96Seitz: ‘View Morphing’ SIGG`96

• http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htmhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

Light Fields / LumigraphsLight Fields / Lumigraphs

• Subset of Plenoptic Function:Subset of Plenoptic Function:– Let Convex Surface surround scene Let Convex Surface surround scene – Place in-facing camera at every surface pointPlace in-facing camera at every surface point– Captures Captures ALLALL viewing rays for object! viewing rays for object!– (But does NOT separate external light efx)(But does NOT separate external light efx)


Two-plane parameterization:Two-plane parameterization:– Generalized camera: TWO image planesGeneralized camera: TWO image planes– Sets angle AND positionSets angle AND position– http://research.http://research.microsoftmicrosoft.com/MSRSIGGRAPH/96/.com/MSRSIGGRAPH/96/LumigraphLumigraph..htmhtm

uu

vv

ss

tt L(u,v,s,t)L(u,v,s,t)


..

uu

Hierarchical Layered Depth ImagesHierarchical Layered Depth Images

• A ‘stack’ of unstructured 3-D sample pointsA ‘stack’ of unstructured 3-D sample points

• Stores, retrieves data at multiple resolutionsStores, retrieves data at multiple resolutions

• Eliminates redundant storageEliminates redundant storage

• Can be updated dynamically with new data Can be updated dynamically with new data (from renderer or camera)(from renderer or camera)

• Demonstrated for real-time walkthroughs…Demonstrated for real-time walkthroughs…

Hybrid Approaches Hybrid Approaches

• Simple Initial Geometry, ‘decorated’ by Simple Initial Geometry, ‘decorated’ by image-based models (Debevec`96)image-based models (Debevec`96)

• ‘‘Voxel-Carving’Voxel-Carving’

• ‘‘Image-based Visual Hull’ SIGG. 2000…Image-based Visual Hull’ SIGG. 2000…

IBR-Motivating OpinionsIBR-Motivating Opinions

““Computer Graphics: Hard”Computer Graphics: Hard”– Complex!Complex! geometry, texture, lighting, shadows, geometry, texture, lighting, shadows,

compositing, BRDF, interreflections, etc. etc., etc., …compositing, BRDF, interreflections, etc. etc., etc., …– Irregular!Irregular! Visibility,Topology, Render Eqn.,… Visibility,Topology, Render Eqn.,…– Isolated!Isolated! Tough to use real objects in CGI Tough to use real objects in CGI– Slow!Slow! compute-bound, off-line only,… compute-bound, off-line only,…

““Digital Imaging: Easy”Digital Imaging: Easy”– Simple!Simple! More quality? Just pump more pixels! More quality? Just pump more pixels!– Regular!Regular! Vectorized, compressible, pipelined… Vectorized, compressible, pipelined…– Accessible!Accessible! Use real OR synthetic (CGI) images! Use real OR synthetic (CGI) images!– Fast!Fast! Scalable, Image reuse, a path to interactivity… Scalable, Image reuse, a path to interactivity…

DifficultiesDifficulties

• Sampling / Reconstruction problemsSampling / Reconstruction problems

• Brute Force: Bandwidth & Memory HogBrute Force: Bandwidth & Memory Hog

• Easier DisplayEasier DisplayDifficult EditingDifficult Editing

• Increased Flexibility Increased Flexibility Reduced Regularity Reduced Regularity– more complex storage / retrieval, more complex storage / retrieval, – more computing, more computing, – less deterministic (cache miss, etc.)less deterministic (cache miss, etc.)

Interesting Recent WorkInteresting Recent Work

• ‘‘Surface Light Fields for 3D Photography’Surface Light Fields for 3D Photography’Wood et al., SIGG2000 Wood et al., SIGG2000 http://www.cs.washington.edu/homes/daniel/siggraph2000-slf.pdfhttp://www.cs.washington.edu/homes/daniel/siggraph2000-slf.pdf

• ‘‘Plenoptic Stitching’Plenoptic Stitching’ Aliaga et al. SIGG2001 Aliaga et al. SIGG2001 http://www.bell-labs.com/user/aliaga/sig01.http://www.bell-labs.com/user/aliaga/sig01.pdfpdf

• ‘‘Single-view Modeling’Single-view Modeling’ Seitz,CVPR2001 Seitz,CVPR2001 http://grail.cs.washington.edu/projects/svm/http://grail.cs.washington.edu/projects/svm/

cs 395: adv. computer graphics light fields and their approximations jack tumblin...

Documents