spring 2003 ibmr: image based modeling and rendering jack tumblin computer science dept.,...

Spring 2003Spring 2003

IBMR: Image Based IBMR: Image Based Modeling and RenderingModeling and Rendering

Jack Tumblin Jack Tumblin Computer Science Dept., Computer Science Dept., Northwestern UniversityNorthwestern University

[email protected]@cs.northwestern.edu

Pat Hanrahan, 1999

Trompe L’oeil Examples (Palo Alto CATrompe L’oeil Examples (Palo Alto CA)

2D 2D Display Display Image(s)Image(s)

IBMR Goal: IBMR Goal: BidirectionalBidirectional Rendering Rendering

• Both forward and ‘inverse’ rendering!Both forward and ‘inverse’ rendering!

Camera PoseCamera PoseCamera View GeomCamera View GeomScene illuminationScene illuminationObject shape, positionObject shape, positionSurface reflectance,… Surface reflectance,… transparencytransparency




IBMRIBMR

TraditionalTraditionalComputerComputerGraphicsGraphics

‘‘3D Scene’ 3D Scene’ DescriptionDescription ‘‘Optical’ Optical’

DescriptionDescription

?! New ?! New Research?Research?

IBMR Apps : Why Bother?IBMR Apps : Why Bother?

• Movie Special FX: Mix real & syntheticMovie Special FX: Mix real & synthetic

• Augmented Photography/Video:Augmented Photography/Video:– Any Camera Position, Any Camera Position, – Any Lighting, Any Lighting, – Any moment in TimeAny moment in Time

• Historical Preservation/RestorationHistorical Preservation/Restoration

• The Persistent, Tantalizing CG/Vision Failures: The Persistent, Tantalizing CG/Vision Failures: Digital/Visual Libraries, 3D Mo-Cap, VR, Digital/Visual Libraries, 3D Mo-Cap, VR, haptics, telepresence, Avatar I/O, Facial ID,...haptics, telepresence, Avatar I/O, Facial ID,...

Synthetic Image + Depth = Surface?Synthetic Image + Depth = Surface?

““Hole-filling”Hole-filling”

Problem;Problem;(incomplete,(incomplete,

unevenuneven

surface data)surface data)

ShadingShading

Problem;Problem;need need

reflectance reflectance w/ow/o

lighting,... lighting,...

McMillan 1996McMillan 1996

Do More pictures (or photos) Help?Do More pictures (or photos) Help?

Does Motion & Video Help?Does Motion & Video Help?

time

time

J. Shade et al., 1998

*OR* Can Images Help Rendering?*OR* Can Images Help Rendering?

““Billboards”Billboards”in the distance,in the distance,or “impostors”or “impostors”instead of instead of complexcomplexgeometry?geometry?

-Sillion et. al 1999-Sillion et. al 1999

““Image-Based Shadows”Image-Based Shadows”Agrawala2000Agrawala2000

Yes,Yes,

For some For some limited butlimited butvery toughvery toughproblems...problems...

““Deep Shadow Maps”Deep Shadow Maps”Lokovic2000Lokovic2000

Yes,Yes,

For some For some limited butlimited butvery toughvery toughproblems...problems...

Early IBR: QuickTime VR Early IBR: QuickTime VR (Chen, Williams ’93)(Chen, Williams ’93)

Four Planar Images Four Planar Images 1 Cylindrical Panorama 1 Cylindrical Panorama

User-steered, real-time view in any directionUser-steered, real-time view in any direction

Early IBR: QuickTime VR Early IBR: QuickTime VR (Chen, Williams ’93)(Chen, Williams ’93)

2) Windowing, Horizontal-only Reprojection:2) Windowing, Horizontal-only Reprojection:

IN:IN:

OUT:OUT:

View Interpolation: How?View Interpolation: How?

• But how can we DISPLACE the camera?But how can we DISPLACE the camera?

• What if no depth is available?What if no depth is available?• Traditional Stereo Disparity Map: Traditional Stereo Disparity Map:

pixel-by-pixel search for pixel-by-pixel search for correspondencecorrespondence (an unreliable mess) (an unreliable mess)

Plenoptic Array: ‘The Matrix Effect’Plenoptic Array: ‘The Matrix Effect’

• Brute force!Brute force!Simple arc, line, or ring array of camerasSimple arc, line, or ring array of cameras

• Synchronized shutter Synchronized shutter http://www.http://www.ruffyruffy.com/.com/firinglinefiringline.html.html

• Warp/blend between images to Warp/blend between images to change viewpoint on ‘time-frozen’ scene:change viewpoint on ‘time-frozen’ scene:

Seitz: ‘View Morphing’ SIGG`96Seitz: ‘View Morphing’ SIGG`96

• http://www.cs. http://www.cs.washingtonwashington.edu/homes/.edu/homes/seitzseitz//vmorphvmorph//vmorphvmorph..htmhtm

1)Manually set some 1)Manually set some

corresp.points corresp.points

(eye corners, etc.)(eye corners, etc.)

2) pre-warp and2) pre-warp and

post-warp to matchpost-warp to match

points in 3D,points in 3D,

3) Reproject for3) Reproject for

Virtual camerasVirtual cameras

A A MostlyMostly 2-D Method2-D Method

Seitz: ‘View Morphing’ SIGG`96Seitz: ‘View Morphing’ SIGG`96

• http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htmhttp://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm


Malzbender, HPlabs 2001


Polynomial Texture MapsPolynomial Texture Maps

Store just Store just 66 coefficientscoefficients

at each pixel, at each pixel, get Interactive get Interactive

re-lighting...re-lighting...

• for a given scene, describe for a given scene, describe ALL rays through ALL rays through – ALLALL pixels, of pixels, of – ALLALL cameras, at cameras, at – ALLALL wavelengths, wavelengths, – ALLALL time time

F(F(x,y,zx,y,z,, ,,,, , , tt))““Eyeballs Everywhere” Eyeballs Everywhere”

function (5-D x 2-D!)function (5-D x 2-D!)

5-D Plenoptic Function 5-D Plenoptic Function (Adelson, Bergen `91)(Adelson, Bergen `91)

………………………………

…………………………

‘‘Scene’ causes 4D Light FieldScene’ causes 4D Light Field

Light field: holds all Light field: holds all outgoingoutgoing light rays light rays

Shape, Shape, Position,Position,Movement,Movement,

BRDF,BRDF,Texture,Texture,ScatteringScattering

Emitted Emitted LightLight

Reflected,Reflected,Scattered,Scattered,Light …Light … Cameras capture Cameras capture

subset of these subset of these rays.rays.

Levoy et al.,“Light Fields” 1996

? Can we recover ? Can we recover Shape Shape ? ?

Can you find ray intersections? Or ray depth?Can you find ray intersections? Or ray depth?

Ray colors might Ray colors might not match for not match for shiny materials shiny materials (BRDF)(BRDF)

? Can we recover ? Can we recover Surface Material Surface Material ??

Can you find ray intersections? Or ray depth?Can you find ray intersections? Or ray depth?

Ray colors might Ray colors might not match for not match for shiny materials shiny materials (BRDF)(BRDF)

Hey, wait…Hey, wait…

• ‘‘Light field’ describes light Light field’ describes light LEAVINGLEAVING the the enclosing surface….enclosing surface….

• ? Isn’t there a complementary ‘light field’ for ? Isn’t there a complementary ‘light field’ for the light the light ENTERINGENTERING the surface? the surface?

• *YES* *YES* ‘Image-Based Lighting’ too! ‘Image-Based Lighting’ too!

• ? How can we capture incoming light?? How can we capture incoming light?

A Mirror Sphere is...A Mirror Sphere is...

A panoramic camera A panoramic camera converter, andconverter, and

A A ‘light probe’‘light probe’ to measure to measure ALLALL incoming light at a point. incoming light at a point.

How can we use it?How can we use it?

A Mirror Sphere is...A Mirror Sphere is...

Image-Based Actual Re-lightingImage-Based Actual Re-lighting

Image-Based Actual Re-lightingImage-Based Actual Re-lighting

Film the background in Milan,Film the background in Milan,Measure incoming light,Measure incoming light,

Light the actress in Los AngelesLight the actress in Los Angeles

Matte the backgroundMatte the background

Matched LA and Milan lighting.Matched LA and Milan lighting.

Debevec et al., SIGG2001

Measure REAL light in a REAL scene...Measure REAL light in a REAL scene...

Debevec et al.,SIGG1998

Render FAKE objects with REAL light,Render FAKE objects with REAL light,

Debevec et al.,SIGG1998

And combine with REAL image:And combine with REAL image:

• Cleaner Formulation:Cleaner Formulation:– Orthographic camera,Orthographic camera,– positioned on sphere positioned on sphere

around object/scenearound object/scene– Orthographic projector,Orthographic projector,– positioned on spherepositioned on sphere

around object/scenearound object/scene

F(F(xxcc,y,ycc,,cc,,cc,x,xll,y,yll ll,,ll,, , t, t))

‘‘Full 8-D Light Field’ Full 8-D Light Field’ (10-D, actually: time, (10-D, actually: time, ))

cameracamera




F(F(xxcc,y,ycc,,cc,,cc,x,xll,y,yll ll,,ll,, , t, t))


cameracamera

cc

cc




F(F(xxcc,y,ycc,,cc,,cc,,xxll,y,yll ll,,ll,, , t, t))


cameracamera

Projector (laser brick)Projector (laser brick)



around object/scenearound object/scene– (and wavelength and time)(and wavelength and time)

F(F(xxcc,y,ycc,,cc,,cc,,xxll,y,yll ll,,ll,, , t, t))


cameracamera

projectorprojector


• ! Complete !! Complete !– Geometry, Lighting, BRDF,…Geometry, Lighting, BRDF,…it’s it’s allall there, there, – It’s all LINEAR,It’s all LINEAR,– But it’s an But it’s an 8-D8-D function! function! Stupendously Huge!Stupendously Huge!

(40GB data sets, 32 hours / object…)(40GB data sets, 32 hours / object…)

• What Subsets Are Useful?What Subsets Are Useful?– Fixed camera, vary lighting? compositing?Fixed camera, vary lighting? compositing?– Fixed lighting, vary camera? animation?Fixed lighting, vary camera? animation?– Recover some shape? silhouettes? Recover some shape? silhouettes? – Recover some reflectance? Recover some reflectance? – Recover some new hybrid?Recover some new hybrid?

Image-Based Synthetic Re-lightingImage-Based Synthetic Re-lighting

Masselus et al., 2002Masselus et al., 2002

Basis imagesIlluminant direction

estimationTake pictures and move hand-held light source

Incidentlight mapCreate weighted sum

…

· W1 + · W2 +

+ · Wn

Relit object

Masselus et al., 2002,Masselus et al., 2002, “Free-Form Light Stage” “Free-Form Light Stage”

Rushmeier, 2001

Fine geometric Details Fine Texture/Normal Details

Image-Based Shape RefinementImage-Based Shape Refinement

Matusik 2002

Matusik 2002

Image-Based Shape ApproximationImage-Based Shape Approximation

Matusik 2002

Image-Based Shape ApproximationImage-Based Shape Approximation

Practical IBMR Practical IBMR (so far...)(so far...)

What useful partial solutions are possible?What useful partial solutions are possible?

• Texture Maps++: Panoramas, Env. MapsTexture Maps++: Panoramas, Env. Maps

• Image(s)+Depth: (3D shell) Image(s)+Depth: (3D shell)

• EstimatingEstimating Depth & Recovering Silhouettes Depth & Recovering Silhouettes

• ‘‘Light Probe’ measures real-world lightLight Probe’ measures real-world light

• Structured Light to Recover Surfaces…Structured Light to Recover Surfaces…

• Hybrids: BTF, stitching, …Hybrids: BTF, stitching, …

IBMR ChallengesIBMR Challenges

• Bulky Data; High-Dimensionality!Bulky Data; High-Dimensionality!

• Representation, Compression, Fast accessRepresentation, Compression, Fast access

• Editing, Control, FlexibilityEditing, Control, Flexibility

• Self-movement, Animation ?Self-movement, Animation ?

• Complex interreflections, scattering,...Complex interreflections, scattering,...

• Compositing, Self-occlusion, lighting/view Compositing, Self-occlusion, lighting/view pose, pose,

• concavities, geometry/detail, resolution ...concavities, geometry/detail, resolution ...

ConclusionConclusion

• Heavy overlap with computer vision: Heavy overlap with computer vision: careful not to re-invent & re-name!careful not to re-invent & re-name!

• Elegant Geometry is Elegant Geometry is at the heart of it allat the heart of it all, , even surface reflectance, illumination, etc. even surface reflectance, illumination, etc. etc. 8-10 dimensional, but linear functions.etc. 8-10 dimensional, but linear functions.

• Melting barrier between real, synthetic 3D.Melting barrier between real, synthetic 3D.

IBMR: Visiting Lecturers IBMR: Visiting Lecturers

3D Scanning for 3D Scanning for

Cultural Heritage ApplicationsCultural Heritage Applications Holly Rushmeier, IBM TJ WatsonHolly Rushmeier, IBM TJ Watson

Friday May 16 3:00pm, Rm 381, CS Dept. Friday May 16 3:00pm, Rm 381, CS Dept.

?Subsurface Scattering? ?Subsurface Scattering? ?Rendering Human Hair??Rendering Human Hair?

Steve Marschner, Cornell UniversitySteve Marschner, Cornell University Friday May 23 3:00pm, Rm 381, CS Dept.Friday May 23 3:00pm, Rm 381, CS Dept.

ENDEND

spring 2003 ibmr: image based modeling and rendering jack tumblin computer science dept.,...

Documents

d methodseitz

d methodmalzbender

d display imagescamera

view morphing sigg

d mocap

d light fieldlight field

realtime view

view interpolation