CS 395: Adv. Computer GraphicsCS 395: Adv. Computer Graphics

Overview: Image-Based Overview: Image-Based Modeling and RenderingModeling and Rendering

Jack TumblinJack Tumblin

jet@cs.northwestern.edujet@cs.northwestern.edu

GOAL: First-Class PrimitiveGOAL: First-Class Primitive

• Want images as ‘first-class’ primitivesWant images as ‘first-class’ primitives– Useful as BOTH Useful as BOTH inputinput and and outputoutput– ConvertConvert to/from traditional scene descriptions to/from traditional scene descriptions

• Want to mix real & synthetic scenes freelyWant to mix real & synthetic scenes freely• Want to extend photographyWant to extend photography

– Easily capture scene:Easily capture scene:shape, movement, surface/BRDF, lighting …shape, movement, surface/BRDF, lighting …

– Modify & Render the captured scene dataModify & Render the captured scene data

• --BUT----BUT-- • images hold only PARTIAL scene informationimages hold only PARTIAL scene information

– You can’t always get what you want”You can’t always get what you want” –(Mick Jagger 1968)

• for a given scene, describe:for a given scene, describe:– ALLALL rays through rays through – ALLALL pixels, of pixels, of – ALLALL cameras, at cameras, at – ALLALL wavelengths, wavelengths, – ALLALL time time

F(F(x,y,zx,y,z,, ,,,, , , tt))““Eyeballs Everywhere” Eyeballs Everywhere”

function (5 or 7-D!)function (5 or 7-D!)

Plenoptic Function Plenoptic Function (Adelson, Bergen `91)(Adelson, Bergen `91)

………………………………

…………………………

‘‘Scene’ causes Light FieldScene’ causes Light Field

Light field: holds all Light field: holds all outgoingoutgoing light rays light rays

Shape, Shape, Position,Position,Movement,Movement,

BRDF,BRDF,Texture,Texture,ScatteringScattering

Emitted Emitted LightLight

Reflected,Reflected,Scattered,Scattered,Light …Light … Scene modulates Scene modulates

outgoing light; outgoing light; light field light field captures it all.captures it all.

A Big Light Field Question:A Big Light Field Question:

Image entraps a Image entraps a partialpartial scene description,… scene description,… – Computer Vision problem: 3D->2DComputer Vision problem: 3D->2D– Image point Image point scene surface point (usually) scene surface point (usually)– Occlusion hides some scene surfacesOcclusion hides some scene surfaces– (BRDF * irradiance) tough to split apart!(BRDF * irradiance) tough to split apart!

?? Does Plenoptic fcn. contain Does Plenoptic fcn. contain fullfull scene scene ??– Exhaustive record of all image raysExhaustive record of all image rays– Even SIMPLEST scene is huge, redundant,Even SIMPLEST scene is huge, redundant,– The ‘consequences’ of all possible renderings*The ‘consequences’ of all possible renderings*

soso

A Big Light Field Answer:A Big Light Field Answer:

Image entraps a Image entraps a partialpartial scene description scene description – Many-to-One map; 3D->2DMany-to-One map; 3D->2D– Occlusion hides some scene featuresOcclusion hides some scene features– (BRDF * irradiance)(BRDF * irradiance) tough to split! tough to split!– limited resolutionlimited resolution

?? Does Plenoptic fcn. contain Does Plenoptic fcn. contain fullfull scene scene ??

Two Options for light field methods: Two Options for light field methods: – Find a limited subset of scene info,Find a limited subset of scene info,– Use Use MOREMORE than plenoptic function data: (vary lights, etc.) than plenoptic function data: (vary lights, etc.)

!NO!!NO!

Shape Problems: CorrespondenceShape Problems: Correspondence

Can you find ray intersections? Or ray depth?Can you find ray intersections? Or ray depth?

Correspondence Correspondence Problem:Problem:Ray colors might not Ray colors might not match for non-diffuse match for non-diffuse materials (BRDF)materials (BRDF)

Shape Problems: CorrespondenceShape Problems: Correspondence

Can you find ray intersections? Or ray depth?Can you find ray intersections? Or ray depth?

Correspondence Correspondence Problem:Problem:Ray colors might not Ray colors might not match for non-diffuse match for non-diffuse materials (BRDF)materials (BRDF)

It gets worse…It gets worse…

A ‘Circular problem’: A ‘Circular problem’:

PLUS! PLUS! depth-of-focus, sampling, indirect illum…depth-of-focus, sampling, indirect illum…

ShapeShape BRDFBRDF

IrradianceIrradiance

SurfaceSurfaceNormalNormal

8-to-10-Dimensional Ideal?8-to-10-Dimensional Ideal?

Light field(4D) + light sources(4D) + time + Light field(4D) + light sources(4D) + time +

Shape, Shape, Position,Position,Movement,Movement,

BRDF,BRDF,Texture,Texture,ScatteringScattering

Emitted Emitted LightLight

Reflected,Reflected,Scattered,Scattered,Light …Light …

• Linking illumination rays to Linking illumination rays to light field rays might do it;light field rays might do it;

• Swaps ‘correspondence’ Swaps ‘correspondence’ problem for ‘massive data problem for ‘massive data problem’.problem’.

• Shortcuts, simplifications?Shortcuts, simplifications?

F(F(xxcc,y,ycc,,cc,,cc,,xxll,y,yll ll,,ll,, , t, t))

Full Plenoptic Fcn? MAYBE...Full Plenoptic Fcn? MAYBE...

cameracamera

projectorprojector

Practical IBMRPractical IBMR

What useful partial solutions are possible?What useful partial solutions are possible?

• Texture Maps++: Texture Maps++:

• Image(s)+Depth: (3D shell) Image(s)+Depth: (3D shell)

• Estimating Depth & SilhouettesEstimating Depth & Silhouettes

• ‘‘Light Probe’ measures real-world lightLight Probe’ measures real-world light

• Light control measures BRDF Light control measures BRDF

• Hybrids: BTF, stitching, …Hybrids: BTF, stitching, …

Texture Maps ++Texture Maps ++

Re-use rendering results:Re-use rendering results:

‘‘Impostors’, ‘Billboards’, ‘3D sprites’Impostors’, ‘Billboards’, ‘3D sprites’

• Render portion of scene as a textureRender portion of scene as a texture

• Apply to mesh or plane Apply to mesh or plane to C.O.P.; to C.O.P.;

• Replace if eyepoint changes too muchReplace if eyepoint changes too much

Images + DepthImages + Depth

• 1 Image + Depth: a ‘thin shell’1 Image + Depth: a ‘thin shell’– Reprojection (well known); Z-buffers can helpReprojection (well known); Z-buffers can help– McMillan`95: 4-way raster ensures depth orderMcMillan`95: 4-way raster ensures depth order– Problem: ‘holes’, occlusion, matchingProblem: ‘holes’, occlusion, matching

• Multiple Images: Multiple Images: – LDI, LDI trees for multiresolutionLDI, LDI trees for multiresolution

• Limitations: Limitations: – Presumes diffuse-only environmentPresumes diffuse-only environment– Depth capture tough: laser TOF reflectometer, Depth capture tough: laser TOF reflectometer,

manual scanner, structured light, or …manual scanner, structured light, or …

Estimating Depth, SilhouettesEstimating Depth, Silhouettes

Mildly new IBMR methods can help…Mildly new IBMR methods can help…• Sparse, manual image correspondences (Debevec, Seitz,)Sparse, manual image correspondences (Debevec, Seitz,)• Video sequences with camera motion trackingVideo sequences with camera motion tracking• Image (silhouette)-based Visual Hulls, ‘voxel carving’ Image (silhouette)-based Visual Hulls, ‘voxel carving’ (VIDEO!)(VIDEO!)

Mostly a Classic Computer Vision Problem:Mostly a Classic Computer Vision Problem:• Epipolar Geometry: reduce search for correspondencesEpipolar Geometry: reduce search for correspondences• Global & local tracking & alignment methods…Global & local tracking & alignment methods…

Light Probe: Irradiance EstimateLight Probe: Irradiance Estimate

• Place mirrored ball in scene,Place mirrored ball in scene,• Photograph (careful! High contrast image!)Photograph (careful! High contrast image!)• Unwrap: map image positionUnwrap: map image positionangleangle• Use as illumination sourceUse as illumination source

•Uses: Uses: –mixing real & synthetic objects mixing real & synthetic objects (Ward 96)(Ward 96)–separating reflectance & illum separating reflectance & illum (Yu 97)(Yu 97)–movies, movies, movies...movies, movies, movies...

Light Stage: Debevec2001Light Stage: Debevec2001

• Carefully control incoming light direction (light stages, Carefully control incoming light direction (light stages, whirling banks of lights, etc)whirling banks of lights, etc)

• Images vs. light(Images vs. light(,,))• Weighted sum of imagesWeighted sum of imagesweighted sum of lightsweighted sum of lights

Debevec et al. 2001

Light Stage Methods Light Stage Methods

• Specular Component is PolarizedSpecular Component is Polarized• Scan surface geometry (before, during)Scan surface geometry (before, during)• Scattered data interpolation to approx. BRDF.Scattered data interpolation to approx. BRDF.

Debevec et al. 2001

Light Control MethodsLight Control Methods

• light probe gathers illum,light probe gathers illum,

• light stage gathers face response to light;light stage gathers face response to light;

• Real face artificially inserted into real sceneReal face artificially inserted into real scene

ConclusionConclusion

• Very active areaVery active area

• Heavy overlap with computer vision: Heavy overlap with computer vision: careful not to re-invent & re-name!careful not to re-invent & re-name!

• Compute-intensive, but easily parallel; Compute-intensive, but easily parallel; applies graphics hardware to broader probs.applies graphics hardware to broader probs.