5/1/2000deepak bandyopadhyay / unc chapel hill 1 3d photography (image-based model acquisition)...

5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill

1

3D Photography(Image-based Model Acquisition)

Funky Image Goes Here

11/6/2000 Deepak Bandyopadhyay / 258 / 3D Photography

2

“Analog” 3D photography !

• “3D stereoscopic imaging” – been around as long as cameras have

– Use camera with 2 or more lenses (or stereo attachment)

– Use stereo viewer to create impression of 3D


3

Motivation

• Digitizing real world objects• Getting realistic models

humans

objects

places


4

3D Photography : Definition

• Sometimes called “3D Scanning”• Use cameras and light to capture the shape &

appearance of real objects• Shape == geometry (point sampling + surface

reconstruction + fairing)• Appearance == surface attributes (color/texture,

material properties, reflectance) • Final result = richly detailed model


5

Applications in Industry

• Human body / head / face scans– Avatar creation for virtual worlds

– 3d conferencing

– medical applications

– product design

– Platforms:

• Cyberware RD3030

• Others (Geomagic, Metacreations, Cyrax, Geometrix…)


6

More applications

• Historical preservation, dissemination of museum artifacts (Digital Michelangelo, Monticello, …)

• CAD/CAM (eg. Legacy motorcycle parts scanned by Geomagic for Harley-Davidson).

• Marketing (models of products on the web)• 3D games & simulation• Reverse engineering


7

Technology Overview

• The Imaging Pipeline– Real World

– Optics

– Recorder

– Digitizer

– Vision & Graphics


8

Quick Notes on Optics

• Model lenses with all their properties - aberration, distortion, flare, vignetting etc.

• We correct for some of these effects (eg. distortion) in the calibration, ignore others.

• CCD (charged coupled devices) are the most popular recording media.


9

Theory : Passive Methods

• Stereo pair matching

• Structure from motion

• Shape from shading

• Photometric stereo


10

Stereo Matching

• Stereo Matching Basics– Needs two images, like stereoscopy

– Given correspondence betweenpoints in 2 views, we can find depth by triangulation

– But correspondence is hard prob!

– A lot of literature on solving it…

• Stereo Matching output• 3D point cloud• Remove outliers and pass through surface reconstructor


11

Structure from Motion

• Camera moving, objects static• Compute camera motion and object geometry from

motion of image points

• Assumption - orthographic projn (use telephoto)

• If: world origin = 3D centroid camera origin = 2D centroid

Then: camera translation drops out


12


• Camera moving, objects static• Compute camera motion and object geometry from

motion of image points


13


• Factorization [Tomasi & Kanade, 92]• Find M, S using Singular Value Decomposition of W.

SVD gives:

S’S modulo linear transform A. Solve for A using constraints on M.


14

More methods

• Shape from shading, [Horn]– Invert Lambert’s Law (L=I k cos )

knowing the intensity at image pointto solve for normal

• Photometric stereo [Woodham]– An extension of the above

– Two or more images under different illumination conditions.

– Each image provides one normal

– Three images provide unique solution for a pixel.


15

Active Sensing

• Passive methods (eg. stereo matching) suffer from ambiguities - many similar regions in an image correspond to a point in the other.

• Project known / regular pattern (“structured light”) into scene to disambiguate

• get precise reconstruction by combining views– Laser rangefinder

– Projectors and imperceptible structured light


16

Desktop 3D PhotographyJean-Yves Bouguet, Pietro Perona

• An active sensing technique using “weak structured lighting”

• Need: camera, lamp, chessboard, pencil, stick• Idea:

– Light object with lamp & aim camera at it

– Move stick around & capture shadow sequence

– Use image of deformed shadow to calc 3D shape


17

Desktop 3D PhotographyJean-Yves Bouguet, Pietro Perona

• Computation of 3d position from the plane of light source, stick and shadow


18

Volumetric MethodsChevette Project, Debevec, 1991


19

Voxel Models from Images• When there are 2 colors in the image - use volume

intersection [Szeliski 1993]– Back-project silhouettes from camera views & intersect


20

Voxel Models from Images

• With more colors but constrained viewpoints, we use voxel coloring [Seitz & Dyer, 1997]– Choose a voxel & project to it from all views

– Color if enough matches

– Prob - determining visibilityof a point from a view

– Solution - depth orderedtraversal using a “view indep.d.o.” (dist from separating plane)


21

Voxel Models from Images

• A view-independent depth order may not exist (for some configuration of viewpoints / scene geometry).

• Use Space Carving [Kutulakos & Seitz, 1998]– Computes 3D (voxel) shape from multiple color photos

– Computes “maximally photo-consistent shape”• maximal superset of all 3D shapes that produce the given photos


22

Space Carving

• Algorithm:a) Initialize V to volume

containing true scene

b) For each voxel, • check if photo-consistent

• if not, remove (“carve”) it.

• Can be shown to converge to maximal photo-consistent scene (union of all photo-consistent scenes).


23

Space Carving : Results• House walkthru - 24 rendered input views

• Results best as seen from one of the original views


24

Modeling from a single view(Criminisi et al, 1999)

• Compute 3D affine measurements of the scene

from single perspective image

• Use minimal geom info – vanishing line for a pencil of

planes || to reference plane

– vanishing point of parallel

lines along a direction

outside reference plane


25

Modeling from a single view(Criminisi et al, 1999)

• Compute “ratio of parallel distances”

• Creating a 3D model from a photograph– horizontal lines used to compute vanishing line

– parallel vertical lines used to compute vanishing point

• Can generate geometrically correct model from a

Renaissance painting (with correct perspective)


26

Extracting color, reflectance

• Photographs have lighting/shading effects that we estimate (reflectance function) and compensate for (specular highlight removal) or change (relighting)

• Work of Paul Debevec & others at Berkeley (acquiring reflectance field)

• Wood et al at U. Washington (surface light lield for 3D photography)


27

Surface Light Field[Wood et al, 2000]

• A 4D function on the surface - at surface parameter (u,v), for every direction (,), stores the color.

• Fixed illumination conditions.• Photographs taken from a lot of different

directions sample the surface light field.• Continuous function (piecewise linear over ,)

estimated by pointwise fairing.


28

Reflectance from Photographs (Yu, Debevec et al, 1999)

• Estimating reflectance for entire scenes– Too general a problem, parameterize thus:

• Assume surface can be divided into patches• Diffuse reflectance function (albedo), varies across a patch

• Specular reflectance function taken as const across a region

• Assume known lighting, calib, geometry known• Approach - Inverse Global Illumination

– Estimate BRDF for direct illumination - f(u,v,,)


29

Reflectance from Photographs (Yu, Debevec et al, 1999)

• Inverse Global Illumination– Known Li (measure), Ii (calc fm known light sources) at

every pixel

– Estimate BRDF for direct illumination - f(u,v,i,i,r,r)

• Write BRDF as a constant diffuse term and a specular term which is a function of incoming & outgoing and roughness.

• Solve for the constants(d, s,)

• For indirect illumination - estimate the parameters (and indirect illumination coeffs with other patches) iteratively


30

Case study - FaçadeDebevec, Taylor & Malik, 1996

• Modeling architectural scenes from photographs• Not fully automatic (user inputs blocky 3D model)

– Using blocks leads to fewer params in architectural models

• User marks corresponding features on photo• Computer solves for block size, scale, camera rotation

by minimizing error of corresponding features

• Reprojects textures from the photographs onto the reconstructed model


32

Arches andSurfaces of Revolution

Taj Mahalmodeled from

one photograph


33

Case study - Digital Michelangelo Project

• 3D scanning of large statues (SIGGRAPH 00)

• Separate geometry and color scans– custom rig : laser scanner & camera mounted concurrently

• Range scan post-processing– Combine range scans from different positions

• Use volumetric modeling methods (Curless, Levoy 1996)

– Fill holes using space carving


34

Case study - Digital Michelangelo Project

• Color scan processing– Compensate for ambient lighting

• subtract image with & without spotlight

– Subtract out shadows & specularities

– find surface orientation (inverse lighting computation)

– convert color to RGB reflectance (acquire light field)• Using estimated BRDF of marble

• modeling subsurface scattering


35

Digital MichelangeloScanning a large object

• calibrated motions– pitch (yellow)– pan (blue)– horizontal translation (orange)

• uncalibrated motions– vertical translation– remounting the scan head– moving the entire gantry


36

References• [Bouguet98] Bouguet, J.-Y., P. Perona. 3D Photography on your Desk. In Proc.

ICCV 1998 • [Bouguet00] Bouguet, J.-Y. Presentation on Desktop 3D Photography, in

SIGGRAPH course notes on 3D Photography, 2000• [Criminisi99] Criminisi, A., I. Reid and A. Zisserman. Single View Metrology.

In Proc. ICCV, pp 434-442, September 1999• [Curless96] Curless, B. and M. Levoy. A Volumetric Method for Building

Complex Models from Range Images. In Proc. SIGGRAPH 1996• [Debevec96] Debevec, P., C. Taylor and J. Malik. Façade - Modeling and

Rendering Architectural Scenes from Photographs. In Proc. SIGGRAPH 1996• [Debevec00a] Debevec, P. Presentation on the Façade, from SIGGRAPH course

notes on 3D Photography, 1999, 2000.• [Debevec00b] Debevec, P., T. Hawkins, C. Tchou, H.P.Duiker, W. Sarokin and

M. Sagar. Acquiring the Reflectance Field of a Human Face. In Proc. SIGGRAPH 2000.


37

More References• [Horn70] Horn, B.K.P. Shape from Shading : A Method for Obtaining the

Shape of a Smooth Opaque Object from One View. Ph.D. Thesis, Dept of EE, MIT, 1970.

• [Kutulakos98] Kutulakos, K. N. and S. Seitz. A Theory of Shape by Space Carving. URCS TR#692, May 1998, appeared in Proc. ICCV 1999.

• [Levoy96] Levoy, M. and P. Hanrahan. Light Field Rendering. In Proc. SIGGRAPH 1996.

• [Levoy00a] Levoy, M., Pulli, K., Curless, B. et al. The Digital Michelangelo Project - 3D Scanning of Large Statues. In Proc. SIGGRAPH 2000.

• [Levoy00b] Levoy, M. Presentation on the Digital Michelangelo Project, in SIGGRAPH course notes on 3D Photography, 2000.

• [Seitz97] Seitz & Dyer. Photorealistic Scene Reconstruction by Voxel Coloring. In Proc. CVPR 1997, pp. 1067-1073.


38

Still More References

• [Seitz00] Seitz, S. SIGGRAPH course notes on 3D photography, 1999, 2000.

• [Szeliski93] Szeliski, R. Rapid Octree Construction from Image Sequences. CGVIP : Image Understanding, vol. 58, no. 1, pp 23-32, 1993.

• [Wood00] Wood, D., D. I. Azuma, K. Aldinger, B. Curless, T. Duchamp, D.H. Salesin and W. Stuetzle. Surface Light Fields for 3D Photography. In Proc. SIGGRAPH 2000.

• [Woodham80] Woodham, R. Photometric Stereo for Determining Surface Orientation from Multiple Images. Journal of Optical Engineering, vol. 19, no. 1, pp 138-144, 1980.

• [Yu99] Yu, Y., P. Debevec, J. Malik and T. Hawkins. Inverse Global Illumination - Recovering Reflectance Models of Real Scenes from Photographs.

5/1/2000deepak bandyopadhyay / unc chapel hill 1 3d photography (image-based model acquisition)...

Documents

d slide

d photography6

d photography4

d photography13 structure

d photography11 structure

d photography12 structure

d photography5 applications

d point cloud