5/1/2000deepak bandyopadhyay / unc chapel hill1 painting on movable objects deepak bandyopadhyay,...

5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill

1

Painting on Movable ObjectsPainting on Movable Objects

Deepak Bandyopadhyay, UNC-CHDeepak Bandyopadhyay, UNC-CH

Ramesh Raskar, MERL Ramesh Raskar, MERL

Henry Fuchs, UNC-CHHenry Fuchs, UNC-CH


2

Dynamic Shader LampsDynamic Shader Lamps

• IntroductionIntroduction

• Related WorkRelated Work

• ImplementationImplementation

• ResultsResults


3

MotivationMotivation

For some applications, it is easier to work For some applications, it is easier to work with a physical object directly, rather than with a physical object directly, rather than using traditional displays or see-through AR using traditional displays or see-through AR HMDs.HMDs.

» Reasons: Ergonomics, Passive HapticsReasons: Ergonomics, Passive Haptics» Example application: virtual 3D paintingExample application: virtual 3D painting

We show how to implement such systems.We show how to implement such systems.


4

OverviewOverview

Projector1 Projector2

Objects

Tracker

PC


5

VideoVideo


6

ApplicationsApplications

• Testbed for AR Testbed for AR applications applications

– projectors on movable objectsprojectors on movable objects

• AR Interaction medium AR Interaction medium – ArtArt– TeleimmersionTeleimmersion– Architecture Architecture

» annotationannotation» visualizationvisualization

– Others, see paperOthers, see paper


7





• ResultsResults


8

Shader Lamps Basic IdeaShader Lamps Basic Idea

Raskar et al 1, 2001

1 Ramesh Raskar, Greg Welch, Kok-Lim Low and Deepak Bandyopadhyay. “Shader Lamps : Animating Real Objectswith Image-based Illumination”, Eurographics Rendering Workshop (EGRW 2001), London.


9

Shader Lamps Basic IdeaShader Lamps Basic Idea

Rearranging terms in optical path


10

Radiance AdjustmentRadiance Adjustment

Desired Virtual

k(x) cos(p )

d(x)2

L’(x,)= Ip (x, p)

L(x,) = F(x, , i) Li (x, i)

di

Projected Real

d ( x )2

k ( x ) cos( p )Ip (x, p) =

k (x)>0L ( x, )


11

Image based IlluminationImage based Illumination

• Basic IdeaBasic Idea» Render images and project on objectsRender images and project on objects» Multiple projectorsMultiple projectors» View and object dependent colorView and object dependent color


12

Related Work : 3D PaintingRelated Work : 3D Painting

• Hanrahan & Haeberli, 1990 Hanrahan & Haeberli, 1990 22

– Painting on 3D modelsPainting on 3D models– Screen & mouse interfaceScreen & mouse interface

• Agrawala & Beers, 1995 Agrawala & Beers, 1995 33

– Painting Scanned SurfacesPainting Scanned Surfaces» Object is staticObject is static» Tracked pointer moved on its surfaceTracked pointer moved on its surface» Separate displaySeparate display

2 Pat Hanrahan and Paul Haeberli, “Direct WYSIWYG Painting and Texturing on 3D Shapes”. SIGGRAPH’90.

3 Maneesh Agrawala, Andrew C. Beers and Marc Levoy, “3D Painting on Scanned Surfaces”, Symposium on Interactive 3D Graphics (I3D’95).


13

Related Work : 3D PaintingRelated Work : 3D Painting

• Gregory, Ehmann & Lin, 1999 Gregory, Ehmann & Lin, 1999 44

– inTouch haptic paintinginTouch haptic painting» Modeling, painting, force feedbackModeling, painting, force feedback» No real object; purely virtualNo real object; purely virtual

• Keefe, Feliz et al, 2001 Keefe, Feliz et al, 2001 55

– CavePaintingCavePainting» Physical interface objectsPhysical interface objects» Display is purely virtualDisplay is purely virtual

4 Arthur Gregory, Stephen Ehmann and Ming C. Lin, “inTouch : Interactive Multiresolution Modeling and 3D Painting with a Haptic Interface”, Virtual Reality (VR’99).

5 Daniel F. Keefe, Daniel A. Feliz, Tomer Moscovitch, David Laidlaw and Joseph LaViola, Jr., “CavePainting : a Fully Immersive 3D Artistic Medium and Interactive Experience”, I3D 2001.


14





• ResultsResults


15

Projector CalibrationProjector Calibration

• Calibrate each projector (5 minutes)Calibrate each projector (5 minutes)– Match 3D points in world Match 3D points in world

space to 2D pixels in space to 2D pixels in projector space.projector space.

– Solve for 3x4 projection Solve for 3x4 projection matrixmatrix

» ignore radial distortionignore radial distortion» minimize least-square error (standard CV method)minimize least-square error (standard CV method)» split into intrinsic (projection) and extrinsic matrixsplit into intrinsic (projection) and extrinsic matrix

• Render 3D points in the world with good Render 3D points in the world with good registrationregistration

2D cross hair


16

Model Generation/AuthoringModel Generation/Authoring

• Need: vertices, faces, normals, texture Need: vertices, faces, normals, texture coordinates and color/material scheme per coordinates and color/material scheme per object to be paintedobject to be painted

– Acquire points using magnetic stylusAcquire points using magnetic stylus– Faces, textures by hand / surface reconstructionFaces, textures by hand / surface reconstruction– Texture coordinates non-repeating, fit geometryTexture coordinates non-repeating, fit geometry– Future: automate with Future: automate with

modeling tools.modeling tools.


17

Rendering Tracked ObjectRendering Tracked Object

• Algorithm each-frame:

Tracker gives 6DOF reading of sensors.For view i (projector i) do

Load projection and model view matrix iFor object j do

Apply world to sensor j matrixDraw object in sensor coordinatesInteract with object


18

3D painting interface3D painting interface

• Objects hand-held or set on table.Objects hand-held or set on table.

• Tracked stylus with spherical tip Tracked stylus with spherical tip – facilitates contact paintingfacilitates contact painting

• Projected touch palette, modeled as Projected touch palette, modeled as a static object with behavior:a static object with behavior:

– choose contact, spray or texture paintchoose contact, spray or texture paint– choose brush colorchoose brush color


19

Painting objectsPainting objects

• Determine object intersections (point-polygon)Determine object intersections (point-polygon)– Find set of polygons to modifyFind set of polygons to modify

• Color blending using brush function Color blending using brush function 66

– ContactContact paint uses a function on surface of object paint uses a function on surface of object

B = 1-(r/R)B = 1-(r/R)22, r<R, r<R

– SpraySpray paint uses a similar brush function in 3D paint uses a similar brush function in 3D– Color (or texture) blending:Color (or texture) blending:

CCfinalfinal = C = Cinitinit.(1-B) + C.(1-B) + Cbrushbrush.B.B

rR

6 From Arthur Gregory’s inTouch system


20

Painting objectsPainting objects

• Triangle rasterization w/ texture map Triangle rasterization w/ texture map modification modification 66

– Scan 2D and 3D simultaneously.Scan 2D and 3D simultaneously.

q

pu

vconcurrent

(x,y,z)

Polygon in 3D

(u,v)

2D texture map

Update colorLook up brush fn

6 From Arthur Gregory’s inTouch system


21





• ResultsResults


22

Performance / numbers ?Performance / numbers ?

• 60fps rendering 3 objects, 500+ polygons 60fps rendering 3 objects, 500+ polygons – ~100 texture maps, mostly 128x128~100 texture maps, mostly 128x128– Running on SGI InfiniteReality2 or 450MHz PC Running on SGI InfiniteReality2 or 450MHz PC

with multi-monitor graphics cardwith multi-monitor graphics card

• Not optimized for larger data setsNot optimized for larger data sets

• 100-110ms latency (high).100-110ms latency (high).– Roughly estimatedRoughly estimated

» 1cm of distance lag when moved at 10cm/s1cm of distance lag when moved at 10cm/s– Later, precisely measuredLater, precisely measured


23

Issues and Future WorkIssues and Future Work

• Tracker LimitationsTracker Limitations– Range Range (Polhemus Fastrak, 50-75 cm)(Polhemus Fastrak, 50-75 cm)

» use wide-area trackeruse wide-area tracker– Technology Technology (ferrous objects distort magnetic field)(ferrous objects distort magnetic field)

• Display limitationsDisplay limitations– Dynamic Intensity BlendingDynamic Intensity Blending– Limited dynamic range of projectorLimited dynamic range of projector– Occlusion, overlap, shadowsOcclusion, overlap, shadows

• Scaling it upScaling it up– Room sized environments, multiple projectorsRoom sized environments, multiple projectors


24

ConclusionsConclusions

• Dynamic Shader LampsDynamic Shader Lamps– A set of techniques for A set of techniques for interactively interactively changing changing

appearance attributes of neutrally colored physical appearance attributes of neutrally colored physical objects, static or movingobjects, static or moving

• Prototype implementationPrototype implementation– Painting on multiple real objectsPainting on multiple real objects– display devices - projectorsdisplay devices - projectors

» 3D display surfaces (real objects)3D display surfaces (real objects)– 3D input devices - tracker & stylus3D input devices - tracker & stylus

• Extensions Extensions - - how to generalize/scale.how to generalize/scale.


25

AcknowledgementsAcknowledgements

• ISAR reviewersISAR reviewers

• Miriam Mintzer Fuchs, age 10Miriam Mintzer Fuchs, age 10

• People at UNC People at UNC (paper, video & demo)(paper, video & demo)– Herman Towles, Greg Welch, Anselmo Lastra, Herman Towles, Greg Welch, Anselmo Lastra,

Russ Taylor, Ming Lin, Scott RussellRuss Taylor, Ming Lin, Scott Russell– Andrei State, Jim Mahaney, Kurtis Keller, John Andrei State, Jim Mahaney, Kurtis Keller, John

Thomas, David Harrison, Arthur GregoryThomas, David Harrison, Arthur Gregory– Kok-Lim Low, Wei-Chao Chen, Ben Lok, Kenny Kok-Lim Low, Wei-Chao Chen, Ben Lok, Kenny

Hoff, Dave McAllister, Miguel Otaduy, Gentaro HirotaHoff, Dave McAllister, Miguel Otaduy, Gentaro Hirota

• STC Tracked Shader Lamps, fundingSTC Tracked Shader Lamps, funding

Deepak Bandyopadhyay / UNC Chapel Hill


Deepak Bandyopadhyay, Ramesh Raskar, Henry FuchsDeepak Bandyopadhyay, Ramesh Raskar, Henry Fuchshttp://www.cs.unc.edu/~debug/papers/DSLpaint/http://www.cs.unc.edu/~debug/papers/DSLpaint/

Painting on Movable Objects

Images

5/1/2000deepak bandyopadhyay / unc chapel hill1 painting on movable objects deepak bandyopadhyay,...

Documents