course 10 realistic materials in computer...

1

Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang

Course 10Realistic Materials in Computer Graphics

Course 10Realistic Materials in Computer Graphics

Environment Matting

Yung-Yu Chuang

National Taiwan University

Environment Matting

Yung-Yu Chuang

National Taiwan University

Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuang

input image alpha composite

Blue screen mattingBlue screen matting


alpha compositesource image

Problem: blue foregroundProblem: blue foreground


Two-screen mattingTwo-screen matting

alpha compositeCourse 10: Realistic Materials in Computer Graphics Yung-Yu Chuang

Problem: refractive objectProblem: refractive object

alpha composite

2


alpha composite photograph

Problem: refractive objectProblem: refractive object


Refracted image of a single pixelRefracted image of a single pixel


Environment matting frameworkEnvironment matting framework

backgroundC

composite

T

F

foreground

Ccomposite

Tbackground

F

foreground

Ccomposite

Tbackground

1-αB

3

F

foreground

Ccomposite

Tbackground

W

weightingfunction

0

C Tweightingfunction

F

foreground

W

backgroundcomposite

C Tweightingfunction

W

backgroundcomposite

Arbitrary weighting function



Environment matting algorithmsEnvironment matting algorithms• A model to approximate the environment

compositing equation

• A method to estimate model parameters

• Complexity of the model determines the complexity of the captured effects

4


Environment Matting and CompositingEnvironment Matting and Compositing

Douglas Zongker Dawn WernerBrian Curless David Salesin

SIGGRAPH 1999 F

foreground

Ccomposite

Tbackground

1-αB

AR


Acquisition setupAcquisition setup


Hierarchical backgroundsHierarchical backgrounds


Hierarchical backgroundsHierarchical backgrounds


Searching for α and ASearching for α and A

2)(computedobserved∑ −= ACCE

hypothesized A

5


),( rl ),( bt

Separate x and y extent searchesSeparate x and y extent searches

foreground alpha(coverage)

background


Environment matte compositeEnvironment matte composite


alpha mattecomposite

environment mattecomposite

Results and comparisonsResults and comparisons


photographenvironment mattecomposite




ResultsResults


6


Problem: glossy surfaceProblem: glossy surface

environment mattecomposite photograph


Problem: multiple mappingsProblem: multiple mappings



Problem: color dispersionProblem: color dispersion



Problem: many photographs Problem: many photographs


Problem: many photographs Problem: many photographs


Environment Matting Extensions: Toward Higher Accuracy and Real-Time Capture

Environment Matting Extensions: Toward Higher Accuracy and Real-Time Capture

Yung-Yu Chuang Douglas Zongker Joel HindorffBrian Curless David Salesin Richard Szeliski

SIGGRAPH 2000

7


O(log n)

higheraccuracy

fewerimages

Environment mattingEnvironment matting


O(log n) 1

fewerimages

higheraccuracy

Towards real-time captureTowards real-time capture

F

foreground

Ccomposite

Tbackground

1-αB

W

weightingfunction

AR

3

3 observations

3 1 4

11 variables

• A, R• α• F

3

4

7 variables

• A, R• α• F

33

3 observations

4

5 variables

• A, R• α• F

13

3 observations

A, ρcolorless

8

3 variables

• A, R• α• F

13

3 observations

A, ρcolorless

T(cx , cy)11

cx , cy , ρspecularlyrefractive


T

M(T , A)A

x

y

(cx, cy)

≈ T(cx, cy)

Stimulus functionStimulus function


x

y

T(x,y)

Ideal plane in RGB cubeIdeal plane in RGB cube


T(x,y)

x

y

(cx ,cy)C′

Calibrated manifold in RGB cubeCalibrated manifold in RGB cube


T

backgroundmanifold

K

W

C′

C′ → (cx ,cy)

Cρρ = KC

KC′

Estimate cx ,cy and ρEstimate cx ,cy and ρ


Problem: noisy matteProblem: noisy matte

9


with filteringwithout filtering

Edge-preserving filteringEdge-preserving filtering


Input imageInput image


Difference thresholdingDifference thresholding


Morphological operationMorphological operation


FeatheringFeathering


Heuristics for specular highlightsHeuristics for specular highlights

T

backgroundmanifold

K

W

Cρ >1+ε C′

10



C=ρT(cx, cy)

ρ cx cy


=

input estimation foreground(highlights)



Composite with highlightsComposite with highlights


O(log n) 1

fewerimages

higheraccuracy

Towards real-time captureTowards real-time capture

11


O(log n)O(n)

fewerimages

higheraccuracy

Towards higher accuracyTowards higher accuracy

1


Multimodal oriented Gaussian Multimodal oriented Gaussian

σu

σv θ

• Better BRDF approximation

• Multiple mappings

• Wavelength-coupled variation

Multimodal oriented Gaussian Unimodal axis-aligned Gaussian

backgroundcomposite

12

Unimodal axis-aligned Gaussian

σu backgroundcomposite


Unimodal axis-aligned Gaussianσs

Tbackground

composite



cu

cv c


13

cσv

σu

Unimodal axis-aligned Gaussian Unimodal oriented Gaussian

θσv

σu

Multimodal oriented Gaussian Multimodal oriented Gaussian

Multimodal oriented Gaussian


14


Glossy surfaceGlossy surface

SIGGRAPH 99 photograph


Glossy surfaceGlossy surface

higher accuracyalgorithm

photograph


Oriented GaussianOriented Gaussian

without orientation photograph


Oriented GaussianOriented Gaussian

with orientation photograph


Multiple mappingsMultiple mappings



Multiple mappingsMultiple mappings


photograph

15


Color dispersionColor dispersion



Color dispersionColor dispersion


photograph


Frequency-based Environment MattingFrequency-based Environment Matting

Jiayuan Zhu Yee-Hong Yang

Pacific Graphics 2004


Frequency-based environment mattingFrequency-based environment matting

spectrum of C

C

B(x1, t)

B(x2, t)

C(t)

x1x2

t


Frequency-based environment mattingFrequency-based environment matting

)()(),( yWxWyxW yx ⋅=

t

xttxB π2sin),( =


Results: refractionResults: refraction

frequency-based environment matting

photograph

16


Results: color dispersionResults: color dispersion


photograph


Results: orientedResults: oriented


photograph


Wavelet Environment MattingWavelet Environment Matting

Pieter Peers Philip DutréEurographics Symposium on Rendering 2003


Wavelet environment mattingWavelet environment matting

∫= WTC ∑=i

iiTaT

∫ ∑ ⎟⎠

⎞⎜⎝

⎛=

iiiTaW

∑ ∫=i

ii TWa

∑=i

iiCa ii TWC ∫=



a1 a2 a3 ak ak+1

iTT

ii TWC ∫=CCourse 10: Realistic Materials in Computer Graphics Yung-Yu Chuang


a1 a2 a3 ak ak+1

17


Error treeError tree

W

W

W

+

+

Course 10: Realistic Materials in Computer Graphics Yung-Yu Chuangreference image

Results: number of basis imagesResults: number of basis images

100 basis images300 basis images600 basis images900 basis images1200 basis images


Results: number of basis imagesResults: number of basis images

reference image 100 basis images300 basis images600 basis images900 basis images1200 basis images


reference image

1000 Haar patterns

1000 Daubechies(9,7) patterns

Results: wavelet patternsResults: wavelet patterns



reference image

1000 Haar patterns

1000 Daubechies (9,7) patterns



reference image

1000 Haar patterns

1000 Daubechies (9,7) patterns

18


Results: diffuse materialsResults: diffuse materials


Image-based Environment MattingImage-based Environment Matting

Yonatan Wexler Andrew Fitzgibbon Andrew Zisserman

Eurographics Workshop on Rendering 2002


Image-based environment mattingImage-based environment matting



p

q

2)()()( qBpCeqP −−=



p

q

2)()()( qBpCeqP −−=



19




ResultsResults


ResultsResults


SummarySummary

number of images

complexity of visual effects

1 logn n

HEM


SummarySummary

number of images


1 logn n

HEM

RTEM


SummarySummary

number of images


1 logn n

HEM

RTEM

GEM

20


SummarySummary

number of images


1 logn n

HEM

RTEM

GEMFBEM


SummarySummary

number of images


1 logn n

HEM

RTEM

GEMFBEM

WEM


SummarySummary

number of images


1 logn n

HEM

RTEM

GEMFBEM

WEM

IBEM


Acquisition and Rendering ofTransparent and Refractive Objects

Acquisition and Rendering ofTransparent and Refractive Objects

Wojciech Matusik Hanspeter PfisterRemo Ziegler Addy Ngan Leonard McMillan

Eurographics Workshop on Rendering 2002

21


The systemThe system


Geometry - opacity hullGeometry - opacity hull

• Visual hull augmented with view-dependent opacity

• Store the opacity of each observation at each point on the visual hull.


ResultsResults


InterpolationInterpolation


ComparisonsComparisons


ReferenceReference

• D. Zongker, D. M. Werner, B. Curless, and D. H. Salesin. Environment Matting and Compositing, SIGGRAPH 1999, pp205-214.

• Yung-Yu Chuang, Douglas E. Zongker, Joel Hindorff, Brian Curless, David H. Salesin, Richard Szeliski, Environment Matting Extensions: Towards Higher Accuracy and Real-Time Capture, SIGGRAPH 2000.

• P. Peers and P. Dutre. Wavelet Environment Matting, EGSR 2003.

• J. Zhu and Y.-H. Yang. Frequency-Based Environment Matting, Pacific Graphics 2004.

• Y. Wexler, A. Fitzgibbon and A. Zisserman. Image-Based Environment Matting, EGWR 2002, pp279-289.

22


Future workFuture work

• More general materials (Specular + diffuse)

• Principled Bayesian approach

• Advanced rendering

• Editing


ConclusionConclusion

• No restriction on material properties

• Linearly combining weighted basis images

• Hierarchical algorithm– Wavelets

– Feedback loop


Future WorkFuture Work

• Extend to full Image Based Relighting technique

• Better stop-criterion

• Environment matting with moving camera


On Refractive Optical FlowOn Refractive Optical Flow

Sameer Agarwal Satya MallickDavid Kriegman Serge Belongie

ECCV 2004


Refractive optical flowRefractive optical flow

refractive optical flow equation


ResultsResults

without water

with water

23


AcknowledgementsAcknowledgements

Douglas Zongker, Dawn Werner, Brian Curless, David Salesin, Joel Hindorff, Richard Szeliski, Jiayuan Zhu, Yee-Hong Yang, Pieter Peers, Philip Dutre, Yonatan Wexler, Andrew Fitzgibbon, Andrew Zisserman, SameerAgarwal, Satya Mallick, David Kriegman, Serge Belongie, Wojciech Matusik, HanspeterPfister, Remo Ziegler, Addy Ngan, Leonard McMillan




photograph




photograph







24




isotropic filter anisotropic filter

Edge-preserving filteringEdge-preserving filtering


Depth correctionDepth correction


Capturing at a single depthCapturing at a single depth


Capturing a second depthCapturing a second depth


Constructing the 3D beamConstructing the 3D beam

25


Rendering at novel depthsRendering at novel depths


Rendering at novel depthsRendering at novel depths



∫= WTC ∑=i

iiTaT

∫ ∑ ⎟⎠

⎞⎜⎝

⎛=

iiiTaW

∑ ∫=i

ii TWa

∑=i

iiCa

ii TWC ∫=


c

A

wx

wy

Estimate wx ,wyEstimate wx ,wy


Reflective objectsReflective objects


Many rays not capturedMany rays not captured

26


Add sidedrops to capture these raysAdd sidedrops to capture these rays


Capturing multiple sidesCapturing multiple sides


Contributions from multiple sidesContributions from multiple sides


Computing R and F from alphaComputing R and F from alpha





27











course 10 realistic materials in computer...

Documents