textures & image-based lighting - 國立臺灣大學cyy/courses/vfx/05... · textures &...
TRANSCRIPT
![Page 1: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/1.jpg)
Textures & Image-Based Lighting
Digital Visual Effects, Spring 2005Yung-Yu Chuang2005/6/15
with slides by Alex Efros, Li-Yi Wei and Paul Debevec
![Page 2: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/2.jpg)
Announcements
• Final project presentation on 6/28 1:30pm in Room 101
• What to hand in?
![Page 3: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/3.jpg)
Outline
• Texture synthesis• Acceleration by multi-resolution and TSVQ• Patch-based texture synthesis• Image analogies• Image-based lighting
![Page 4: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/4.jpg)
Texture synthesis
![Page 5: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/5.jpg)
Texture synthesis
• Given a finite sample of some texture, the goal is to synthesize other samples from that same texture. – The sample needs to be "large enough"
synthesis
generated image
input image
![Page 6: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/6.jpg)
The challenge
• How to capture the essence of texture? • Need to model the whole spectrum: from
repeated to stochastic texture
![Page 7: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/7.jpg)
Ni
Markov property
• P(fi|fi-1,fi-2,fi-3,…f0)= P(fi|fi-1,fi-2,…fi-n)
ffiFfi-1Ff0
S
fi
Ffi-2… Ffi-n …
• P(fi|fS-{i})= P(fi|fNi)
![Page 8: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/8.jpg)
Motivation from language
• [Shannon’48] proposed a way to generate English-looking text using N-grams:– Assume a generalized Markov model– Use a large text to compute probability
distributions of each letter given N-1 previous letters
• precompute or sample randomly
– Starting from a seed repeatedly sample this Markov chain to generate new letters
– One can use whole words instead of letters too.
![Page 9: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/9.jpg)
Mark V. Shaney (Bell Labs)
• Results (using alt.singles corpus):– "One morning I shot an elephant in my arms and
kissed him.”– "I spent an interesting evening recently with a
grain of salt"
• Notice how well local structure is preserved!– Now let’s try this in 2D...
![Page 10: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/10.jpg)
– Assuming Markov property, what is conditional probability distribution of p, given the neighbourhood window?
– Instead of constructing a model, let’s directly search the input image for all such neighbourhoods to produce a histogram for p
– To synthesize p, just pick one match at random
SAMPLE
p
Ideally
Infinite sample imagegenerated image
![Page 11: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/11.jpg)
finite sample image
Generated image
p
– However, since our sample image is finite, an exact neighbourhood match might not be present
– So we find the best match using SSD error (weighted by a Gaussian to emphasize local structure), and take all samples within some distance from that match
– Using Gaussian-weighted SSD is very important
SAMPLE
In reality
![Page 12: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/12.jpg)
Neighborhood size matters
![Page 13: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/13.jpg)
More results
Increasing window size
![Page 14: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/14.jpg)
french canvas rafia weave
More results
![Page 15: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/15.jpg)
woodMore results
brick wall
![Page 16: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/16.jpg)
Failure cases
Growing garbage Verbatim copying
![Page 17: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/17.jpg)
Inpainting
• Growing is in “onion peeling” order– within each “layer”, pixels with most neighbors
are synthesized first
![Page 18: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/18.jpg)
Inpainting
![Page 19: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/19.jpg)
Inpainting
![Page 20: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/20.jpg)
Results
![Page 21: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/21.jpg)
Recent inpaiting algorithms
Obtain structure first, add details by texture synthesis
![Page 22: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/22.jpg)
Summary of the basic algorithm
• Exhaustively search neighborhoods
![Page 23: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/23.jpg)
Neighborhood
• Neighborhood size determines the quality & cost
3×3 5×5 7×7
9×9 11×11 41×41
423 s 528 s 739 s
1020 s 1445 s 24350 s
![Page 24: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/24.jpg)
Summary
• Advantages:– conceptually simple– models a wide range of real-world textures– naturally does hole-filling
• Disadvantages:– it’s slow – it’s a heuristic
![Page 25: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/25.jpg)
Acceleration by Wei & Levoy
• Multi-resolution• Tree-structure
![Page 26: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/26.jpg)
Multi-resolution pyramid
High resolution Low resolution
![Page 27: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/27.jpg)
Multi-resolution algorithm
![Page 28: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/28.jpg)
Benefits
• Better image quality & faster computation
1 level5×5
3 levels5×5
1 level11×11
![Page 29: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/29.jpg)
Results
Random Oriented
Regular Semi-regular
![Page 30: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/30.jpg)
Failures
• Non-planar structures
• Global information
![Page 31: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/31.jpg)
Acceleration
• Computation bottleneck: neighborhood search
![Page 32: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/32.jpg)
Nearest point search
• Treat neighborhoods as high dimensional points
1 2 3 4 5
6 7 8 9 10
11 12
Neighborhood
1 2 3 4 5 6 7 8 9 10 11 12
High dimensional point/vector
![Page 33: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/33.jpg)
Tree-Structured Vector Quantization
![Page 34: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/34.jpg)
Timing
• Time complexity : O(log N) instead of O(N)– 2 orders of magnitude speedup for non-trivial
images
1941 secs 503 secs 12 secs
Efros 99 Full searching TSVQ
![Page 35: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/35.jpg)
Results
Input Exhaustive: 360 s TSVQ: 7.5 s
![Page 36: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/36.jpg)
pp
Patch-based methods
• Observation: neighbor pixels are highly correlated
Input image
non-parametricsampling
BB
Idea:Idea: unit of synthesis = blockunit of synthesis = block• Exactly the same but now we want P(B|N(B))
• Much faster: synthesize all pixels in a block at once
Synthesizing a block
![Page 37: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/37.jpg)
Philosophy
• The “Corrupt Professor’s Algorithm”:– Plagiarize as much of the source image as you can– Then try to cover up the evidence
• Rationale: – Texture blocks are by definition correct samples of
texture so problem only connecting them together
![Page 38: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/38.jpg)
Algorithm
– Pick size of block and size of overlap– Synthesize blocks in raster order
– Search input texture for block that satisfies overlap constraints (above and left)
– Paste new block into resulting texture• blending• use dynamic programming to compute minimal error boundary
cut
![Page 39: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/39.jpg)
Input texture
B1 B2
Random placement of blocks
block
B1 B2
Neighboring blocksconstrained by overlap
B1 B2
Minimal errorboundary cut
![Page 40: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/40.jpg)
min. error boundary
Minimal error boundary
overlapping blocks vertical boundary
__ ==22
overlap error
![Page 41: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/41.jpg)
Results
![Page 42: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/42.jpg)
Results
![Page 43: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/43.jpg)
Failure cases
![Page 44: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/44.jpg)
GraphCut textures
![Page 45: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/45.jpg)
GraphCut textures
![Page 46: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/46.jpg)
Photomontage
![Page 47: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/47.jpg)
Photomontage
![Page 48: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/48.jpg)
Photomontage
![Page 49: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/49.jpg)
Sourcetexture
Target image
Sourcecorrespondence
image
Targetcorrespondence image
Texture transfer
![Page 50: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/50.jpg)
++ ==
Texture transfer
![Page 51: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/51.jpg)
Image Analogies
![Page 52: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/52.jpg)
Coherence search
![Page 53: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/53.jpg)
Image Analogies Implementation
![Page 54: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/54.jpg)
Image Analogies Implementation
![Page 55: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/55.jpg)
Image Analogies Implementation
![Page 56: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/56.jpg)
Balance between approximate and coherence searches
![Page 57: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/57.jpg)
Learn to blur
![Page 58: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/58.jpg)
Super-resolution
![Page 59: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/59.jpg)
![Page 60: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/60.jpg)
Colorization
![Page 61: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/61.jpg)
Artistic filters
![Page 62: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/62.jpg)
:
B B’
::
![Page 63: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/63.jpg)
![Page 64: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/64.jpg)
:
B B’
::
![Page 65: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/65.jpg)
![Page 66: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/66.jpg)
Texture by numbers
![Page 67: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/67.jpg)
Texture by numbers
![Page 68: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/68.jpg)
Image-based lighting
![Page 69: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/69.jpg)
Rendering
• Rendering is a function of geometry, reflectance, lighting and viewing.
• To synthesize CGI into real scene, we have to match the above four factors.
• Viewing can be obtained from calibration or structure from motion.
• Geometry can be captured using 3D photography or made by hands.
• How to capture lighting and reflectance?
![Page 70: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/70.jpg)
HDRI Sky Probe
![Page 71: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/71.jpg)
Clipped Sky + Sun Source
![Page 72: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/72.jpg)
Lit by sun only
![Page 73: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/73.jpg)
Lit by sky only
![Page 74: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/74.jpg)
Lit by sun and sky
![Page 75: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/75.jpg)
Acquiring the Light Probe
![Page 76: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/76.jpg)
Illuminating a Small Scene
![Page 77: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/77.jpg)
![Page 78: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/78.jpg)
Real Scene Example
• Goal: place synthetic objects on table
![Page 79: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/79.jpg)
Light Probe / Calibration Grid
![Page 80: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/80.jpg)
real scenereal scene
Modeling the Scene
light-based modellight-based model
![Page 81: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/81.jpg)
The Light-Based Room Model
![Page 82: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/82.jpg)
Rendering into the Scene
• Background Plate
![Page 83: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/83.jpg)
Rendering into the scene
• Objects and Local Scene matched to Scene
![Page 84: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/84.jpg)
Differential rendering
• Local scene w/o objects, illuminated by model
![Page 85: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/85.jpg)
Differential rendering
==--
![Page 86: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/86.jpg)
Differential Rendering
• Final Result
![Page 87: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/87.jpg)
Environment map from single image?
![Page 88: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/88.jpg)
Eye as light probe! (Nayar et al)
![Page 89: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/89.jpg)
Cornea is an ellipsoid
![Page 90: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/90.jpg)
Ellipsoid fitting
![Page 91: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/91.jpg)
Results
![Page 92: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/92.jpg)
Reflectance
• The Bidirectional Reflection Distribution Function– Given an incoming ray and outgoing ray
what proportion of the incoming light is reflected along outgoing ray?
Answer given by the BRDF:
surface normal
![Page 93: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/93.jpg)
Capturing reflectance
![Page 94: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/94.jpg)
Application in “The Matrix Reloaded”
![Page 95: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/95.jpg)
Reference• Alexei A. Efros, Thomas K. Leung, Texture Synthesis by Non-
parametric Sampling, ICCV 1999. • Li-Yi Wei, Marc Levoy, Fast Texture Synthesis Using Tree-
Structured Vector Quantization, SIGGRAPH 2000. • Michael Ashikhmin, Synthesizing Natutal Textures, I3D 2001. • Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless,
David H. Salesin, Image Analogies, SIGGRAPH 2001. • Alexei A. Efros, William T. Freeman, Image Quilting for Texture
Synthesis and Transfer, SIGGRAPH 2001. • Vivek Kwatra, Arno Schodl, Irfan Essa, Greg Turk, Aaron Bobick,
Graphcut Textures: Image and Video Texture Synthesis Using Graph Cuts, SIGGRAPH 2003.
• Michael F. Cohen, Jonathan Shade, Stefan Hiller, Oliver Deussen, Wang Tiles for Image and Texture Generation, SIGGRAPH 2003.
• A. Criminisi, P. Perez, K. Toyama, Object Removal by Examplar-Based Inpainting, CVPR 2003.
• Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David H. Salesin, Michael F. Cohen, Interactive Digital Photomontage, SIGGRAPH 2004.
![Page 96: Textures & Image-Based Lighting - 國立臺灣大學cyy/courses/vfx/05... · Textures & Image-Based Lighting Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/6/15 with slides](https://reader034.vdocuments.net/reader034/viewer/2022042916/5f54c720031a6147c075e324/html5/thumbnails/96.jpg)
Reference• Paul Debevec, Rendering Synthetic Objects into Real Scenes:
Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography, SIGGRAPH 1998.
• Haarm-Pieter Duiker, Lighting Reconstruction for "The Matrix Reloaded", SIGGRAPH 2003 Sketch and Applications.
• George Borshukov, Measured BRDF in Film Production - Realistic Cloth Appearance for "The Matrix Reloaded", SIGGRAPH 2003 Sketch and Applications.
• Ko Nishino, Shree K. Nayar, Eyes for Relighting, SIGGRAPH 2004.