andrew nealen and marc alexa, discrete geometric modeling group, tu darmstadt, 2003 hybrid texture...

Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Hybrid Texture SynthesisHybrid Texture Synthesis

Andrew NealenAndrew NealenMarc AlexaMarc Alexa

Discrete Geometric Modeling Group (DGM)Discrete Geometric Modeling Group (DGM)Technische Universität DarmstadtTechnische Universität Darmstadt

2D Texture Synthesis

nxm Input Texture

NxM Output Texture

► The goal: Synthesize an output texture which is perceptually similar to the input texture. Also ensure that the result contains sufficient variation.

Our Inspiration

► Existing (and Impressive) Technology• Pixel-Based:

─ Non-parametric Sampling [Efros and Leung 1999]─ Tree-structured Vector Quantization [Wei and Levoy 2000]─ Synthesizing Natural Textures [Ashikhmin 2001]─ Image Analogies [Hertzmann et al. 2001]

• Patch-Based:─ Patch-Based Sampling [Guo et al. 2001]─ Image Quilting [Efros and Freeman 2001]─ Hierarchical Pattern Mapping [Soler et al. 2002]

Our Inspiration

► Possible Drawbacks ?• Loss of global structure• Loss of scale• Boundary mismatch• Blurring

Our Inspiration

Wei/Levoy Algorithm:Can occur when using structured Textures with rich histograms.This is due to the L2 norm.

Our Inspiration

Patch-Based Sampling Algorithm:Can occur when texture features are not well-aligned across patch boundaries

Our Inspiration

► Is it possible to reduce all deficiencies for an even larger class of textures ?

Pixel-Based Texture Synthesis

Input Output

Step n in Algorithm

Input Output

Build L-shaped Neighborhood

Input Output

Find Matching Neighborhood(s) in Input

Input Output

Select Best Neighborhood from all Candidates

Input Output

Copy Selected Pixel to Output

Input Output

Repeat for all Pixels in this Row...

Input Output

...and all other Output Rows

► Main difference between Efros/Leung (EL), Wei/Levoy (WL) and Ashikhmin (AS): Per-Pixel Search Strategy.

► Ashikhmin’s insight: “WL tends to produce textures with smoothed out edges in some cases.”

► Ashikhmin’s solution: consider only coherent pixels in the search, which encourages region growing.

► Problem: inconsistencies can still occur for highly-structured textures.

Patch-Based Texture Synthesis

Input Output

Step n in Algorithm

Input Output

Patch to be Synthesized in this Step

Input Output

Build Neighborhood (Image Mask)

Input Output

Find Matching Neighborhood(s) in Input

Input Output

Select Best Neighborhood from all Candidates

Input Output

Copy Selected Patch to Output

Input Output

Apply some Overlap Repair Strategy ...

Input Output

... And we are done with this patch

Input Output

Repeat for all Patches in Output

Patch-Based Texture SynthesisGeneralization of Pixel-Based Synthesis

► Main (visual) difference between Image Quilting (IQ) and Patch-based Sampling (PBS): Overlap Repair Strategy.

► IQ performs a minimum-error-boundary-cut (MEBC) through the error surface (least cost path; Dijkstra).

► PBS feathers the overlap region (alpha-blending).

► Pro: patch-based algorithms are good at maintaining global structure.

► Problems: boundary mismatch (MEBC, top) and blurring (PBS, bottom).

Hybrid Texture Synthesis (HTS)

Algorithm

► Hybrid, two-fold approach to texture synthesis• Adaptive patch sampling [Soler et al. 2002]

─ Start with a uniform quadrilateral grid of patches (the output)─ Recursively split a patch if the best fit taken from the input

texture is not good enough (a user-defined tradeoff max)

• Overlap re-synthesis (novel overlap repair strategy)─ Mark invalid pixels in the overlap region

─ Validity map is defined by the error surface E(x)=(xsrc – xdst)2

and a user-defined tolerance parameter max

─ Define an ordering for the invalid pixels and re-synthesize them using a per-pixel strategy

Adaptive Patch SamplingPrinciple

► Essentially: Hierarchical Pattern Mapping applied to the plane [Soler et al. 2002]

► Analogy: Assembling a quadtree puzzle

p > max Split PatchSynthesize White Patch p < max

Adaptive Patch SamplingTrade-off

► Trade-off between preserving global structure and avoiding detail artifacts

max = 1.0max = 0.01 max = 0.04

In Search of Good PatchesSynthesizing a single patch

Synthesize Black Patch i

In Search of Good PatchesExtracting the image mask

Synthesize Black Patch i

► Grow patch by overlap (toroidally)

In Search of Good PatchesExtracting the image mask

Image Mask (Ii)Synthesize Black Patch i Binary Support (Ji)

► Extract image mask (Ii) and binary support (Ji), and circularly shift to upper left corner

In Search of Good PatchesComputing the error image

Image Mask (Ii)Input Texture (T) Binary Support (Ji)

► Compute error Ei(x0) between Ii and T for each circular shift x0=(x,y) of the input texture T

In Search of Good PatchesComputing the error image

Error Image (Ei)Input Texture (T)

xxxxxc

cicicii

i TIwJκ

E 20,,0 ))()(()(

In Search of Good PatchesSelecting a Patch

Error Image (E)Input Texture (T) Selected Patch (P)

xxxxxc

cicicii

i TIwJκ

E 20,,0 ))()(()(

In Search of Good PatchesSelecting a Patch

Error Image (E)Input Texture (T) Selected Patch (P)

► Note: error image can be computed efficiently in the Fourier domain. Complexity: O(n log n) per patch.

Overlap Re-synthesisOverlap error and pixel invalidation

Patch and Image Mask Selected Patch (P)

Image Mask (I) Selected Patch (P)

cicicii PIwJS 2,, ))()(()()( xxxx

Image Mask (I) Selected Patch (P)Error Surface (S)

cicicii PIwJS 2,, ))()(()()( xxxx

Patch and Image Mask P with Invalid Pixels (blue)Error Surface (S)

► Trade-off: user-defined max is the pixel error tolerance. Setting to 1 results in pure feathering.

Overlap Re-synthesisCompositing

Intermediate Result Patch with Invalid Pixels

P OVER Ri-1

Overlap Re-synthesisCompositing

Composited Result Patch with Invalid Pixels

► Simple scanline re-synthesis: possibly insufficient causal neighborhood within valid pixel region

► Solution: alternative ordering (Pixel Traversal Map)

Overlap Re-synthesisPixel Traversal Map (Ordering)

Composited Result Patch with Invalid PixelsPixel Traversal Map (M)

► Pixel Traversal Map: repeated morphological dilation of the binary support for valid pixels with a euclidian disk of radius 1 (city-block distance transform).

► Pixel Traversal Map: step 1

► Pixel Traversal Map: step 2 ...

► Pixel Traversal Map: ... step n

► Art Restorer Analogy: stepwise restoration of the hole from the boundary of the existing image.

Overlap Re-synthesisPer-Pixel Re-synthesis

Pixel Traversal Map (M)

► Synthesize red pixel from valid (or already re-synthesized) pixels

Intermediate Result

Image Mask (Ij)

Binary Support (Jj)

► Extract image mask (Ij) and binary support (Jj) for best-pixel search in the input texture. Complexity: O(n log n) per pixel.

Intermediate Result

Intermediate Result Pixel Traversal Map (M)

Intermediate Result Shifted

Intermediate Result with Traversal Map

Overlap Re-synthesisStep 1

Overlap Re-synthesisResult

ResultsOverlap Repair Comparisons

No Repair HTSPBSIQ

ResultsSynthesis Comparisons

Efros/Leung Wei/Levoy

IQ PBS HTS

PBS HTS

Results

Conclusions and Future Work

► Improve Computational Complexity• Pixel neighborhoods of patch and pixel stage are not

known a priori, so precomputation is not straightforward• We can solve this in the pixel stage by employing k-

coherence search [Tong et al. 2002] [Ashikhmin 2001]

► Improve Error Metric• Still using the L2 norm due to its simplicity

• Develop a metric which takes feature mismatch into account

• Texton approach looks promising [Zhang et al. 2003]

Questions ?

► Contact Information

Andrew Nealenandy@nealen.com

Marc Alexaalexa@informatik.tu-darmstadt.de

http://www.dgm.informatik.tu-darmstadt.de

Matlab code and extended Thesis:http://www.nealen.com

andrew nealen and marc alexa, discrete geometric modeling group, tu darmstadt, 2003 hybrid texture...

tu darmstadt

output texture

algorithm slide

input slide

output slide

texture features

d texture synthesis

shaped neighborhood

Documents

fotografia alexa

alexa video

please meet amazon alexa and the alexa skills kit

slide alexa

2013 05-16 alexa and alexa xt dimensions

alexa for business - alexa for business api · alexa for...

alexa meade

alexa buenaÑo

all about alexa

alexa metrcs seom.bg

einrichten von amazon alexa im sonos-systeml... · 2017....

portfolio alexa

andrew nealen tu berlin takeo igarashi the university of...

by hailey nealen, naomi belcher, and alexis beyke

a sketch-based interface for detail-preserving mesh editing...

practicas alexa

monografía alexa

mecanografia alexa

bolton, alexa

alexa layar depan & belakang hybrid scale alexa þeeeee...