In Search of Textons

Harry Shum

Microsoft Research, Asia

Texture analysis and synthesisn Texture images

n 2D textons and texton distribution

n Bi-directional texture functions (BTF) n “3D”textons and BTF synthesis

n Motion texture n “1D”Motion textons and motion synthesis

2D texture imagesn An image

n 2D spatial distribution of pixels

n A texture image n Inhomogeneous at the limit of resolution n Homogeneous at some scale larger than image resolutionn Perceived homogeneous by human observers n Stochastic, nearly periodic, 2D MRF

Are they textures?

Structured Semi-structured Stochastic

2D texture images

n In psychology, basic texture elements are called “texton”or “texel”[Julesz’81]

n In early vision, natural visual patterns consist of multiple layers of stochastic processes [Marr’82]

But what are image textons?

?

Open question: what is a texton? L

Integrating descriptive and generative models-- (Guo et.al. ’01)

image

⊕=

layer2layer1

textons

texton maps

Texture modeling for synthesis

n Analyze n Generalize the stochastic and dynamic nature

n Synthesize n Realistic texture and visually acceptable

n Edit n Modify texture at different levels

n Analyze n Generalize the stochastic and dynamic nature

n Synthesize n Realistic texture and visually acceptable

n Edit n Modify texture at different levels

Real-Time Texture Synthesis By Patch-Based Sampling

Lin Liang, Ce Liu, Ying-Qing Xu, Baining Guo and Heung-Yeung Shum

ACM Transactions of Graphics, July 2001

Patch-based sampling (Liang et. al. ’01)

patches + MRF

Patch-based Sampling Scheme— Sampling Strategy

RI∂)|( RR IIp ∂

RI

RI∂ RI

RI

∂R

wE

Patch-based Sampling Scheme— Synthesizing Texture

inI

kB

outI

Synthesis Result— High-quality Texture

Input sample Efros and Leung Wei and levoy Patch-based

Synthesis Result— High-quality Texture

input texture synthesized texture input texture synthesized texture

input texture synthesized texture input texture synthesized texture

Click here to see more synthesis result

Real-Time Texture Synthesis By Patch-Based Sampling

0.02s

667MHz PC

128×128Input texture

200×200Synthesized texture

How to synthesize high-quality texture in real-time

n Patch-Based Sampling Schemen Patches of the sample texture as building blocksn Advantages:

n Orders of magnitude faster than existing algorithmsn Synthesize high-quality textures from regular to stochastic.

Avoid growing garbage. Subjectively better natural textures

n Quality-first Acceleration Schemen General acceleration: an optimized kd-treen Domain specific acceleration: quad-tree pyramid (QTP)n Data specific acceleration: PCA

2D textons = image patches?

Texture Intensity Channel K-Means Clustering

Texture Principle Components K-Means Clustering

Patch Size (1)

10% 15% 25%

Patch Size (2)

10% 15% 25%

Patch Size (3)

10% 15% 25%

Textons -- A Different View

Texton Map (Label Map) Affinity Matrix

Synthesis with Texton Map

Pixel-based (Ashikhmin) Our Patch-based

Some Results

Source

Pixel based

Texton based

Why textons and texton map?

Synthesis of Bidirectional Texture Functions on Arbitrary Surfaces

X. Tong, J. Zhang, L. Liu, X. Wang, B. Guo, H.-Y. Shum

Siggraph’2002

Real World Texture

View

Light

•Geometry Details (Mesostructure) on Surface

•Self-Occlusion, Self-Shadow, and Specularity

Bidirectional Texture Functions (BTF)

A collection of images of the same surface under different lighting and viewing directions.

― [Dana et. al. 97]

n 6D Function ( x , y , lθ , lφ , vθ , vφ )n Dense Sampling in Viewing/Lighting Directionn Capturing Appearance of Real World Surface

3D Texton [Leung & Malik 2001]

n Capturing Small Number of Microstructures and Reflectance Variations in BTF

n Represented by Filter Bank Responses Vectorn Use 48 Gussian derivative filters for each samplen Appearance vector

Algorithm Overview

Texton AnalysisTexton Analysis

Texton Map Synthesis Texton Map Synthesis on Surfaceon Surface

BTF Sample Sample Texton Map

MeshSurface Texton Map

RenderingRendering

2D Texture Synthesis on Surface

RetilingRetiling FlattenFlatten

ResamplingResampling

SearchingSearching

CopyCopy

s

t

v

Texton Analysis

Texton AnalysisTexton Analysis

Texton Map Synthesis Texton Map Synthesis on Surfaceon Surface

BTF Sample Sample Texton Map

MeshSurface Texton Map

RenderingRendering

Texton Analysis T1 T2 T3 Tn

3D Textons

BTF Samples

Filter Bank

Appearance Vectors

Clustering

T1 T2 T3 Tn

3D Textons

Sample Texton Map

BTF Samples

Labeling

Dot-product Matrix

n

n

n n n n

TT TT TTT T T T T T

T T TT T T

1 1 1 2 1

2 1 2 2 2

1 2

LL

M M O ML

Synthesis from Real World Samples

Synthesis from Synthetic Samples

3D textons = BTF clusters

But what about other textons?

Motion Texture

A Two-level Statistical Model for Motion Synthesis

Yan Li Tianshu Wang Heung-Yeung Shum

Siggraph’2002

n Motion editingnInteractive motion editing

n [Bruderlin’95, Witkin’95, Lee’99, Pullen’02]

nAdaptation to new characters and environmentsn [Hodgins’95, Gleicher’98, Popovic’99]

n Motion synthesisnRetain the realism of original captured datanAllow the user to control and direct the character

Motion capture data re-use

Motion synthesis

n Reordering the original datan Chop into motion clips & model their transitions

[Kovar’02, Lee’02, Arikan’02], [Schodl’00] Video Texture

n Problemn No generalization ability

n Generative modelsn Hidden Markov Model (HMM)

[Brand’00, Tanco’00]

n Auto-regressive model for simple movements[Pavlovic’00] SLDS, [Soatto’01] Dynamic Texture

Motion textureRepresentationn A two-level statistical model n Motion texton --- Linear dynamic system (LDS)n Texton distribution --- Markov process

Analogous to 2D texture imagen Textons (Julesz’81) n 2D spatial distribution of textons

Motion texture analysis/synthesis

Challengesn How to learn the motion texture n How to synthesize from motion texturen How to deal with high dimensional motion data

Motion data representationn Configuration of an articulated character

n Linear components :n Angular components :

=

)(

)()()(

)( 2

1

t

ttt

t

nq

qqp

mM

Position of the root joint

Orientations of the body joint

Orientation of the root joint

3)( Rp ∈t3)( Sq i ∈t

Dealing with motion data

−−

=

))(log(

))(log())(log(

)1()(

)( 2

1

t

tttt

t

nq

qqpp

Y

M

=

)(

)()()(

)( 2

1

t

ttt

t

nq

qqp

m

M

=

)(

)()()(

)( 3

2

t

ttt

t

mx

xxx

X

1

M

motion space linearized motion state space

Motion textureA graphical model representation

1−tx tx 1+tx

ty 1+ty1−ty

is

1−tx tx 1+tx

ty 1+ty1−ty

js

Markov model )|( ij ssP

Dynamics model ),|( 1 jtt sxxP −

Observation model ),|( jtt sxyP

Motion textons

n A set of basic motion elementsn Representation: LDS

An auto-regressive moving average process (ARMA)

Dynamics model:

Observation model:ttt1t VXAX +=+

tttt WXCY +=tA

tCtV

tWW)V,C,(A,? =

model parameters

Linear dynamic system (LDS)

n Previous workn Tracking and gait recognition [Bregler’97, North’00,

Pavlovic’00, Bissacco’01]n Video synthesis [Soatto’01, Fitzgibbon’01]

n Challenge for character motion synthesisn Captured motion is non-stationary

n Our solutionn Segment-based stationary processes (LDS’)

n Estimation by EM algorithm (segmentation!)n E-step: how many segments and where they are n M-step: fitting LDS parameters for the segments labeled

by the same textonn Transition graph (M): counting segment labels

n A maximum likelihood solution

Learning motion texture

?

graph)n transitio:M ,parameters LDS :(?

M)?,|P(Yargmax}M,?{ T:1M}{?,

ˆˆ =

With the texton distribution n Random walk

n ABDCBDEC… …

n Multiple paths between any pairsn {ABDC}, {AEC}

n Add variations to the synthesized motion

n {AEC}

B

E

DA C

A two-step synthesis algorithmn Texton path planning using DP

n Finding the lowest cost pathn Specifying the path length

n Synthesizing a single textonn Smooth transition between textons

Texton synthesis by constrained LDS

0V 1V 2V 1−nV

Sampling noise and incorporating both boundary constraintsSampling noise and incorporating both boundary constraints

1Xdynamics

model 2Xdynamics

model...dynamics

modeldynamics

model

1Y 2Y ...

Observationmodel

Observationmodel

Solving a block-banded linear system

nX

nY

Observationmodel

0X

Observationmodel

0Y

Texton synthesis by sampling noise

1Xdynamics

model

0V

0X

observationmodel

0Y

2X

1V

dynamicsmodel

...

nX

2V 1−nV

...dynamicsmodel

dynamicsmodel

1Y

observationmodel

2Y

observationmodel

nY

observationmodel

Error accumulation t

Without end constraintsWithout end constraints

Video: synthesis and editing results

Video: synthesized dance motion

Summary

n Texture model n Two levels: textons and distribution

n What are textons? n 2D textonsn 3D BTF textonsn 1D motion textons

n Texture synthesisn Sample individual textonsn Sample texton distributions

Acknowledgements

n Many colleagues in MSRA and MSR