recognizing surfaces using three-dimensional textons thomas leung and jitendra malik computer...

Recognizing Surfaces using Three-Dimensional Textons

Thomas Leung and Jitendra Malik

Computer Science Division

University of California at Berkeley

ICCV '99, Corfu, Greece

Traditional Texture Recognition

• Assume texture to be planar;

• Assume constant illumination and viewing directions;

• Ignore 3D nature of natural materials, i.e. no shadowing, occlusions, etc…

• E.g. Puzicha et al, Jain et al, Greenspan et al, etc….


Example Natural Materials

Terrycloth Rough Plastic Plaster-b

Sponge Rug-a Painted Spheres

Columbia-Utrecht Database (http://www.cs.columbia.edu/CAVE)


Materials under different illumination and viewing directions

Differentilluminationand viewingdirections

Plaster-a CrumpledPaper

Concrete Plaster-b(zoomed)


TaskFelt?Polyester?Terrycloth?Rough Plaster?Leather?Plaster?Concrete?Crumpled Paper?Sponge?Limestone?Brick?

?

?


3D Texture Models

• Analytical models: – Simple parametric surface height distribution;– compute image statistics;– Dana & Nayar 97, 98, 99; Koenderink et al 96, 98;

Leung & Malik 97; Chantler et al 97, 98;

• Computer graphics models:– bump maps, displacement maps, point clouds, etc. – difficult to obtain for natural materials;


Problem Formulation

Image Database

Recognize

new sample

of different

light/view

Task


Main Idea

• Natural materials are made up of local features (geometric and photometric);

• There exists a universal set of local features for all materials;

• How these local features change appearance with different illumination and viewing directions determine how the materials look.


Outline• Learning the universal vocabulary of local

structures

• Material models

• Results



structures

– Introduce 2D textons for planar texture;

– Extend to 3D textons for natural materials;

• Material models

• Results


2D Textons

• Julesz suggests a universal vocabulary for such features --- textons [Julesz 81];

• crossings, line-ends, junctions, etc…

• Define textons for real images.


2D Textons• Goal: find canonical local features in a texture;

1) Filter image with linear filters:

2) Vector quantization on filter outputs;

3) Quantization centers are the textons.

• Spatial distribution of textons defines the texture;


2D Textons (cont’d)


3D Textons

• Consider textures with 3D features, e.g. bumps, grooves, ridges, etc…

• Want textons to capture local 3D geometric and photometric features;

• One image is ambiguous: different features can look the same under certain illumination and viewing conditions;

• More images will discriminate between the different cases.


Learning 3D Textons

Rough Plastic Concrete

Light/view 1

Light/view 2

Light/view N

3D textons

Texton 1

Texton K

Texton 2


Algorithm for Learning Vocabulary

• Register all 20 images for each material;

• Filter images with filter bank of 48 kernels;

• Concatenate filter responses of the 20 images;

• Each pixel becomes a 960 (20x48) dimensional feature vector;

• Apply K-means to the feature vectors of all materials together;

• Resulting centers are the 3D textons.


Algorithm for 3D Textons


Universal 3D Texton Vocabulary

• Columbia-Utrecht Database (60 materials, each with 205 images)

• Vocabulary of textons learned from 20 training materials;

• Use 20 different light/view images for each material.


Examples of 3D Textons

Texton 1

Texton 2

Texton 3

Texton 4

Texton 5

Texton 6

Texton 7

Different illuminationand viewing directions


Quantization Errors

• Reconstruct images after quantization;• SSD error within 5%.



structures;

• Material models;

– Image to texton representation;

– Material representation using textons;

• Results.


Texton Labeling

• Each pixel labeled to texton i (1 to K) which is most similar in appearance;

• Similarity measured by the Euclidean distance between the filter responses;


Material Representation

• Each material is now represented as a spatial arrangement of symbols from the texton vocabulary;

• Recognition --- ignore spatial arrangement, use histogram (K=100);


Histogram Models for Recognition

Terrycloth

Rough Plastic

Pebbles

Plaster-b


Similarity of materials

• Similarity between histograms measured using chi-square difference:

N

n nhnh

nhnhhh

1 21

221

212

)()(

))()((),(


Similarity Matrix

j) sample , i (materialSimilarity ije

Plaster-a Plaster-b

AluminumFoil

Cork



structures

• Material models

• Results– Material recognition from single image;– Synthesis of novel images.


Recognition from Single Image• 4 images to build histogram for model;

• 1 image of novel illumination and/or viewing directions to be recognized;

Image Database

?

Novel image


Novel Image from Material i?

• Build texton histogram for novel image.

• Compare with texton histogram for material i.

• However, texton labeling from 1 image is difficult, because in 1 light/view, several textons may have same appearance.

• Each pixel has N possible texton labels;

• Need to find the labeling that maximizes Similarity(novel image, material i)


Markov chain Monte Carlo for finding labeling

• Randomly label each pixel to one of N possibilities. Call this the initial state x(t),t=0

• Compute P(x(t)|material i);

• Obtain x’ by randomly changing M labels of x(t);

• Compute P(x’|material i);

• Compute

• If , the x’ is accepted, otherwise, accept with probability .

))((

)'(

txP

xP

1


P(detection) vs P(false alarm)


Synthesis of images with novel illumination and viewing directions

Map eachpixel totextons

Textons tellus how

appearancechanges


Synthesis of novel light/view images

• Keep exact spatial arrangement of textons


Synthesis Results

TextureMapping

GroundTruth

3D TextonModel

TextureMapping

GroundTruth

3D TextonModel

Plaster-a Concrete


Synthesis Results

TextureMapping

GroundTruth

3D TextonModel

TextureMapping

GroundTruth

3D TextonModel

Crumpled paper Plaster-b (zoomed)


Synthesis Results

TextureMapping

GroundTruth

3D TextonModel

TextureMapping

GroundTruth

3D TextonModel

Rough Plastic Sponge


Similarity to Appearance-based Object Recognition

• Object Recognition: objects are represented by a collection of images under different illumination and viewing conditions;

• Material Recognition: materials are represented by 3D textons, each of which is represented by the appearances under different illumination and viewing conditions.


Conclusions

• Model natural materials through images;

• Learn a universal vocabulary of 3D textons;

• Use the vocabulary to

– recognize materials from a single image of novel illumination and viewing directions;

– synthesize materials at novel illumination and viewing directions.

recognizing surfaces using three-dimensional textons thomas leung and jitendra malik computer...

Documents

greece materials

texture slide

d textons contd slide

d features

d textons texton

greece learning

d textons julesz

d textons goal