content based image retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · content based image...

Content Based Image Content Based Image RetrievalRetrievalNatalia [email protected]

HP Labs Russia

© 2008 Hewlett-Packard Company

Tutorial outlineTutorial outline• Lecture 1

− Introduction− Introduction

− Applications

• Lecture 2− Performance measurement− Performance measurement

− Visual perception

− Color features

• Lecture 3• Lecture 3− Texture features

− Shape features

− Fusion methods− Fusion methods

• Lecture 4− Segmentation

Local descriptors− Local descriptors

• Lecture 5− Multidimensional indexing

2/51

− Survey of existing systems

Lecture 3Lecture 3

Texture featuresTexture features

Shape featuresShape features

Fusion methods

Lecture 3: Outline

• Texture features

Lecture 3: Outline

• Texture features• Texture features

− Statistical

− Spectral


− Statistical

− Spectral− Spectral

− Comparison

− Spectral

− Comparison

• Shape features

− Boundary based

• Shape features

− Boundary based− Boundary based

− Region based

− Comparison

− Boundary based

− Region based

− Comparison− Comparison

• Fusion methods

− Comparison

• Fusion methods

4/51


• What is texture?• What is texture?

5/51

Smooth Rough Regular


Texture features

General statistics parameters

Haralick’s co-occurrence matrices

Statistical

PWT

Spectral

Model basedTamura features

Geometrical

TWT

DCT, DST, DHT

Complex wavelets

Gabor filters

ICA filters

Markov random fields

Fractals

Model-based

Voronoi tesselation features

Structural methods

GeometricalICA filters

6/51


• General statistics• General statistics

Based on intensity histogram of the whole image or its regions:

– histogram of intensity, L – number of intensity levels.– histogram of intensity, L – number of intensity levels.

– central moment of order n.

– average intensity.

– variance, is a measure of contrast.

, R=0 where intensity is equal., R=0 where intensity is equal.

– a measure of histogram assimetry.

7/51

– a measure of histogram assimetry.


• General statistics (2)• General statistics (2)

– a measure of contrast of homogeneity

(max for homogeneous areas ).(max for homogeneous areas ).

– entropy, a measure of variability

(0 for homogeneous areas ).(0 for homogeneous areas ).

Texture Average Deviation R µ3 U Entropy

SmoothSmooth

Rough

Regular

8/51


Grey Level Co-occurrence Matrices (GLCM):Grey Level Co-occurrence Matrices (GLCM):

GLCM - matrix of frequencies at which two pixels, separated by a certain vector, occur in the image.

∑∑ =∆+∆+=

=N M

otherwise

jyqxpIiqpIifjiC

0

1

,

),(,),(,),( ∑∑

= = =

p q otherwisejiC

1 1 0,),(

),( yx ∆∆ – separation vector;

I(p,q) – intensity of a pixel in position (p, q).

9/51

GLCM – an exampleGLCM – an example

10/51

)0,1(),( =∆∆ yx

GLCM – descriptorsGLCM – descriptors

Statistical parameters calculated from GLCM values:

∑∑= jiCEnergy ),(2

Statistical parameters calculated from GLCM values:

– is minimal when all elements are equal∑∑=i j

jiCEnergy ),(

∑∑−= jiCjiCEntropy ),(log),(

– is minimal when all elements are equal

– a measure of chaos, is maximal when all elements are equal

∑∑i j

∑∑ −=i j

jiCjiContrast ),()( 2

is maximal when all elements are equal

– has small values when big elements are near the main diagonal

∑∑i j

∑∑ −+=

i j ji

jiCMomentDifferenceInverse

2)(1

),(

near the main diagonal

−+i j ji )(1

– has small values when big elements are far from the main diagonal

11/51

Texture features: Tamura featuresTexture features: Tamura features

Features, which are important for visual perception:Features, which are important for visual perception:

� Coarseness Tamura image:Coarseness

� Contrast

� Directionality

Coarseness-coNtrast-Directionality –

points in 3-D space CND

� Directionality

� Line-likeness

Regularity

� Euclidean distance in 3D (QBIC)

� 3D histogram (Mars)

Features:

� Regularity

� Roughness

� 3D histogram (Mars)

12/51

Texture features: spectralTexture features: spectral

Texture features

General statistics parameters

Haralick’s co-occurrence matrices

Statistical

PWT

Spectral

Model basedTamura features

Geometrical

TWT

DCT, DST, DHT

Complex wavelets

Gabor filters

ICA filters

Markov random fields

Fractals

Model-based

Voronoi tesselation features

Structural methods

GeometricalICA filters

13/51

Texture features: wavelet basedTexture features: wavelet basedWavelet analysis – decomposition of a signal:

)()( ,

,

xxf kj

kj

kψα∑=

Basis functions: Basis functions:

)()(,,

)2(2

2

2/

,

RLxkj

kxjj

kj

∈Ζ∈

−=

ϕ

ϕψ – scaling function

– mother wavelet

A set of basis functions – filters bank

Image

– mother wavelet

Filter 1

Image

Energy 1Filter 1

Filter 2

Filter N

Energy 1

Energy 2

Energy N

Feature vector

14/51

Filter N Energy N

Texture features: Gabor filtersTexture features: Gabor filters

Mother wavelet: Gabor functionMother wavelet: Gabor function

+

+−

= jWxyx

yxg π21exp

1),(

22

+

+−

= jWx

yxyxg

yxyx

πσσσπσ

22

1exp

2

1),(

22

1,-S0,1,...,minteger,,,1),,(),( ==>′′= − nmayxgayxg m

mn

Filters bank:

),cossin(

),sincos(

1,-S0,1,...,minteger,,,1),,(),(

Θ+Θ−=′Θ+Θ=′

==>′′=

−

−

yxay

yxax

nmayxgayxg

m

m

mn

),cossin( Θ+Θ−=′ yxay

Kn /π=Θ)1/(1)/( −−= S

lh UUa

К – a number of directions, S – a number of scales,Uh, Ul – max and min of frequencies taken into consideration.

15/51

lh

Texture features: ICA filtersTexture features: ICA filtersFilters are obtained using Independent Component Analysis

I1

dist(I1,I2) = KLH(H1i , H2i)Σi=1

N

H. Borgne, A. Guerin-Dugue, A. Antoniadis.

Representation of images for classification

with independent features. Pattern with independent features. Pattern

Recognition Letters, vol. 25, p. 141-154,

2004I2

…

N filters

16/51

N filters

ICA FiltersICA Filters

17/51

Lecture 3: Outline


Lecture 3: Outline


− Statistical


− Comparison

• Shape features


− Region based


• Fusion methods

18/51

Texture features: comparisonTexture features: comparison

In the context of In the context of image retrieval!

P. Howarth, S. Rüger. Robust texture features for still image retrieval.

19/51

P. Howarth, S. Rüger. Robust texture features for still image retrieval. In Proc. IEE Vis. Image Signal Processing, vol. 152, No. 6, December 2006

Texture features: comparison (2)Texture features: comparison (2)

Gabor filters v. s. ICA filtersGabor filters v. s. ICA filters

Image classification task:

� Collection of angiographic images

� ICA filters performs better by 13%� ICA filters performs better by 13%

� Brodatz texture collection

� ICA filters perform better by 4%

Snitkowska, E. Kasprzak, W. Independent Component Analysis of Textures in Angiography Images. Computational Imaging and Vision, vol. 32, pages 367-372, 2006.

20/51

Lecture 3: Outline


Lecture 3: Outline


− Statistical


− Comparison

• Shape features


− Region based


• Fusion methods

21/51



Boundary-based methodsRegion-based methods

Perimeter

EccentricityCentroid Distance

Complex Coordinates

Gometrical Signatures

Geometrical

Moment invariants

Global

Curvature

Axes directionality

Complex Coordinates

Curvature signature

Turning Angle

Area

Compactness

Euler number

Moment invariants

Zernike moments

Pseudo Zernike moments

Grid method

Others

Fourier Descriptors

UNL-Fourier

NFD

Wavelet Descriptors

Signature descriptors

Triangulation

Medial Axis Transform

(Skeleton Transform)

Decompasition

Chain codes

Others

Wavelet Descriptors

B-Splines

( )

Spectral descriptors

22/51

Requirements to the shape featuresRequirements to the shape features

� Translation invariance

� Scale invariance� Scale invariance

� Rotational invariance� Rotational invariance

� Stability against small form changes

Low computation complexity� Low computation complexity

� Low comparison complexity� Low comparison complexity

23/51

Boundary-based featuresBoundary-based features

Shape features

Boundary-based methodsRegion based methodsRegion-based methods

Geometrical

Signatures GeometricalGlobal

Signature desciptors

Geometrical

Decomposition Others

Fourier Descriptors

NFD

...

Chain codes

24/51

Chain codesChain codesDirections for 4-connected and 8-connected chain codes:

A: 03001033332322121111 A: 03001033332322121111

B: 70016665533222

Example:Example:

0 0

0 03 1 3

31

10 0

1 6

62

2 7

Starting point invariance: minimal code

70016665533222 -> 00166655332227

22

22

3

3

31

1

1 6

5

53

3

270016665533222 -> 00166655332227

Rotation invariance: codes subtractiona) б) в)

A B

Rotation invariance: codes subtraction

00166655332227 -> 01500706070051

25/51

Fourier descriptorsFourier descriptors1. Signature calculation (2D -> 1D):

� Centroid – contour distance� Centroid – contour distance

� Complex coordinates: z(t) = x(t) + iy(t)

� ...

2. Perform the discrete Fourier transform, take coefficients (s(t) – signature):

∑−

−=1

/2)(1 N

Nntjetsu π∑=

−=0

/2)(1

t

Nntj

n etsN

u π

3. Normalization (NFD – Normalized Fourier Descriptors):

0

1

0

2

0

1,...,,

u

u

u

u

u

u N−

2

12

)(∑ −=cN

n

J

n

I ffd

4. Comparison:

26/51

0

)(∑=

−=n

JI ffd

Region-based featuresRegion-based features


Boundary-based methodsRegion-based methods

Geometrical

Signatures

Geometrical

Global

Signature representation

Moment invariants

Zernike moments

Pseudo Zernike moments

Grid method

DecompositionOthers

27/51

Grid-methodGrid-methodА: 001111000 011111111 111111111 111111111 111110111 0111000011А

Б: 001100000 011100000 111100000 111101111 111111110 001111000Б

Invariance:

Normalization by major axe:

� direction;

� scale;� scale;

� position.

28/51

Moment invariantsMoment invariantsThe moment of order (p+q) for a two-dimension continuous function:

∫∫= dxdyyxfyxm qp

pq ),(

Central moments for f(x,y) – discrete image:

mmqp ==−−=∑∑µ00

01

00

10 ,),,()()(m

my

m

mxyxfyyxx

x y

qp

pq ==−−=∑∑µ

Seven scale, translation and rotation invariant moments were derived based

Feature vector:

Seven scale, translation and rotation invariant moments were derived based

on central normalized moments of order p + q = 2; 3.

29/51

Lecture 3: Outline


Lecture 3: Outline


− Statistical


− Comparison

• Shape features


− Region based


• Fusion methods

30/51

Shape features comparisonShape features comparison

Mehtre B. M., Kankanhalli M. S., Lee W. F. Shape measures for content based image retrieval: a comparison. Inf. Processing and Management, vol. 33, No. 3, pages 319-337, 1997.

31/51

comparison. Inf. Processing and Management, vol. 33, No. 3, pages 319-337, 1997.

Lecture 3: Outline


Lecture 3: Outline


− Statistical


− Comparison

• Shape features


− Region based


• Fusion methods

32/51

Data fusion in CBIRData fusion in CBIR

color (2)annotations

texture shapecolor

� Combined search

fusion

� Combined search

(different features)

� Refine search results fusion� Refine search results

(different algorithms for the same

feature)

result� Supplement search results

(different datasets)

33/51

Fusion of retrieval result sets

Fusion of weighted lists with ranked elements:

(x11, r

11), (x

12, r

12), … , (x1

n, r1n)ω1

(x21, r

21), (x

22, r

22), … , (x2

k, r2n)ω2

…?

(xm1, r

m1), (x

m2, r

m2), … , (xm

l, rm

l)ωm

Existing approaches in text retrieval:Existing approaches in text retrieval:

� CombMax, CombMin, CombSum

� CombAVG� CombAVG

� CombMNZ = CombSUM * number of nonzero similarities

� ProbFuse

34/51

� ProbFuse

� HSC3D

Fusion function: propertiesFusion function: properties

1) Depend on both weight and rank1) Depend on both weight and rank

2) Symmetric

3) Monotony by weight and rank

},...,,max{},...,,min{)()()()0()()()( 2121 NN rrrrrrr

αααααα ≤≤

3) Monotony by weight and rank

4) MinMax condition /CombMin, CombMax, CombAVG/:

},...,,max{},...,,min{)()()()0()()()( 2121 NN

xxxxxxx rrrrrrrαααααα ≤≤

5) Additional property – “conic” property: non-linear dependency from 5) Additional property – “conic” property: non-linear dependency from

weight and rank; high weight, high rank – influence bigger to the

result than several inputs with low weight, low rank.

35/51

Weighted Total with Gravitation Function

CombAVG as a base, but use gravitation function instead of weight:CombAVG as a base, but use gravitation function instead of weight:

where

36/51

WTGF: some resultsWTGF: some results

• Experiments on search in semi annotated collections and • Experiments on search in semi annotated collections and of color and texture fusion (compare with CombMNZ)

• WTGF is good when:• WTGF is good when:

− There are a lot of viewpoints.

− Viewpoints are very different (different opinions regarding the rank − Viewpoints are very different (different opinions regarding the rank of the same element).

− Viewpoints have different reliability.

• CombMNZ is good when:

− Viewpoints have the same reliability.

Viewpoints have similar opinions.− Viewpoints have similar opinions.

Natalia Vassilieva, Alexander Dolnik, Ilya Markov. Image Retrieval. Combining multiple search

37/51

Natalia Vassilieva, Alexander Dolnik, Ilya Markov. Image Retrieval. Combining multiple search

methods’ results. In "Internet-mathematics" Collection, 46—55, 2007.

Adaptive merge: color and textureAdaptive merge: color and texture

Dist(I, Q) = α*C(I, Q) + (1 - α)*Т(I, Q),Dist(I, Q) = α*C(I, Q) + (1 - α)*Т(I, Q),

C(I, Q) – color distance between I and Q;

T(I, Q) – texture distance between I and Q; T(I, Q) – texture distance between I and Q;

0 ≤ α ≤ 1

Hypothesis:

Optimal α depends on features of query Q. It is possible Optimal α depends on features of query Q. It is possible to distinguish common features for images that have the same “best” α.

Ilya Markov, Natalia Vassilieva, Alexander Yaremchuk. Image retrieval. Optimal weights for color

and texture fusion based on query object. In Proceedings of the Ninth National Russian Research

38/51

and texture fusion based on query object. In Proceedings of the Ninth National Russian Research Conference RCDL'2007

Example: texture searchExample: texture search

39/51

Example: color searchExample: color search

40/51

Mixed metrics: semantic groupsMixed metrics: semantic groups

2a

1a

a41/51

3a

Experimental results 1Experimental results 1

• It is possible to select the best value of a• It is possible to select the best value of a

0,6

0,4

0,5

Pre

cis

ion

0,2

0,3

полнота Кластер 6

Кластер 7

Кластер 8

Pre

cis

ion

Cluster 7

Cluster 6

Cluster 8

0,1

0,2

0

0 0,25 0,5 0,75 1

коэффициент смешанной метрики (а)Value of a

42/51

Experimental results 2Experimental results 2

• Adaptive mixed-metrics increase precision • Adaptive mixed-metrics increase precision

43/51

Adaptive merge: color and colorAdaptive merge: color and color1,2

0,8

1

0,6

0,8

Pre

cis

ion

0,2

0,4

0

0 1 2 3 4 5 6 7 8 9 10

Top N

Moments, Clouds Moments, Fields

HSL, 6*2*2, Clouds HSL, 6*2*2, Fields

44/51

Adaptive merge: color and colorAdaptive merge: color and color

1,2

1

1,2

0,6

0,8

Pre

cis

ion

0,2

0,4

Pre

cis

ion

0

0 1 2 3 4 5 6 7 8 9 10

Top NTop N

Moments, People Moments, Bears

HSL, 6*2*2, People HSL, 6*2*2, Bears

45/51

Color fusionColor fusion

CombMNZ (Moments + HSL histogram)CombMNZ (Moments + HSL histogram)

0,8

0,6

0,7

0,4

0,5

Moments

HSL, 6*2*2

Color mixed

0,2

0,3

Color mixed

0

0,1

46/51

0

0 1 2 3 4 5 6 7 8 9 10

Ranked lists fusion: application area

� Search by textual query in semi annotated image � Search by textual query in semi annotated image collection

Textual query

…tr1by annotations

base

d

TextResult1, textrank1…tr2

conte

nt-

base

d

Result

TR2, tr2,

... …conte

nt

47/51

Retrieve by text: fusion resultsRetrieve by text: fusion results

∑

⋅=Moverlap

xRMxR

)()(

)0(

α∑M

i

xR i )()(α

∑

⋅=M

i

overlap

xR

xRMxR

i )(

)()(

)(

)0(

α

⋅= xNMxN

)()(

)0(

Size of input lists ∑=

M

i

overlap

xN

xN

i )(

)()(α

48/51

Lecture 3: ResumeLecture 3: Resume

• Texture features• Texture features

− Statistics (Haralik’s co-occurance matrices, Tamura features)

− Spectral features are more efficient (Gabor filters, ICA filters)− Spectral features are more efficient (Gabor filters, ICA filters)

• Shape features

− Boundary-based (Fourier descriptors)

− Region-based (Moment invariants)

• Fusion methods

Are very important− Are very important

− Need to choose based on a particular fusion task

49/51

Lecture 3: BibliographyLecture 3: Bibliography

• Haralick R. M., Shanmugam K., Dienstein I. Textural features for image classification. In IEEE Transactions on Systems, Man and Cybernetics, classification. In IEEE Transactions on Systems, Man and Cybernetics, vol. 3(6), pp. 610 – 621, Nov. 1973.

• Tamura H., Mori S., Yamawaki T. Textural features corresponding to visual perception. In IEEE Transactions on Systems, Man and visual perception. In IEEE Transactions on Systems, Man and Cybernetics, vol. 8, pp. 460 – 472, 1978.

• Tuceryan M., Jain A. K. Texture analysis. The Handbook of Pattern Recognition and Computer Vision (2nd Edition), by C. H. Chen, L. F. Pau, Recognition and Computer Vision (2nd Edition), by C. H. Chen, L. F. Pau, P. S. P. Wang (eds.), pp. 207-248, World Scientific Publishing Co., 1998.

• Tuceryan M., Jain A. Texture segmentation using Voronoi polygons. In IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, No 2, pp. 211 – 216, February 1990.

• Walker R., Jackway P., Longstaff I. D. Improving co-occurrence matrix feature discrimination. In Proc. of DICTA’95, The 3rd Conference on feature discrimination. In Proc. of DICTA’95, The 3rd Conference on Digital Image Computing: Techniques and Applications, pp. 643 – 648, 6-8 December, 1995.

50/51

Lecture 3: BibliographyLecture 3: Bibliography• Li B., Ma S. D. On the relation between region and contour representation. In Proc.

of the IEEE International Conference on Pattern Recognition, vol. 1, pp. 352 – 355, of the IEEE International Conference on Pattern Recognition, vol. 1, pp. 352 – 355, 1994.

• Lin T.-W., Chou Y.-F. A Comparative Study of Zernike Moments for Image Retrieval. In Proc. of 16th IPPR Conference on Computer Vision, Graphics and Image Processing (CVGIP 2003), pp. 621 – 629, 2003.Image Processing (CVGIP 2003), pp. 621 – 629, 2003.

• Loncaric S. A survey of shape analysis techniques. In Pattern Recognition, vol. 31(8), pp. 983 – 1001, 1998.

• Luren Y., Fritz A. Fast computation of invariant geometric moments: A new method • Luren Y., Fritz A. Fast computation of invariant geometric moments: A new method giving correct results. In Proc. of IEEE International Conference on Image Processing, 1994.

• Zakaria M. F., Vroomen L. J., Zsombor-Murray P. J. A., van Kessel J. M. H. M. Fast algorithm for the computation of moment invariants. In Pattern Recognition, vol.

• Zakaria M. F., Vroomen L. J., Zsombor-Murray P. J. A., van Kessel J. M. H. M. Fast algorithm for the computation of moment invariants. In Pattern Recognition, vol. 20(6), pp. 639 – 643, 1987.

• Zernike polynomials. Wikipedia, the free encyclopedia. http://en.wikipedia.org/wiki/Zernike_polynomialshttp://en.wikipedia.org/wiki/Zernike_polynomials

• Zhang D., Lu G. Shape-based image retrieval using generic Fourier descriptor. In Signal Processing: Image Communication, vol. 17, pp. 825 – 848, 2002.

• Zhang D., Lu G. A Comparative Study on Shape Retrieval Using Fourier

51/51

• Zhang D., Lu G. A Comparative Study on Shape Retrieval Using Fourier Descriptors with Different Shape Signatures. In Proc. of the International Conference on Multimedia, 2001.

content based image retrievalromip.ru/russir2008/slides/cbir/lecture-3.pdf · content based image...

Documents