shape representation shape representation - what for? 4 4 to reason about an entity we must first...

Shape Shape RepresentationRepresentation

Shape Representation - what for?Shape Representation - what for?

To reason about an entity we must first represent the entity.

• Real world

Image processing

• Shape representation

•Image understanding \ shape recognition

011001011001000100010001000000100111

00101001100111101

Shape Representation - what for?Shape Representation - what for?

TopicsTopics

DefinitionsAttributesPopular Strategies

TopicsTopics

DefinitionsDefinitionsObject Abstraction Representation

ObjectObject

An object is something that

can be seen or touched,

material thing

(Oxford dictionary)

AbstractionAbstraction

Idea of quality separate from

actual examples

(Oxford dictionary)

• ball

• sphere

RepresentationRepresentation

A way of symbolizing an object

2222 zyxr

Attributes Of A Good Attributes Of A Good RepresentationRepresentation

sufficient wide domain unique unambiguous generative stable convenient All will be explained in next All will be explained in next

slidesslides

SufficientSufficient

...Is this representation sufficient enough? Depends on the application

Wide DomainWide Domain

Able to represent many different classes of entities E.g. numbers for elements in a queue

123

UniqueUnique every distinct member of its domain has a

single distinct representation.

Not unique : dog dog dog

unique : pitbull collie cocker-spaniel

UnambiguousUnambiguous

An entity may have different representations but no two distinct entities may have a common representation.

3 three III 2 two II

GenerativeGenerative Capable of directly generating

(recovering) the represented entity

Chain-code :436476872832

12

3

45

6

7

8

StableStable small perturbations do not induce large changes

in the representation of the entity.

ConvenientConvenient A representation may exhibit all the characteristics

that we have discussed so far and yet not be convenient for a task.– e.g. an assembly line robotassembly line robot who’s task is to filter out

rectangels from other shapes might use chaine code representation.

Yet for a police computerized camera that is supposed to compare faces in the crowd to a data base of suspects this is not enough.

Popular Strategies for shape Popular Strategies for shape encodingencoding

Volume based :– Describe the object volumetrically

– Combination of primitive volumes commonly used, eg. cubes, tetrahedra, and discs

– Provide more access to global relationships

– Do not provide direct surface information about the object

– Can represent only closed (boundaryless) surfaces

e.g.

Symmetric Axis Transform

Popular Strategies for shape Popular Strategies for shape encodingencoding

Surface based :– Surface of object represented by a single closed

parametric grid.

– Better suited for partial surfaces.

E.g.

Parametric Bicubic Patches

Gaussian- Image Representations

Popular StrategiesPopular Strategies Others:

– Chain code.– Distance vs. Angle.– Fourier Transform Moment– Generalized Cylinders – Visual Potential

We will discuss these We will discuss these strategiesstrategies

Chain CodeChain Code

This example shows that the Chain Code is This example shows that the Chain Code is independent of Location, Starting Point and independent of Location, Starting Point and

orientationorientation

AttributesAttributes

1. wide domain

2. unique

3. unambiguous

4. generative - 2D only

5. stable - depends on tolerance

Distance vs. AngleDistance vs. Angle

Distance vs. AngleDistance vs. Angle

1. Find point of balance.

2. Measure distance to edges.

3. Plot the graph of distance vs. angle.

Taking the dist. vs. angle one step further we get ...

FourierFourier DescriptorDescriptor


1. wide domain

2. unique

3. unambiguous

4. generative

5. stable - depends upon tolerance

MomentsMoments 1. find center of mass

2. find the moment:

Feature extraction takes segmented image data and outputs a list of features which are combined to give the feature vector v.– Area of a binary object can be calculated by simply adding up the

pixel values in the neighbourhood.– Perimeter obtained by subtracting eroded object from original and

adding up the resulting pixel values.– Compactness is a basic roundness measurement. Circle has low

compactness.– Correlation is the convolution of an image with a reference shape.

Zero-and first-order moments give area and centroid.

Conclusion: Conclusion: How to calculate area How to calculate area and centroid position using moments.and centroid position using moments.

What are Moments?What are Moments?1. Used to calculate:

1. area,

2. position,

3. other more complex shape features such as elongation.

2. Provides a general framework.

3. For an image A, where each pixel brightness is denoted by Ai,j, the moment of order k+l is given by:

i j

jilk

kl Ajim ,

Area = Area = mm0000

i jji

i jji

i jji

i jji

lkkl

Am

AAjim

ji

lk

Ajim

,00

,,00

00

00

,

11

1,1

0,0

i j

jiAArea ,

Object PositionObject PositionCalculate the object position

for a specific example.

Using the same process show how you

can generate a general expression

for object position.

Show that this general expression is equal

to a combination of zero- and first-order moments.

j 1 2 3 4 5 6 7j 1 2 3 4 5 6 7i i 11 22 33 44 55 66 77

Determine pixel at center of pattern. Its position gives the object position.

Called the centroid,– denoted by (ic,jc).

ic is the average i value of all white pixels.

jc is the average j value of all white pixels.

Object PositionObject Position

How do we calculate How do we calculate jjcc??

j 1 2 3 4 5 6 7j 1 2 3 4 5 6 7i i 11 22 33 44 55 66 77

j 1 2 3 4 5 6 7j 1 2 3 4 5 6 7

Generate 1-d plot of the number of Generate 1-d plot of the number of pixels with a particular value of j versus jpixels with a particular value of j versus j

PositionPosition

j 1 2 3 4 5 6 7j 1 2 3 4 5 6 7

49.3

17/66172524124

55644322

c

c

c

j

j

Areaj

jc = sum of j values

over all white pixels divided by

the total number of white pixels Total number of white pixels =Area

Position Position jjcc

i jji

i jji

i jji

j ijic

ii

ii

i iii

ii

ii

iic

AAjAAjj

AreaAAAAAAAj

,,,,

7,6,5,4,3,2,1, 7654321

j 1 2 3 4 5 6 7j 1 2 3 4 5 6 7

i

jiA ,

Centroid PositionCentroid Position Centroid position, (ic,jc) can

be calculated using zero-, m00,

and first-order, m10 and m01, moments.

jc = m01 / m00

i jji

i jji

c

i jji

i jji

i jji

i jji

lkkl

A

Aj

j

AjAjim

lk

Am

Ajim

,

,

,,10

01

,00

,

1,0

How do we calculate How do we calculate iicc??

j 1 2 3 4 5 6 7j 1 2 3 4 5 6 7 i i 11 22 33 44 55 66 77

i i 11 22 33 44 55 66 77

PositionPosition

43.4

17/731714121516124

272635444322

c

c

c

i

i

Areai

ic = sum of i values over all pixels divided by the

total number of pixels Total number of pixels

=Areai 1 2 3 4 5 6 7i 1 2 3 4 5 6 7

Position Position iicc

i jji

i jji

i jji

i jjic

j jj

jj

jjj

jj

jj

jjc

AAiAAii

AreaAAAAAAAi

,,,,

,7,6,5,4,3,2,1 7654321

i 1 2 3 4 5 6 7i 1 2 3 4 5 6 7

j

jiA ,

Centroid PositionCentroid Position

Centroid position, (ic,jc) can be calculated using zero-, m00,

and first-order, m10 and m01, moments.

ic = m10 / m00

i jji

i jji

c

i jji

i jji

i jji

i jji

lkkl

A

Ai

i

AiAjim

lk

Am

Ajim

,

,

,,01

10

,00

,

0,1

Centroid PositionCentroid Position (ic,jc)=(4,4)

j 1 2 3 4 5 6 7j 1 2 3 4 5 6 7i i 11 22 33 44 55 66 77

XX

2-2-D Invariant D Invariant DescriptorsDescriptors

Why Invariant Descriptors?Why Invariant Descriptors? Shape-Based Image Retrieval 2-D Invariant Descriptors

– Invariant to translation, rotation, scale change

Moment invariants (region-based measure)– Hu's moment invariants [Hu‘61]

• M.-K. Hu, Pattern recognition by moment invariants, Proc. IRE, vol. 49, p.1428, Sept. 1961

– Taubin's moment invariants [Taubin‘92]– Flusser's moment invariants [Flusser‘93]– Zernike moments [Taegu‘80]

• M. R. Taegue, Image analysis via the general theory of moments, J. Opt. Soc, Amer., vol. 70, pp. 920-930, Aug. 1980

Fourier descriptors [Zhan‘72] (boundary-based measure)

• C. T. Zhan, R. S. Roskies, Fourier descriptors for plane closed curves, IEEE Trans. Comput., vol. C-21, 1972, pp. 269-281

What Kind of?What Kind of?

1,...,0))},(),({( Niiyix

1,...,0),()()( Niijyixiz

1

0

)2

exp()()(N

i N

kijizkZ

12

,...,2,1,...,2

},|)1(|

|)(|{

mmk

Z

kZ

Boundary pointsBoundary points

Considering in complex planeConsidering in complex plane

Fourier descriptorsFourier descriptors

NormalizationNormalization

Fourier descriptorsFourier descriptors

)8(),...,( 81 mFFF

What Kind of?What Kind of?

Syx

qppq yxSm

),(

)(

)(

)(,

)(

)(,)()()(

00

01

00

10

),( Sm

Smy

Sm

SmxyyxxS

Syx

qppq

,..3,2,12

,)(00

qpqp

S pqpq

Moment invariantsMoment invariants

Region pointsRegion points

MomentMoment

Central momentCentral moment

Normalized central momentNormalized central moment

}1),(|),{( yxfyxS

Various waysVarious ways

Moment InvariantsMoment Invariants

])()(3)[)(3(

])(3))[()(3(

))((4

])())[((

])())[()(3(

])(3))[()(3(

)()(

)3()3(

4)(

20321

2123003211230

20321

21230123003217

0321123011

20321

2123002206

20321

2123003210321

20321

21230123012305

20321

212304

20321

212303

211

202202

02201

H

H

H

H

H

H

H

M

M

M

M

M

M

M),...,( 71 HHH MMM

• HuHu[‘61] : 2nd-3rd order NCM -> 7 invariant moments[‘61] : 2nd-3rd order NCM -> 7 invariant moments

• TaubinTaubin & Cooper[‘92] : the concept of covariant matrix & Cooper[‘92] : the concept of covariant matrix

-> affine invariant moments-> affine invariant moments

)(/, 00'

'02

'11

'11

'20

11 SmM pqpq

ILLMll

lL T

11

2221

11 ,0

qp

Syxpq yylxxlxxl

Sm)]()([)]([

)(

12221

),(11

00

'

2

1,

2

1,, 21

'042

'131

'222

'131

'22

'311

'222

'312

'402

'22'

031'12

'211

'121

'21

'301'

12

cc

ccc

cc

ccc

Mcc

ccM

),...,( 81 TTT MMM

)(41

),(),(),( '04

'40122212221212 TT MMMMMM

CM of order 2 -> 2x2 matrixCM of order 2 -> 2x2 matrix

Lower triangular matrixLower triangular matrix

New momentsNew moments

3rd-4th order moments -> two matrices3rd-4th order moments -> two matrices

Moment invariantsMoment invariants

Moment InvariantsMoment Invariants

MomentMoment

MomentsMoments

since we get invariant values the moments are not affected by transition or rotation.

By normalizing the values, we can make the

moment indifferent to scaling

Attributes of momentsAttributes of moments

1. wide domain

2. unique - no

3. unambiguous - no

4. generative - no

5. stable - no

)2(

)34(

)61296

8186

12966(

)]()()([

3446(

)(

322

23104

21240122231220440

9006

22212310440

6005

230

3022130

20211123002

211

221

202201230

20220

033031112210211200330021120

032121120

21202

220032102

220031211

220

303

320

11004

2211230021221033011

212032120

7003

212

221

32130

3123003122130

203

230

10002

2110220

4001

F

F

F

F

F

F

M

M

M

M

M

M

),...,( 61 FFF MMM

• FlusserFlusser & Suk[‘93]: affine moment invariants & Suk[‘93]: affine moment invariants

• Taegue[‘80]: Taegue[‘80]: ZernikeZernike moments -> moment invariants moments -> moment invariants• Invariant under rotationInvariant under rotation

• Invariant under general 2-D affine transformationInvariant under general 2-D affine transformation

• Normalization -> invariant under translation and scalingNormalization -> invariant under translation and scaling• translate origin to centroidtranslate origin to centroid• scale so that max distance from centroid == 1scale so that max distance from centroid == 1• ignore first two momentsignore first two moments

2/|)|(

0

2

1,),(

*

!2/|)|(!2/|)|(!

)!()1()(

),exp()(),(,),(1

22

mn

s

snsnm

nmnmyxSyxnmnm

smnsmns

snR

jmRVVn

A

evenis||and,,...,1,0 mnnmnn

|| nmA

),...,( 101 ZZZ MMM

|||,| 1100 AA

|||,||,||,||,||,||,||,||,||,| 55535144424033312220 AAAAAAAAAA

What’s the Best?What’s the Best? 44Size : 337 x 145Size : 337 x 145Rotation : -400,400,8Rotation : -400,400,8Scaling : -2,2,0.04Scaling : -2,2,0.04

Recent Solution.Recent Solution. Difficulty (Current Problem in Digital Image

Analysis)– Moment invariants are not sufficient for distinguishing– Sensitive to noise

Region-based measure + boundary-based measure– Gives the better average retrieval efficiency

Two-stage similarity scheme – Moment invariants (1st retrieval stage) -> n images– Fourier descriptors (2nd verification stage) -> m < n

images

Comparison of computation time. (unit=sec)--> Indigo2 IMPACT (MIPS R1000), 337 x 145, 718 boundary points, 17761 region points

Method FD HM TM FM ZM FH FZ

Time 0.01 0.02 0.02 0.02 0.93 0.03 0.94

• Comparison to other similarity schemes (Top5 Comparison to other similarity schemes (Top5 / Top 10)/ Top 10)--> 10 rabbit images / 85 animal images--> 10 rabbit images / 85 animal images

Query\method FD HM TM FM ZM FH FZquery 1 5/8 4/7 4/7 2/3 5/9 5/8 5/9query 2 5/8 5/8 4/5 2/5 5/9 5/8 5/9query 3 5/8 2/6 1/1 1/3 4/8 5/8 5/8query 4 5/8 5/7 4/6 1/3 5/9 5/8 5/9

FD=Fourier descriptorsFD=Fourier descriptors

HM=Hu's moment invariantsHM=Hu's moment invariants

TM=Taubin's moment invariantsTM=Taubin's moment invariants

FM=Flusser's moment invariantsFM=Flusser's moment invariants

ZM=Zernike momentsZM=Zernike moments

FH=FD+HMFH=FD+HM

FZ=FD+ZMFZ=FD+ZM

Recent Solution.Recent Solution.

QueryQuery

FH/FZ Top 5FH/FZ Top 5

FH/FZ Top 10FH/FZ Top 10 FZ Top 10FZ Top 10 MissMiss

FalseFalse

Database: 1195 animal imagesDatabase: 1195 animal imagesSadegh Abbasi, Farzin Mokhtarian, Josef Kittler, S. SclaroffSadegh Abbasi, Farzin Mokhtarian, Josef Kittler, S. Sclaroffhttp://www.ee.surrey.ac.uk/Research/VSSP/imagedb/demo.htmhttp://www.ee.surrey.ac.uk/Research/VSSP/imagedb/demo.htmll

Parametric Parametric Bicubic PatchesBicubic Patches

Split the object to simpler bicubic patches that can be represented by relatively simple mathematical equations.

Piecewise Parametric SurfacesPiecewise Parametric Surfaces

Piecewise PatchesPiecewise Patches

Bezier PatchesBezier Patches


1. generative

2. stable

3. popular in computer graphics

4. Possibility of several equally acceptable bicubic approximations to any given surface makes it inappropriate for surface matching

Symmetric Axis Symmetric Axis TransformTransform

• A.k.a Blum Transform, Medial Axis Transform.• Formally : the object is the logical union of all its maximal discs.

May be thought of as grass fire spreading from the border inwards.

Description : – Locus of centers a.k.a symmetric axis.Locus of centers a.k.a symmetric axis.– Radius at each pointRadius at each point

Symmetric Axis TransformSymmetric Axis Transform


1. wide domain

2. unique

3. unambiguous

4. generative

5. not stable - small changes affect dramatically

6. Primarily used in biological applications

Generalized Generalized CylindersCylinders

Binford 1971 an extension of SAT. Shape represented by an ordinary cylinder sweeping the cross-section along

an arbitrary space curve (axis/spine)

Generalized CylindersGeneralized Cylinders

Decomposition of 3D shape-description problems into lower-order problems

Generalized CylindersGeneralized CylindersA generalized cylinder is thus defined by

a cross section, a cross section, an axisan axis a sweeping rule. a sweeping rule.

Generalized CylindersGeneralized Cylinders


1. wide domain - depends on implementation (redundancy vs. wide domain )

2. unique

3. generative

4. stability is doubtful

5. Decomposition of complex structures is difficult.

6. Used mainly for object recognition

7. been used in working systems.

Gaussian Gaussian ImageImage

Surface normal vector information for any object can be mapped onto a unit sphere, called the Gaussian sphere.

Mapping is called the Gaussian image of the object.

Extended Gaussian ImageExtended Gaussian Image

The mapping is: Surface normals for each point of the object are placed so that their

tails lie at the center of the Gaussian spheretails lie at the center of the Gaussian sphere

heads lie on a point on the sphere appropriate to heads lie on a point on the sphere appropriate to the particular surface orientationthe particular surface orientation

Extended Gaussian Image (EGI)Extended Gaussian Image (EGI)

We can extend this process so that a weight is assigned to each point on the Gaussian sphere

equal to the area of the surface having the given normal This mapping is called the extended Gaussian image (EGI). Weights are represented by vectors parallel to the surface

normals, with length equal to the weight

EGIEGIExtended Gaussian Image (EGI)Extended Gaussian Image (EGI)


1. wide domain

2. not unique

3. not unambiguous

4. not generative

5. stable - with limitations

6. invariant to translation and scaling

7. position independence

Visual Visual PotentialPotential

Aspect - topological structure of the singularities in a single view.

A graph in which each node represents an aspect of the object, and each edge the possibility of transiting from one aspect to another under motion of the observer [Turner74].

Represents in a concise way any visual experience an observer can obtain by looking at the object when traversing any orbit through space.

Visual PotentialVisual Potential


1. wide domain

2. unique

3. unambiguous

4. generative

5. not stable

SummarySummary

The above methods are deterministic; in practice, uncertainties in shape that result from the noise in measurement have to be considered [Ayache88, Ikeuchi88]

The above methods are based on geometric models, they are more appropriate for describing specific objects, particularly artificial ones with regular structures

ConclusionConclusion

The Proposed Hybrid Method– FD + Hu's/Zernike MI– FD + Hu's MI

Future Works– Invariant to 3-D perspective

transform– Recognition of partially

recovered/occluded objects

SummarySummary

Symbolic models are more appropriate for natural objects that are better defined in terms of generic characteristics (e.g.. small, red, rough) than precise shape; useful for matching

ReferencesReferences

Books :– Nalwa “a guided tour of computer vision”

– Gonzales, “Image analysis and computer vision”

– Jain

Web:– http://web.mit.edu/manoli/ecimorph/www/code/MMorph.html

– http://www.cs.ubc.ca/~clee/cg/proj/index.html

– http://www.dai.ed.ac.uk/CVonline/LOCAL_COPIES/MARSHALL/node53.html#figuresame_EGI

– http://www.cs.uwa.edu.au/~cheng/index.html

[email protected]@[email protected]@etri.re.krHoward Schultz, UMass Howard Schultz, UMass Yaron BerlinskyYaron Berlinsky

SourcesSources

shape representation shape representation - what for? 4 4 to reason about an entity we must first...

Documents

shape representation

strategies slide

angle slide

tolerance slide

orientation slide

application slide

slides slide

chain code slide