e.g.m. petrakisvisual content1 retrieval of visual content images comprise the vast majority of...
Post on 22-Dec-2015
219 views
TRANSCRIPT
![Page 1: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/1.jpg)
E.G.M. Petrakis Visual Content 1
Retrieval of Visual Content
Images comprise the vast majority of data in many application domains Remote sensing (NASA, 1 terabyte per
day)AstronomyGeographic Information Systems (GIS)Medicine (CT, MRI, etc.)Criminal investigationTrademark authentication
![Page 2: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/2.jpg)
E.G.M. Petrakis Visual Content 2
Images in Multimedia Systems
Images co-exist with other types of data in Multimedia Documentstextattributevideosound
![Page 3: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/3.jpg)
E.G.M. Petrakis Visual Content 3
Content-Based Image Retrieval
Descriptions of image content are extracted and stored
Manually: mainly text descriptions DifficultSubjective
Automatically: features from contentComputationally expensiveInexactDomain specific
![Page 4: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/4.jpg)
E.G.M. Petrakis Visual Content 4
System Architecture
![Page 5: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/5.jpg)
E.G.M. Petrakis Visual Content 5
Design Issues
Feature Extraction (functions)Feature SelectionOrganization of stored information, file
structures, indexingSearch and retrieval strategies
Sequential / Indexed search / Query refinement
Query language: conditional / example queries
User interface design
![Page 6: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/6.jpg)
E.G.M. Petrakis Visual Content 6
Image Descriptions
Subjective interpretation of content: means different things to different people
Different features for different applicationsColour is important of out-door image but not
for X-rays, CT, MRI etc.Motion features are sometimes important
(ultrasound)
Different systems for different applications
![Page 7: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/7.jpg)
E.G.M. Petrakis Visual Content 7
Levels Representation
Low at pixel level (e.g., intensities, colors)
Intermediate at region level (e.g., region, shape, motion features, motion)
High – Semantic human interpretations (e.g., a class per object or image or domain concepts such as diagnosis …)
![Page 8: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/8.jpg)
E.G.M. Petrakis Visual Content 8
Conflicting issues
Dependence on image content, computational overhead and uncertainty increases from low to high level
Selection depends on application, image type, user requirements, query types
![Page 9: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/9.jpg)
E.G.M. Petrakis Visual Content 9
Reliability Criteria
Uniqueness Proportionality of variationRobustness against noiseInvariance under translation,
rotation, scalingComputationally efficientContent at various level of detail
![Page 10: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/10.jpg)
E.G.M. Petrakis Visual Content 10
Generic Features
Feature vectors of intensity / colortexturespatial relationshipsmotioncombinations of the above
Two kinds of featuresglobal: computed for the entire imagelocal: computed for objects or image
parts
![Page 11: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/11.jpg)
E.G.M. Petrakis Visual Content 11
RGB Color Space
Popular hardware oriented scheme
Colors form a unit cube r = R/(R+G+B)g = G/(R+G+B)b = B/(R+G+B)
RGB is good for acquisition and display but not for the perception of colors
![Page 12: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/12.jpg)
E.G.M. Petrakis Visual Content 12
Munsell Color Space
Color in cylindrical coordinates Brightness: vertical
axis Hue: angular
displacement Saturation:
cylindrical radius
![Page 13: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/13.jpg)
E.G.M. Petrakis Visual Content 13
Color Definitions
Brightness: intensity of color, average intensity over all wavelengths
Hue: proportional to the average wavelength of the color percept
Saturation: amount of white, highly saturated colors have no whitedeep red has S=1pinks have S=0
![Page 14: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/14.jpg)
E.G.M. Petrakis Visual Content 14
HSV Color Space
Value
Hue
Saturation
H = undefined for S = 0H = 360 – H if B/V > G/V
)++(31
= BGRV
))(()(2
212
cosBGBRGR
BGRH
),,min(1= ++3 BGRS BGR-
![Page 15: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/15.jpg)
E.G.M. Petrakis Visual Content 15
Color in Retrievals
Color Histograms are very commonSimple to compute and compareFor the entire image or for image parts 3D histogram on RGB or HSV space (224 bins!)1D histogram over the 3 primaries (256 bins)
Use HSV histograms: changes in lighting and viewing angles may cause major variations in RGB histograms
Invariant under translation, rotation, viewing angle and scaling
![Page 16: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/16.jpg)
E.G.M. Petrakis Visual Content 16
A.Del Bimbo 99
1D histogram
![Page 17: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/17.jpg)
E.G.M. Petrakis Visual Content 17
Histogram Comparison
Histogram intersectionQ, I: histograms of a
query and database imageN: histogram bins3D (RGB, HSV)
intersection is defined accordingly
N
i i
N
i ii
Q
QIQIS
1
1),min(
),(
A.Del Bimbo 99
normalized intersection
![Page 18: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/18.jpg)
E.G.M. Petrakis Visual Content 18
Reducing Complexity
Reduce number of histogram binsTransform RGB histogram to (rg,by,wb)rg = R – G, by = 2B – R – G, wb = R + G +
BIntensity wb is more coarsely sampled than
rg, by wb (8 sections), rg, by (16 sections)The resulting histogram has 2048 binsReduced sensitivity to variations of
intensity
![Page 19: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/19.jpg)
E.G.M. Petrakis Visual Content 19
Reducing Complexity (cont,d)
Clustering detects the K most prominent colors (e.g., K-means)Histogram with K bins (e.g., K=64 or
256)Each bin is the normalized count of
pixels in the cluster
![Page 20: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/20.jpg)
E.G.M. Petrakis Visual Content 20
Reducing Complexity (cont,d)
Recognize that only a small number of bins capture the majority of pixelsThreshold to take only the large binsSmall bins are likely to be noisy bins
thus distorting the intersectionDoes not degrade the performance
![Page 21: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/21.jpg)
E.G.M. Petrakis Visual Content 21
Distance Function
Certain pairs of bins correspond to perceptually similar colorsIn intersection all bins are compared
independently of each otherDefine new Distance function:
A=(aij) represents bin proximity
aij based on proximity in the L*u*v space
K
i
K
j jijiijt yxyxaQIAQIQID
1 1
2 ))(()()(),(
![Page 22: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/22.jpg)
E.G.M. Petrakis Visual Content 22
A.Del Bimbo 99
L*u*v color space
![Page 23: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/23.jpg)
E.G.M. Petrakis Visual Content 23
Color Indexing
Color (feature) vector: histogram Problems:
K is large (K=64 or 256)Quadratic complexity of matchingSAMs assume independent attributes
Solution: GEMINIMap to low dimensionality feature space Lower bound distance: Df(I,Q) <= D(I,Q)
![Page 24: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/24.jpg)
E.G.M. Petrakis Visual Content 24
Definition of Df(I,Q)
Take some average color value on color space (e.g., R,G,B)average color of image: (Ravg,Gavg,Bavg)=
and
P
i
P
i
P
i
avgavgavg
pBPpGPpRP
BGR
111)(/1),(/1),(/1
),,(
3
1
2)-()-()-(),(i ii
tf yxyxyxQID
![Page 25: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/25.jpg)
E.G.M. Petrakis Visual Content 25
GEMINI Approach
Indexing in the 3D color space Df < D(I,Q): see QBIC paper for proof
Map query Q to the same 3D space Search the feature space Clean-up answer set to eliminate
false drops
![Page 26: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/26.jpg)
E.G.M. Petrakis Visual Content 26
Texture
Repeative patterns of local variations of intensity
Structural: identify placement rules of structural primitives the less effective approach
Statistical: characterize spatial distribution of intensity in terms of measurementsHaralick, Tamura features etc.
![Page 27: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/27.jpg)
E.G.M. Petrakis Visual Content 27
Texture Examples
Ballard and Brown 84
![Page 28: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/28.jpg)
E.G.M. Petrakis Visual Content 28
Structural Texture
Ballard and Brown 84
![Page 29: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/29.jpg)
E.G.M. Petrakis Visual Content 29
Statistical Texture
Ballard and Brown 84
![Page 30: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/30.jpg)
E.G.M. Petrakis Visual Content 30
Haralick Features [Haralick 73]
Set of 4 features characterizing the intensity transitions of neighboring pixels in various directions using Gray-Tone Spatial-Dependence (GTSD)
arraysOne GTSD for each pixel neighborhood Neighborhood: pixels in direction θ and
distance d
![Page 31: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/31.jpg)
E.G.M. Petrakis Visual Content 31
GTSD Array Pd,θ[i,j]
Counts pixel pairs in distance d having gray levels i, j in direction θOne GTSD for θ=(00, 450,900, 1350) and d=(1,2,..)Intensity in range [0,k-1]: Pd,θ[i,j] is a k x k matrix
2 1 2 0 1
0 2 1 1 2
0 1 2 2 0
1 2 2 0 1
2 0 1 0 1
i
j
0 2 2
2 1 2
2 3 2
1
16
P[i,j]d = 1
0 1 2
0
1
2
![Page 32: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/32.jpg)
E.G.M. Petrakis Visual Content 32
Computing Pd,θ[i,j]
Count all pairs of pixels in which the first pixel has value i and its matching pair displaced by d=1 in θ = 450 or 1350 direction has value j
Enter this count in the (i,j) position of Pd,θ[i,j]E.g., there are 3 pairs [2,1], then P[2,1] = 3Pd,θ[i,j] is not symmetric: Pd,θ[i,j] < > Pd,θ[j,i]
Normalize Pd,θ[i,j] by the total number of pairs
Pd,θ[i,j]: probability mass function
![Page 33: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/33.jpg)
E.G.M. Petrakis Visual Content 33
Texture Features
1) Angular Second Moment (ASM):
Small values for non homogeneous regions
2) Contrast: Large values for many large transitions
or for many transitions
1
0
1
0
21 ),(
k
i
k
jjipf
1
0
1
0
22 )](,[
k
i
k
jjijipf
![Page 34: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/34.jpg)
E.G.M. Petrakis Visual Content 34
Texture Features (cont,d)
3) Correlation:
Frequency of intensity transitions
1
0y
1
0j
2y
1
0
1
1x
1
0
2x
1
0
1
0
1
03
],[ p ,][)( σ ,][
],[ p , ][)( σ ,][
],[
k
i
k
yy
k
j yy
k
j
k
i xx
k
i xx
yx
yx
k
i
k
j
jipjpjjjp
jipipiiip
jiijpf
![Page 35: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/35.jpg)
E.G.M. Petrakis Visual Content 35
Texture Features (cont,d)
4) Entropy:
High values for uniform p[i,j] i.e., no preferred gray-level (no texture)
A vector for each Tθ,d=(f1,f2,f3,f4) or A vector for every θ, d taking all Tθ,d in a
sequence Correlated features: apply K-L to de-
correlate and to reduce dimensionality
1
0
1
04 ],[log],[k
i
k
jjipjipf
![Page 36: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/36.jpg)
E.G.M. Petrakis Visual Content 36
Shape
Assume that objects are extracted Requires image segmentationDifficult problem
Criteria for reliable shape recognitionUniqueness of representationRobustness against noise and distortionProportionality of variationInvariance under scale, rotation and translationEfficiency of computationOcclusion: handle partially visible shapes
![Page 37: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/37.jpg)
E.G.M. Petrakis Visual Content 37
Shape Matching Methods
Two categories of methods based on: Regions: represent and match
properties of regionsContours: represent and match
properties of boundariesTechniques: local/global, model
based, fuzzy, statistical, neural networks
![Page 38: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/38.jpg)
E.G.M. Petrakis Visual Content 38
Input/Output
For any two shapes and compute:Their distanceThe correspondences between similar parts
Petrakis 02
![Page 39: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/39.jpg)
E.G.M. Petrakis Visual Content 39
Moment Invariants
An object is represented by its binary image
A set of 7 features can be defined based on central moments
00
01
00
10
),( m
my ,
m
mxyxm
Ryx
qppq
R
Ryx
qppq yyxx
),(
0,1,2...qp, ),)((
![Page 40: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/40.jpg)
E.G.M. Petrakis Visual Content 40
Central Moments [Hu 62]
Invariant to translation and rotation Use ηpq=μpq/μγ
00 where γ=(p+q)/2 + 1 for p+q=2,3… instead of μ’s in the above formulas to achieve scale invariance
![Page 41: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/41.jpg)
E.G.M. Petrakis Visual Content 41
More Shape Methods Moments can also be defined on the closed
bounding contours of objects [Gupta and Shinath 87]
Moments can also be defined for open curves [Koch and Kashyap 89]
Methods based on the Fourier Transform of the bounding contour have also been used [Wallace and Wintz 80, Rauber and Steiger 92]
More efficient methods has also been proposed [Petrakis, Diplaros and Milios 2002]. Examines many of the above methods based on Fourier and Moments and shows many experiments and comparisons
![Page 42: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/42.jpg)
E.G.M. Petrakis Visual Content 42
Spatial Relationships
Find images showing similar objects in similar spatial relationships find X-rays similar to Smith’s examinationfind images showing a tree close to a houseone of the two images may contain extra objects
Q I Petrakis02
![Page 43: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/43.jpg)
E.G.M. Petrakis Visual Content 43
Methods
Two main categories of methods Spatial projections (2D strings and
variants like 2D C strings, Expanded 2D strings etc).
Attributed Relational Graphs (ARGs)Image distance is defined
accordinglyEditing distance on ARGs2D string matching
![Page 44: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/44.jpg)
E.G.M. Petrakis Visual Content 44
Image Segmentation
All methods assume segmented images image are segmented manually or semi-
manuallyimage segmentation is a difficult problem
Petrakis02
![Page 45: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/45.jpg)
E.G.M. Petrakis Visual Content 45
Image Features
Individual objects: 5-dimensional vectorsSize: number of pixels in a regionPerimeter: length of bounding contourRoundness: ratio of smallest/largest second
momentorientation: angle with x direction (sin,cos)
Spatial Relationships: 4-dimensional vectorsPosition: inside or outsideDistance: minimum distance of contoursOrientation: angle with x (sin,cos) of c.g.’s
![Page 46: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/46.jpg)
E.G.M. Petrakis Visual Content 46
Attributes Relational Graphs (ARGs)
Objects are represented by nodes
Relationships are represented by arcs
Nodes and arcs are labeled by feature vectors
Matching: ARG editing distance, Hungarian [Petrakis 02]
Petrakis02
![Page 47: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/47.jpg)
E.G.M. Petrakis Visual Content 47
ARG Editing Distance
Matching: sequence of edit operations that transform a query Q to an image IEdit operation: node or arc insertion,
deletion or substitution
F combines the costs of edit operationsf is the cost of an edit operation defined
as a vector distance
)(),...(),(min
))((min),(
11)(
)'(
fffF
ISFIQD
kkIS
GS
![Page 48: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/48.jpg)
E.G.M. Petrakis Visual Content 48
Matching Algorithm [Messmer95]
Find the sequence of edit operations that yield the minimum total cost
Formulated as tree search problemExpand all possible matching sequencesBranch and bound Tree node: matching of ARG nodeTree arc: matching of ARG edgesSubtree: matching of subgraphs of Q, I
![Page 49: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/49.jpg)
E.G.M. Petrakis Visual Content 49
Query Q Model I
![Page 50: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/50.jpg)
E.G.M. Petrakis Visual Content 50
Hungarian Method [Petrakis 02]
Matching: assignment problem
The relationships are ignored
F: cost of a mappingC(i,F(i)): vector
distance
n
iFF iFiCIQD1
))(,(min),(
![Page 51: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/51.jpg)
E.G.M. Petrakis Visual Content 51
2D String [Chang 87]
2D string: projections of c.g.’s along x and y Each object is represented by a name or class
Matching: string matching (type 0,1 and 2)
![Page 52: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/52.jpg)
E.G.M. Petrakis Visual Content 52
Discussion [Petrakis 02] The ARG editing distance is the most
accurate method followed by Hungarian and 2D strings
2D strings is the faster method followed by Hungarian and ARG distance
Speed and Accuracy are traded-off: the most accurate a method the slower it is
Indexing: Petrakis 2002, Petrakis & Faloutsos 97 (ARGs), Petrakis 93 (2D strings)
http://www.ced.tuc.gr/~petrakis/publications/publications.htm
![Page 53: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/53.jpg)
E.G.M. Petrakis Visual Content 53
Image Segmentation
All methods assumed segmented imagesSegmentation is the process of partitioning
an image into groups of connected pixels (regions) with similar properties Gray levelsColors TexturesMotion characteristics (motion vectors)Edge continuity
![Page 54: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/54.jpg)
E.G.M. Petrakis Visual Content 54
Segmentation Methods
Two approaches Region segmentationEdge segmentation
Regions may correspond to objectsNot always perfect (noise, bad
illumination, 3D world etc.)Further reading: "Machine Vision'', R.
Jain, R. Kasturi, B. G. Schunck, Mc Graw-Hill, 1995
![Page 55: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/55.jpg)
E.G.M. Petrakis Visual Content 55
Region Segmentation
Converts a gray-level image into a binary one by applying carefully selected thresholds on intensity histograms
The image is partitioned into two setsBlack pixels: objects White pixels: background
![Page 56: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/56.jpg)
E.G.M. Petrakis Visual Content 56
Histogram Thresholding
The threshold distinguishes the objects from the background
The objects have similar gray-level values
![Page 57: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/57.jpg)
E.G.M. Petrakis Visual Content 57
Thresholding
Find thresholds automatically by analyzing the gray value distribution (histogram) of the image
Objects are dark against a light backgroundTheir gray-value distributions can be separated
putting thresholds between them
Automatic thresholding is based on peackiness and valleyness measurements at each point of the histogram
![Page 58: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/58.jpg)
E.G.M. Petrakis Visual Content 58
Further Reading C. Faloutsos et.al. “Efficient and Effective Querying by Image Content
”, Journal of Intelligent Information Systems, Vol. 3, No. ¾, pp. 231-262, 1994
M. Flicknet et.al. “Query by Image and Video Content: the QBIC Systems”, IEEE Computer, Vol. 28, No. 9, pp. 13-32, Sept. 1995
R.C.Veltkamp and M.Tanase “Content-Based Image Retrieval Systems: A Survey”, TR UU-CS-2000-34, Utrecht University, March 2001
A.W.M.Smeulders et.al., “Content Based Image Retrieval at the End of the Early years”, IEEE Transactions on PAMI, 22(12): 1349-1380, 2000
R.Schettini, et al. “A Survey on Methods for Colour Image Indexing and Retrieval in Image Databases” , in: R.Luo and L.MacDonald (Eds.), Color Imaging Science: Exploiting Digital Media, John Wiley, 2001
M. Swain, D.H.Ballard, “Color Indexing”, Intern. Journ. of Comp. Vision, Vol. 7, No. 1, pp. 11-32, 1991
D. Androutsos et.al. “A Novel Vector-Based Approach to Color Image Retrieval Using a Vector Angular-Based Distance Measure”, Comp. Vision and Image Understanding, Vol. 75, No. ½, July/Aug. 1999, pp. 46-58.
![Page 59: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/59.jpg)
E.G.M. Petrakis Visual Content 59
References J.R.Smith, S-F.Chang, “Tools and Techniques for Color Image Retrieval”,
IS&T/SPIE Proc., Vol. 2670, Storage and Retrieval for Image and Video Databases IV
R.M. Haralick, K. Shanmungam, I. Dinstein “Textural Features for Image Classification”, IEEE Trans. on Systems Man and Cybernetics, 1973, pp. 610-621.
E.G.M. Petrakis, A. Diplaros and E. Milios: "Matching and Retrieval of Distorted and Occluded Shapes using Dynamic Programming", IEEE Trans. on PAMI, Vol. 24, No. 11, Nov. 2002, pp. 1-16.
M.-K. Hu. Visual Pattern Recogn. by Moment Invariants. IRE Trans. on Info. Theory, IT-8:179–187, 1962.
T. P. Wallace and P. A. Wintz. An Efficient Three-Dimensional Aircraft Recognition Algorithm Using Normalized Fourier Descriptors. Computer Graphics and Image Processing, 13:99–126, 1980.
T.W. Rauber and A.S. Steiger-Carcao, “Shape Description by UNL Fourier Features – An Application to Handwritten Character Recognition, 11th IAPR Intern. Conf. on Pattern Recogn., 30.Aug.-3.Sept. 1992, The Hague, The Netherlands (click here for implementation).
![Page 60: E.G.M. PetrakisVisual Content1 Retrieval of Visual Content Images comprise the vast majority of data in many application domains Remote sensing (NASA,](https://reader031.vdocuments.net/reader031/viewer/2022032310/56649d7f5503460f94a62c02/html5/thumbnails/60.jpg)
E.G.M. Petrakis Visual Content 60
References L. Gupta and M.D. Shrinath, “Contour Sequence Moments for the
Classification of Closed Planar Shapes”, Pattern Recognition, Vol. 20, No. 3, pp. 267-272, 1987
M.W.Koch and R.L.Kashyap, “Matching Polygon Fragments”, Pattern Recognition Letters, No. 10, pp. 297-308,1989.
Euripides G.M. Petrakis, Aristeidis Diplaros and Evangelos Milios: "Matching and Retrieval of Distorted and Occluded Shapes using Dynamic Programming", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No. 11, November 2002, pp. 1-16.
Euripides G.M. Petrakis, Aristeidis Diplaros and Evangelos Milios: "Matching and Retrieval of Distorted and Occluded Shapes using Dynamic Programming", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No. 11, November 2002, pp. 1-16.
Euripides G.M. Petrakis: "Fast Retrieval by Spatial Structure in Image DataBases", Journal of Visual Languages and Computing, Vol. 13, No. 5, October 2002, pp. 545-569.
Euripides G.M. Petrakis: "Design and Evaluation of Spatial Similarity Approaches for Image Retrieval", Image and Vision Computing, January 2002, Number 1, Volume 20, pp. 59-76.
Euripides G.M. Petrakis and Christos Faloutsos: "Similarity Searching in Medical Image Databases", IEEE Transactions on Knowledge and Data Engineering, Vol. 9, No. 3, pp. 435-447, May/June 1997.