large scale tag recommendation using different image representations
TRANSCRIPT
![Page 1: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/1.jpg)
Web Science & Technologies
University of Koblenz ▪ Landau, Germany
Large Scale Tag Recommendation
Using Different Image Representations
Rabeeh Abbasi
Marcin Grzegorzek
Steffen Staab
![Page 2: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/2.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria2 of 20
Tag Recommendation
http://www.flickr.com/photos/64255998@N00/283890351/
ClocktowerUhrturmGrazAustriaEuropeHerbstAutumnNightNacht
47° 4′ 32″ N 15° 26′ 12″ E
ImageImage TagsTags
![Page 3: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/3.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria3 of 20
Tag Recommendation
http://www.flickr.com/photos/64255998@N00/283890351/
Tag Recommender
ClocktowerUhrturmGrazAustriaEuropeHerbstAutumnNightNacht
InputInput OutputOutput
47° 4′ 32″ N 15° 26′ 12″ E
![Page 4: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/4.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria4 of 20
Tag Recommendation
TagsGeographical Coordinates
Low-level Features
Tag Recommender
ClocktowerUhrturmGrazAustriaEuropeHerbstAutumnNightNacht
InputInput OutputOutput
47° 4′ 32″ N 15° 26′ 12″ E
![Page 5: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/5.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria5 of 20
Tag Recommendation
TagsGeographical Coordinates
Low-level Features
Which image features to use?
Tag Recommender
ClocktowerUhrturmGrazAustriaEuropeHerbstAutumnNightNacht
InputInput OutputOutput
47° 4′ 32″ N 15° 26′ 12″ E
![Page 6: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/6.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria6 of 20
Features Available in Large Datasets
CoPhIR Dataset contains106 million images from Flickr with
Title, location, tag, comment, etcFive standard MPEG-7 Features
![Page 7: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/7.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria7 of 20
Tag Recommender – System Overview
TrainingData
Location 1
Location N
Location 1Clusters
Location NClusters
Model
Features
Features
![Page 8: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/8.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria8 of 20
Tag Recommender
Tag Recommender – System Overview
TrainingData
Location 1
Location N
Location 1Clusters
Location NClusters
Model
Features
Features
Image Features
Tags
ClocktowerGrazAustriaEuropeAutumnNightUhrturmNachtHerbstHDRPhotomatix
InputInput OutputOutput
47° 4′ 32″ N 15° 26′ 12″ E
![Page 9: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/9.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria9 of 20
Tag Recommender – Clustering
Used Standard K-Means clustering
![Page 10: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/10.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria10 of 20
Tag Recommender – Representative Tags
Given a cluster containing homogeneous imagesFind most representative tags
Tag Rank ~ Frequency of Users
Clocktower (4)Graz (4)Austria (3)Europe (3)Autumn (2)Night (2)Uhrturm (1)Nacht (1)Herbst (1)HDR (1)Photomatix (1)
Cluster Tags
User 2 User 4
User 1 User 3
![Page 11: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/11.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria11 of 20
Tag Recommender – Classification
Map a new image to one of the existing clustersFind the nearest cluster centroid
Assign representative tags of the mapped cluster to the image
RepresentativeTags
NearestCentroid
![Page 12: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/12.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria12 of 20
Evaluation
Evaluated on tags of ~400,000 Flickr images75% training data25% test data / gold standard
Data is randomly split based on users (not resources)Ignored
10 most frequent tags for each location (city) and Tags used by less than three users
Evaluation based on the correctly identified tags of test dataPrecisionRecallF-Measure
![Page 13: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/13.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria13 of 20
Micro Precision
1 2 3 4 5 10 15 20
0.0000
0.0200
0.0400
0.0600
0.0800
0.1000
0.1200
0.1400
0.1600
0.1385
0.1008
0.0867
0.0783
0.0686
0.0478
0.03820.0331
0.05020.0441
0.0412 0.0388 0.03640.0302
0.02640.0236
0.0451
0.03820.0326
0.0295 0.0274
0.02030.0170 0.0150
0.03380.0302 0.0284 0.0282 0.0275
0.0237 0.0211 0.0193
Geo EHD Tags Random
Number of tags recommended
Pre
cisi
on
Geo CoordinatesLow-level FeaturesTagsBaseline (Random)
![Page 14: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/14.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria14 of 20
Micro Recall
1 2 3 4 5 10 15 20
0.0000
0.0500
0.1000
0.1500
0.2000
0.2500
0.0515
0.0749
0.0966
0.1163
0.1273
0.1749
0.2071
0.2357
0.0187
0.0328
0.0460
0.0577
0.0677
0.1120
0.1469
0.1748
0.0168
0.02790.0351
0.04170.0478
0.0668
0.07950.0882
0.01260.0225
0.03170.0419
0.0511
0.0880
0.1174
0.1432
Geo EHD Tags Random
Number of tags recommended
Re
call
![Page 15: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/15.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria15 of 20
Micro F-Measure
1 2 3 4 5 10 15 20
0
0.02
0.04
0.06
0.08
0.1
0.0751
0.0859
0.09140.0936
0.0892
0.0750
0.0645
0.0581
0.0272
0.0376
0.04350.0464 0.0474 0.0475
0.04480.0417
0.0244
0.0323 0.0338 0.0346 0.03490.0311
0.02810.0256
0.0183
0.02580.0300
0.03370.0357 0.0373 0.0358 0.0341
Geo EHD Tags Random
Number of tags recommended
F-M
eas
ure
![Page 16: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/16.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria16 of 20
Coverage
Number of different tags correctly suggested
800
1000
1200
1400
1600
1800
2000
Geo EHD CL Tag Random
Nu
mb
er
of ta
gs c
orr
ectly
su
gg
este
d
![Page 17: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/17.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria17 of 20
Edge Histogram vs. Color Layout
1 2 3 4 5 10 15 20
0.03
0.03
0.04
0.04
0.05
0.05
0.0272
0.0376
0.0435
0.0464
0.0474 0.0475
0.0448
0.0417
0.0280
0.0390
0.0434
0.04540.0459
0.0466
0.0435
0.0408
EHD CL
Number of tags recommended
F-M
eas
ure
![Page 18: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/18.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria18 of 20
Cosine vs. Euclidean
1 2 3 4 5 10 15 20
0.02
0.03
0.03
0.04
0.04
0.05
0.05
0.0451
0.0416
0.0382
0.03680.0360
0.0311
0.0281
0.0256
0.0311 0.03080.0302
0.0293 0.0291
0.0277
0.0261
0.0249
Tags (Cos) Tags (Eucl)
Number of tags recommended
F-M
eas
ure
![Page 19: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/19.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria19 of 20
Euclidean vs. Manhattan
1 2 3 4 5 10 15 20
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1
Geo (Eucl) EHD (Eucl) Random (Eucl)Geo (Manh) EHD (Manh) Random (Manh)
Number of tags recommended
F-M
eas
ure
![Page 20: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/20.jpg)
Web Science & Technologies Rabeeh [email protected]
SAMT'09, Graz, Austria20 of 20
Conclusions
New images can be automatically taggedGeographical coordinates give best results
Require less computational power (just 2 dimensions)No low-level image feature analysis requiredLimited to location based tag recommendations only!
Future WorkOptimize individual modules of the system
Clustering, representative tags, classification etc.Combination of multiple image features
![Page 21: Large Scale Tag Recommendation Using Different Image Representations](https://reader030.vdocuments.net/reader030/viewer/2022032419/55a34d2b1a28ab496e8b47ad/html5/thumbnails/21.jpg)
Thank You
☺