harmony voc 2010 - university of oxfordhost.robots.ox.ac.uk/pascal/voc/voc2010/workshop/cvc.pdf ·...
TRANSCRIPT
![Page 1: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/1.jpg)
Harmony Poten,als: Fusing Global and Local Scale for Seman,c Image Segmenta,on
J. M. Gonfaus X. Boix
F. S. Khan J. van de Weijer
A. Bagdanov M. Pedersoli
J. Serrat X. Roca
J. Gonzàlez
![Page 2: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/2.jpg)
Mo,va,on (I)
• Why combine global and local scale?
![Page 3: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/3.jpg)
Mo,va,on (I)
• Why combine global and local scale?
![Page 4: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/4.jpg)
Mo,va,on (I) • Classifica,on is oPen impossible based on local appearance only.
Image Classifier
0 0,5 1
Chair
Plant
Sofa
Bus
Aeroplane
Context is a powerful and dis,nc,ve cue
![Page 5: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/5.jpg)
Mo,va,on (II)
• How can we improve local classifiers?
0 0,5 1
Dog
Cat
Horse
Cow
Aeroplane
Is this the foreground or the background of…
0 0,5 1
Dog
Cat
Horse
Cow
Aeroplane
Is this object X or some other object
Why not combine them? Bad class discrimina,on
Good figure segmenta,on Inaccurate segmenta,on
Good class discrimina,on
![Page 6: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/6.jpg)
Mo,va,on (II)
• How can we improve local classifiers?
0 0,5 1
Dog
Cat
Horse
Cow
Aeroplane
Is this the foreground or the background of…
0 0,5 1
Dog
Cat
Horse
Cow
Aeroplane
Is this object X or some other object
Why not combine them? Bad class discrimina,on
Good figure segmenta,on Inaccurate segmenta,on
Good class discrimina,on
![Page 7: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/7.jpg)
Mo,va,on (II)
• How can we improve local classifiers? – More informa,on sources • Mid-‐level informa,on through object detectors
![Page 8: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/8.jpg)
Outline
• Overview of our method • How to fuse local and global scale – Harmony Poten,als* – CVC_Harmony submission (35.4% on test)
• Improving local classifiers – CVC_Harmony+Det submission (40.1% on test)
• Results • Conclusions
*J.M. Gonfaus, X. Boix, J. Van de Weijer, A. D. Bagdanov, J. Serrat, J. Gonzàlez “Harmony Poten,als for Joint Classifica,on and Segmenta,on”, in CVPR 2010
![Page 9: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/9.jpg)
Overview of our method
![Page 10: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/10.jpg)
Overview of our method
• Unsupervised segmenta,on. – Around 500 superpixels/image
![Page 11: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/11.jpg)
Overview of our method
• Unsupervised segmenta,on.
• Superpixel nodes
– Unary poten,al • BoW inside AND neighborhood
– Smoothness poten,al
• Pairwise Pois poten,al – BoW
• SIFT, RGB Histogram, SSIM • Mul,scale: 12, 24, 36, 48 square patches • Step size 50% of the patch
• Quan,zed to 1000, 400, 300 words • Learned on SVM with 8000 samples +
retraining
(CVC_Harmony)
![Page 12: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/12.jpg)
Overview of our method
• Unsupervised segmenta,on.
• Superpixel nodes
– Unary poten,al • BoW inside AND neighborhood
• Detec,on scores • Loca,on prior
– Smoothness poten,al
• Pairwise Pois poten,al – BoW
• SIFT, RGB Histogram, SSIM • Mul,scale: 12, 24, 36, 48 square patches • Step size 50% of the patch
• Quan,zed to 1000, 400, 300 words • Learned on SVM with 8000 samples +
retraining
(CVC_Harmony+det)
![Page 13: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/13.jpg)
Overview of our method
• Unsupervised segmenta,on.
• Superpixel nodes
• Global Node – Unary poten,al:
• Global classifier method • CVC_flat submission:
mAP: 61% for classifica,on task
– Consistency poten,al • From global node to each sp
• Harmony Poten,al
![Page 14: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/14.jpg)
Model
Unary Poten,al Smoothness Poten,al Consistency Poten,al
![Page 15: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/15.jpg)
Model
Consistency Poten,al
![Page 16: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/16.jpg)
Consistency poten,al
• Ground-‐Truth
• Unary Poten,als
• Pois-‐based Poten,als
• Robust PN Poten,als • Harmony Poten,als
![Page 17: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/17.jpg)
Consistency poten,al
• Ground-‐Truth
• Unary Poten,als
• Pois-‐based Poten,als
• Robust PN Poten,als • Harmony Poten,als
GT
![Page 18: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/18.jpg)
Consistency poten,al
• Ground-‐Truth
• Unary Poten,als
• Pois-‐based Poten,als
• Robust PN Poten,als • Harmony Poten,als
GT
![Page 19: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/19.jpg)
Consistency poten,al
Free • Ground-‐Truth
• Unary Poten,als
• Pois-‐based Poten,als
• Robust PN Poten,als • Harmony Poten,als
GT
![Page 20: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/20.jpg)
Consistency poten,al
• Ground-‐Truth
• Unary Poten,als
• Pois-‐based Poten,als
• Robust PN Poten,als • Harmony Poten,als
GT
![Page 21: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/21.jpg)
Consistency poten,al
=
All possible label combina,ons is unfeasible
![Page 22: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/22.jpg)
Consistency poten,al
Prior From the training data we extract the co-‐occurrence sta,s,cs of labels
Likelihood Image classifica,on scores each combina,on
• Ranked subsampling of Few best combina,ons are required to saturate the performance
![Page 23: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/23.jpg)
Model
Unary Poten,al
![Page 24: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/24.jpg)
Unary poten,al
• Local classifiers are weak classifiers – Too ambiguous because liile informa,on is used
• Combining mul,ple classifiers makes our local unary poten,al stronger.
• Features: – foreground/background – class versus others – object detec,ons – spa,al loca,on prior
![Page 25: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/25.jpg)
Ffg-‐bg: Fore-‐Background
• Easy to iden,fy whether the superpixel belongs to the object class or to its common background
![Page 26: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/26.jpg)
Ffg-‐bg: Fore-‐Background
• Easy to iden,fy whether the superpixel belongs to the object class or to its common background
![Page 27: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/27.jpg)
Ffg-‐bg: Fore-‐Background
• Easy to iden,fy whether the superpixel belongs to the object class or to its common background
![Page 28: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/28.jpg)
Fclass: Class vs. other classes
• Learning how different an object is from its common background becomes difficult for certain class combina,ons
Foreground Background
![Page 29: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/29.jpg)
Fclass: Class vs. other classes
• Learning how different an object is from its common background becomes difficult for certain class combina,ons
![Page 30: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/30.jpg)
Fposi,on: Loca,on prior
• Objects tend to appear in class-‐specific, par,cular loca,ons (and not at the borders)
![Page 31: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/31.jpg)
Fdet: Object detector* scores
• Mid-‐level informa,on is added by considering object detec,ons [Felzenszwalb et al. 2010].
• Average over superpixel area with maximum detec,on score at each pixel.
• Scores = [-‐1 , ∞)
• Class specific “No detec,on” score is learned. • Keeps the CRF and the model simple.
*Felzenszwalb, Girshick, McAllester, Ramanan, “Object Detec,on with Discriminately Trained Part based models”, PAMI 2010
![Page 32: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/32.jpg)
Fdet: Object detector* scores
![Page 33: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/33.jpg)
Results on valida,on set 2010
33,2
23,4
20
26
34,5
36,6
40,1
15 20 25 30 35 40
Fg_Bk
Class
Loc
Det
Fg_Bk + Loc
Fg_Bk + Class
All
Mean Average Precision
2009 submission
![Page 34: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/34.jpg)
Combina,on of features
• Naïve Bayes approach • Specific sigmoid per class and per classifier
• Total number of parameters to be learned:
2x20x4 + 20 + 4 + 1 = 185 parameters
• All parameters are jointly op,mized by stochas,c steepest ascent
€
φ(xi) =1
1+ exp(−a f xif + b f )f ∈F
∏
feature sigmoids no_detec,on score
CRF weights
background probability
![Page 35: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/35.jpg)
Results on valida,on set 2010
33,2
23,4
20
26
34,5
36,6
39,2
15 20 25 30 35 40
Fg_Bk
Class
Loc
Det
Fg_Bk + Loc
Fg_Bk + Class
All
Mean Average Precision
CVC_Harmony 2010 submission 35,4 on test
2009 submission
CVC_Harmony_Det 2010 submission 40,1 on test
![Page 36: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/36.jpg)
Illustra,ve examples
Fg/bg class
*
*
*
det
*
*
*
loc
*
*
*
=
=
=
=
=
=
=
=
=
final unary
![Page 37: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/37.jpg)
Illustra,ve examples
Fg/bg class
*
*
*
det
*
*
*
loc
*
*
*
=
=
=
=
=
=
=
=
=
final unary
![Page 38: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/38.jpg)
Final results
![Page 39: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/39.jpg)
Conclusions
• Harmony poten,al is an effec,ve way to fuse global and local scales for seman,c image segmenta,on.
• We have focused on improving the local classifiers
• Baseline: 29% + combining fg/bg and mul,class classifiers (+2%) + object detec,on (+3%) + loca,on prior (+1%) + per class parameter op,miza,on (+5%)
more details: hip://iselab.cvc.uab.es/pvoc2010
![Page 40: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/40.jpg)
Thanks for your aien,on!
Gràcies per la vostra atenció!
Ευχαριστω για την προσοχη σας
![Page 41: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/41.jpg)
Full Prac,cal Example
![Page 42: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/42.jpg)
Ffgbg: Fore-‐Back ground
![Page 43: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/43.jpg)
Fclass: Class against other classes
![Page 44: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/44.jpg)
Close-‐up comparison
Fore-‐Back ground learning
Class against others learning
![Page 45: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/45.jpg)
Ffgbg * Fclass
![Page 46: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/46.jpg)
Fdet: Detector Scores
![Page 47: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/47.jpg)
Ffgbg*Fclass*Fdet
![Page 48: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/48.jpg)
Floca,on: Loca,on Prior
![Page 49: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/49.jpg)
Ffgbg*Fclass*Fdet*Floc
![Page 50: harmony voc 2010 - University of Oxfordhost.robots.ox.ac.uk/pascal/VOC/voc2010/workshop/cvc.pdf · Harmony(Poten,als:(Fusing(Global(and(Local(Scale(for(Seman,c(Image(Segmenta,on(J.(M.(Gonfaus((X.(Boix((F.(S.(Khan(J.(van(de(Weijer(A.(Bagdanov(](https://reader034.vdocuments.net/reader034/viewer/2022042312/5eda254fb3745412b570d877/html5/thumbnails/50.jpg)
Result