CVPR 2013 Diversity Tutorial

Closing Remarks:What can we do with multiple

diverse solutions?

Dhruv Batra Virginia Tech

CVPR 2013 Diversity Tutorial

(C) Dhruv Batra 2

Example Result

Now what?

CVPR 2013 Diversity Tutorial

Your Options• Nothing

– User in the loop

• (Approximate) Min Bayes Risk– Use solutions to estimate the distribution and optimize

Bayes Risk

• Re-ranking– Pick a good solution from the list

(C) Dhruv Batra 3

Increasing Side Information

CVPR 2013 Diversity Tutorial

Interactive Segmentation• Setup

– Model: Color/Texture + Potts Grid CRF– Inference: Graph-cuts– Dataset: 50 train/val/test images

(C) Dhruv Batra 4

Image + Scribbles Diverse 2nd Best2nd Best MAPMAP

1-2 Nodes Flipped 100-500 Nodes Flipped

CVPR 2013 Diversity Tutorial

Interactive Segmentation

(C) Dhruv Batra 5

MAP M-Best-MAP Confidence DivMBest89%

90%

91%

92%

93%

94%

95%

96%

+0.05%

+1.61%

+3.62%

(Oracle) (Oracle) (Oracle)

M=6

Seg

men

tatio

n A

ccur

acy

Better

CVPR 2013 Diversity Tutorial

Your Options• Nothing

– User in the loop

• (Approximate) Min Bayes Risk– Use solutions to estimate the distribution and optimize

Bayes Risk

• Re-ranking– Pick a good solution from the list

(C) Dhruv Batra 6

CVPR 2013 Diversity Tutorial

Statistics 101• Loss

– PCP, Pascal Loss, etc

• “True” Distribution

• Expected Loss:

• Min Bayes Risk

(C) Dhruv Batra 7

CVPR 2013 Diversity Tutorial

Structured Output Problems• Min Bayes Risk

• Two Problems

• Approximate MBR:

(C) Dhruv Batra 8

IntractableIntractable

CVPR 2013 Diversity Tutorial

Semantic Segmentation• Setup

– Models: • Hierarchical CRF [Ladicky et al. ECCV ’10, ICCV ‘09]

• Second-Order Pooling [Carreira ECCV ‘12]

– Inference: • Alpha-expansion• Greedy

– Dataset: Pascal Segmentation Challenge (VOC 2012) • 20 categories + background; ~1500 train/val/test images

(C) Dhruv Batra 9

CVPR 2013 Diversity Tutorial

(C) Dhruv Batra 10

Large-Margin Re-ranking

CVPR 2013 Diversity Tutorial

Semantic Segmentation

(C) Dhruv Batra 11

Input MAP Best of 10-Div

CVPR 2013 Diversity Tutorial

Semantic Segmentation

(C) Dhruv Batra 12

PAC

AL

Acc

urac

y

Better

#Solutions / Image

1 2 3 4 5 6 7 8 9 1044%

47%

50%

53%

56%

59%

MAP[State-of-art circa 2012]

15%-gain possible

Same FeaturesSame Model

DivMBest (Oracle)

Rand (Re-rank)

MBR

CVPR 2013 Diversity Tutorial

Your Options• Nothing

– User in the loop

• (Approximate) Min Bayes Risk– Use solutions to estimate the distribution and optimize

Bayes Risk

• Re-ranking– Pick a good solution from the list

(C) Dhruv Batra 13

CVPR 2013 Diversity Tutorial

(C) Dhruv Batra 14

Large-Margin Re-ranking

CVPR 2013 Diversity Tutorial

(C) Dhruv Batra 15

Large-Margin Re-ranking

CVPR 2013 Diversity Tutorial

(C) Dhruv Batra 16

Large-Margin Re-ranking

CVPR 2013 Diversity Tutorial

(C) Dhruv Batra 17

Large-Margin Re-ranking

Discriminative Re-ranking of Diverse Segmentation

[Yadollahpour et al., CVPR13, Wednesday Poster]

CVPR 2013 Diversity Tutorial

Semantic Segmentation

(C) Dhruv Batra 18

PAC

AL

Acc

urac

y

Better

#Solutions / Image

1 2 3 4 5 6 7 8 9 1044%

47%

50%

53%

56%

59%

MAP[State-of-art circa 2012]

DivMBest (Oracle)

Rand (Re-rank)

DivMBest (Re-ranked) [Y.B.S., CVPR ‘13]

MBR

CVPR 2013 Diversity Tutorial

Qualitative Results: Success

(C) Dhruv Batra 19

CVPR 2013 Diversity Tutorial

Qualitative Results: Success

(C) Dhruv Batra 20

CVPR 2013 Diversity Tutorial

Qualitative Results: Success

(C) Dhruv Batra 21

CVPR 2013 Diversity Tutorial

Qualitative Results: Failures

(C) Dhruv Batra 22

CVPR 2013 Diversity Tutorial

Qualitative Results: Failures

(C) Dhruv Batra 23

CVPR 2013 Diversity Tutorial

Qualitative Results: Failures

(C) Dhruv Batra 24

CVPR 2013 Diversity Tutorial

Summary• All models are wrong

• Some beliefs are useful

• Diverse Multiple Solutions– A way to get useful beliefs out.

• DivMBest + Reranking– Big impact possible on many applications!

(C) Dhruv Batra 25

CVPR 2013 Diversity Tutorial

Summary

• What does my model believe?

(C) Dhruv Batra 26

Posterior Summary

CVPR 2013 Diversity Tutorial

Thanks!

(C) Dhruv Batra 27