Attributes for Classifier Feedback

Amar Parkash and Devi Parikh

You can teach a child by examples...

And on, and on, and on…

Is this a giraffe? No.

Is this a giraffe? Yes.

Is this a giraffe? No.

And on, and on, and on…

Proposed Active Learning Scenario

I think this is a giraffe. What do you think?

No, its neck is too short for it to be a giraffe.

Ah! These must not be giraffes

either then.

[Animals with even shorter necks]

……

Current belief Focused feedbackKnowledge of the world

Feedback on one, transferred to many

• Learner learns better from its mistakes• Accelerated discriminative learning with few examples

Communication

Need a language that is• Machine understandable• Human understandable

Attributes!• Mid-level shareable• Visual • Semantic

Proposed Active Learning

Concepts to teach:

C1 C2 CK…

Unlabeled pool of images

Classifiers:

h1 h2 hK…

[Label-feedback][Attributes-based feedback]

Attribute predictors:

a1 a2 aM…

[Predicted Label]

Any feature spaceAny discriminative learning algorithm

Relative Attributes[Parikh and Grauman, ICCV 2011]

Openness

Unlabeled pool of images

Attribute predictors:

a1 a2 aM…

Image featuresParameters

Attributes-based Feedback

Unlabeled pool of images

Unlabeled pool of images

No,It is too open to be a forest

Attribute predictors:

a1 a2 aM…

Forest

Openness

Not Forest

Attributes-based Feedback

Unlabeled pool of images

Unlabeled pool of images

No,It is too open to be a forest

Attribute predictors:

a1 a2 aM…

Forest

Classifiers:

h1 h2 hK…

Not Forest

Proposed Active Learning

Concepts to teach:

C1 C2 CK…

Unlabeled pool of images

Classifiers:

h1 h2 hK…

[Label-feedback][Attributes-based feedback]

Attribute predictors:

a1 a2 aM…

[Predicted Label]

Label-based Feedback

• Not our contribution

• Experiment with different scenarios

• Benefits of attributes-based feedback– Small when label-based is very informative– Large when label-based feedback is weak

Unlabeled pool of images

Forest

Label-based Feedback

• Accept: Yes, this is a forest.– Strong: It is not anything else

Example: Classification– Weak: It can be other things

Example: Annotation

Label-based Feedback

• Reject: – Strong: No, it is a coast.

Example: Classification with few classes– Weak: No, this is not a forest.

Example: Large-scale classificationExample: Biased binary classification

• 4 different scenarios in experiments

Unlabeled pool of images

Forest

Datasets

• Datasets and relative attribute predictors from [Parikh and Grauman, ICCV 2011]

Datasets

• Faces: – 8 celebrity categories– 11 attributes (chubby, white, etc.)

[Kumar et al., ICCV 2009]

Datasets

• Scenes:– 8 categories– 6 attributes (open, natural, etc.)

[Oliva and Torralba, IJCV 2001]

Settings

• Feedback from MTurk• Features:– Raw image features (gist, color)– Attribute scores

• Category classifiers: SVM with RBF kernel• Results on– 2 datasets x 2 features x 4 label-feedback scenarios– Show 2 here, rest in paper.

This image doesn’t have enough perspective to be a street scene.

0 10 20 30 40 5025

35

45

55

65

75

Baseline

Proposed (Human)

Proposed (Oracle)

Results

Faces, Attribute features, Strong label-feedback

# iterations

Accu

racy

0 10 20 30 400

10

20

30

40

Baseline

Proposed (Human)

Proposed (Or-acle)

Results

Scenes, Image features, Weak label-feedback

# iterations

Accu

racy

1/4th !

More results in the paper.

Conclusion

• Attributes for providing classifier feedback• Novel learning paradigm with enhanced

human-machine communication• Discriminative learning + domain knowledge• Learning with few examples

• Connections to semi-supervised learning– Shrivastava, Singh and Gupta: Up Next!

Thank you!

Backup Slides

Negative Feedback

• Many reasons come together to make a concept– Hard to describe why an image is a concept

• One reason can break a concept– Easier to describe why an image is not a concept

• (Arguably) waste to give feedback when right

Discriminative– No domain knowledge

is conveyed– Many training images– Discriminative model– Classification in any

feature space– State-of-art

performance

Related WorkZero-shot learning– Convey domain

knowledge – Zero training images– Generative model– Classification in

attribute space– Performance

compromised

Proposed paradigm– Convey domain

knowledge to transfer– Few training images– Discriminative model– Classification in any

feature space– Performance

maintained

bear

turtle rabbit

furry

big

[Lampert et al., CVPR 2009]

C

Smili

ng

Age

S

J H

C is younger than H C smiles more than H

MM is younger than J

M smiles more than J

[Parikh and Grauman, ICCV 2011]++

+–

– –

Related Work

Focused discrimination– Mining of hard negatives [Felzenszwalb 2010]– To understand classifier [Golland 2001]– Here, supervisor provides the discriminative

direction by verbalizing semantic knowledge

[Golland, NIPS 2001][Felzenszwalb et al., CVPR 2008]

Related Work

What we are not doing:• Collecting deeper annotations of images

• Segmentation masks [Russell 2008], Parts [Farhadi 2010], Pose [Bourdev 2009], Attributes [Kumar 2009]• We use attributes for broad propagation of category

labels to unlabeled images

Related Work

What we are not doing:• Actively interleaving attribute annotations

• Object & attributes [Kovashka 2011], Image, boxes & segments [Vijaynarsimhan 2008], Parts & attributes [Wah 2011], etc.• Human-in-the-loop at test time [Branson 2010]• Our supervisor provides additional information at

training time which is leveraged for better category models

Related Work

• Rationales– Human feature selection in NLP [Raghavan 2005]– Spatial and attribute rationales [Donahue 2011]– Restricted to classification in attribute space– We can operate in any feature space

[Donahue and Grauman, ICCV 2011]

Imperfect Attribute Predictors

• In the end, all images labeled with ground truth

• Discriminative training can deal with outliers• Attributes are pre-trained, and so unlikely to

be severely flawed • Experiments: used predictors directly from

Parikh and Grauman, 2011

Large-scale Classification

• Categories may require expert knowledge, attributes need not

• Can show a few exemplars to verify category

0 10 20 30 40 5025

35

45

55

65

75

Baseline

Proposed (Human)

Proposed (Oracle)

Zero-shot

Results

Faces, Classification, Attribute features

(overestimate)

# iterations

Accu

racy

Label-based Feedback

Classification Large-scale Classification

Annotation

StreetOutdoor

CityGrayscale

….

Biased Binary Classification

• Simulate the different scenarios in responsesBenefits of attributes-based feedback• Small when label-based is very informative• Large when label-based feedback is weak

0 10 20 30 4042444648505254

Baseline

Proposed (Human)

Proposed (Or-acle)

Results

Scenes, Annotation, Image features

# iterations

Accu

racy

0 5 10 15 2025

35

45

55

65

75

Baseline

Proposed (Human)

Proposed (Or-acle)

Results

Faces, Biased binary classification, Attribute features

# iterations

Accu

racy

More results in the paper.

Traditional Active Learning

Concepts to teach:

C1 C2 CK…

Unlabeled pool of images

Classifiers:

h1 h2 hK…

[Label]Highest Entropy