relative attributes presenter: shuai zheng (kyle) supervised by philip h.s. torr author: devi parikh...
Post on 19-Dec-2015
223 views
TRANSCRIPT
Relative Attributes
Presenter: Shuai Zheng (Kyle)Supervised by Philip H.S. Torr
Author: Devi Parikh (TTI-Chicago) and Kristen Grauman (UT-Austin)
What is visual attributes?
• Attributes are properties observable in images that have human-designated names, such as ‘Orange’, ‘striped’, or ‘Furry’.
Learning Binary Attributes
• In PASCAL VOC Challenge, we learn to predict binary attributes. (e.g., dog? Or not a dog?)
Vittorio Ferrari, Andrew Zisserman. Learning Visual Attributes. NIPS 2007.
Not a catdog
O. Parkhi, A.Vedaldi C.V.Jawahar, A.Zisserman. The Truth About Cats and Dogs. ICCV 2011.
Problems within Binary Attributes
• Given an attribute it is easy to get labelled data on AMT(Amazon Mechanical Turk).
• But, where do attributes come from? Can we find a easier way to ask more people rather than experts to tag the images?
Problems within Binary AttributesSome tags are binary while some are relative.
Is furry Has four-legs
Has tail
Tail longer than donkeys’
Legs shorter than horses’
relativeBinary
Mule
Labeling data
What is relative attributes?
• Relative attribute indicates the strength of an attribute in an image with respect to other image rather than simply predicting the presence of an attribute.
Advantages of Relative Attributes
• Enhanced human-machine communication
• More informative
• Natural for humans
8
Contributions
• Propose a model to learn relative attributes– Allow relating images and categories to each
other– Learn ranking function for each attribute
• Give two novel applications based on the model– Zero-shot learning from attribute comparisons– Automatically generating relative image
Marr Prize 2011!
Contributions
• Propose a model to learn relative attributes– Allow relating images and categories to each
other– Learn ranking function for each attribute
• Give two novel applications based on the model– Zero-shot learning from attribute comparisons– Automatically generating relative image
Marr Prize 2011!
Learning Relative Attributes
For each attribute
Supervision is
open
Learning Relative Attributes
Learn a scoring function
that best satisfies constraints:
12
Image features
Learned parameters
Learning Relative Attributes
13
Max-margin learning to rank formulation
Based on [Joachims 2002]
12
34
56
Rank Margin
Image Relative Attribute Score
Learning binary attributes v.s. Learning relative attributes
Contributions
• Propose a model to learn relative attributes– Allow relating images and categories to each
other– Learn ranking function for each attribute
• Give two novel applications based on the model– Zero-shot learning from attribute comparisons– Automatically generating relative image
Marr Prize 2011!
Relative Zero-shot Learning
Training: Images from S seen categories and Descriptions of U unseen
categories
Need not use all attributes, or all seen categoriesTesting: Categorize image into one of S+U categories 16
Age: ScarlettCliveHugh Jared Miley
Smiling: JaredMiley
Relative Zero-shot Learning
Clive
Infer image category using max-likelihood
Can predict new classes based on their relationships to existing classes – without training images
17
Age: ScarlettCliveHughJared Miley
Smiling: JaredMileySm
iling
Age
Miley
S
J H
Contributions
• Propose a model to learn relative attributes– Allow relating images and categories to each
other– Learn ranking function for each attribute
• Give two novel applications based on the model– Zero-shot learning from attribute comparisons– Automatically generating relative image
Marr Prize 2011!
Automatic Relative Image Description
Density
Conventional binary description: not dense
Dense: Not dense:
Novel image
19
more dense than less dense than
DensityNovel image
20
Automatic Relative Image Description
C C H H H C F H H M F F I F
more dense than Highways, less dense than Forests
DensityNovel image
21
Automatic Relative Image Description
Contributions
• Propose a model to learn relative attributes– Allow relating images and categories to each
other– Learn ranking function for each attribute
• Give two novel applications based on the model– Zero-shot learning from attribute comparisons– Automatically generating relative image
Marr Prize 2011!
DatasetsOutdoor Scene Recognition (OSR)[Oliva 2001]
8 classes, ~2700 images, Gist6 attributes: open, natural, etc.
Public Figures Face (PubFig)[Kumar 2009]
8 classes, ~800 images, Gist+color11 attributes: white, chubby, etc.
23Attributes labeled at category levelhttp://ttic.uchicago.edu/~dparikh/relative.html
• Zero-shot learning– Binary attributes:
Direct Attribute Prediction [Lampert 2009]– Relative attributes via
classifier scores• Automatic image-description
– Binary attributes
24
+
+
+
–
– –
Baselines
6
4
5
3
2 1
bear turtle rabbit
furrybig
Relative Zero-shot Learning
An attribute is more discriminative when used relatively
Binary attributes
Rel. att. (classifier)
25
Rel. att.(ranker)
Relative (ours):
More natural than insidecity Less natural than highway
More open than street Less open than coast
Has more perspective than highway Has less perspective than insidecity
Binary (existing):
Not natural
Not open
Has perspective
Automatic Relative Image Description
26
Automatic Relative Image Description
18 subjects
Test cases:10OSR, 20 PubFig
27
Traditional Recognition
Dog Chimpanzee Tiger ???Tiger
28
Attributes-based Recognition
FurryWhite
BlackBig
StripedYellow
StripedBlackWhite
Big
Attributes provide a mode of
communication between humans and
machines!
[Lampert 2009][Farhadi 2009][Kumar 2009][Berg 2010][Parikh 2010]…
Zero-shot learningDescribing objectsFace verificationAttribute discoveryNameable attributes…
29
Dog Chimpanzee Tiger
Conclusions and Future WorkRelative attributes learnt as ranking functions
– Natural and accurate zero-shot learning of novel concepts by relating them to existing concepts
– Precise image descriptions for human interpretation
Attributes-based recognition is an interesting direction for the future object/scenes recognition.
30
Enhanced human-machine communication
