second’internaonal’workshop’on’parts’and’a5ributes …smaji/presentations/lexicon... ·...
TRANSCRIPT
![Page 1: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/1.jpg)
Second Interna,onal Workshop on Parts and A5ributesECCV 2012, Firenze, Italy
October, 2012Discovering a Lexicon of Parts and Attributes
Subhransu MajiResearch Assistant Professor
Toyota Technological Institute at Chicago
![Page 2: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/2.jpg)
Motivation
• Detailed object recognition
• Communication requires a lexicon
Diverse Visual Categories
High%heel(Blue(Shoe(
Berg et al., 10 Farhadi et al, 09
CUB 200 dataset, Visipedia project
![Page 3: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/3.jpg)
Source of part and attribute lexicons
• Field guides provide exhaustive lists when available
• Expert vs. Layman
• Task specific vs. not
![Page 4: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/4.jpg)
Source of part and attribute lexicons
• Captioned images
Limited by sources of such text
Image from Berg et al., ECCV’10
Descriptions are often not visual
![Page 5: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/5.jpg)
What are good attribute lexicons?
• Properties
• It should be easy to communicate
• It should be easy to differentiate instances from one another
images from google, cub 200
![Page 6: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/6.jpg)
Discriminative description task
Describe the (visual) differences between the two
![Page 7: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/7.jpg)
Discriminative description task
![Page 8: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/8.jpg)
Discriminative description task
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
Description
![Page 9: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/9.jpg)
Discriminative description task
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
Description
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
Discriminative Description
Helps elicit a lexicon that enables fine grained discriminationIs task specific by design
![Page 10: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/10.jpg)
Discriminative description task
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
Description
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
Discriminative Description
Helps elicit a lexicon that enables fine grained discriminationIs task specific by design
Non-accidental properties, Biederman 87Pragmatics of Language, Levinson 83
![Page 11: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/11.jpg)
Collecting descriptions on MTurkInterface on AMT
free form text(separated by ‘vs’)
Example annotations
Minimizes instruction biasKeeps the interface simple
![Page 12: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/12.jpg)
Example annotations: airplanespair 82/999; 5 good
facing left facing rightturbofan powered plane propeller powered planelonger tail shorter tailgreen rudder white rudderpassenger door open baggage hold door open"
pair 65/999; 5 good
propeller to the body propeller to the wingone rudder two ruddersthin body fat bodylow wings high wingsfacing towards left side facing slightly towards"
Images from airliners.net
![Page 13: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/13.jpg)
Example annotations: birdspair 10/1600; 5 good
black and white wings spotted wingswhite body spotted bodylarge eyes small eyessmall tail long tailv shaped beak pointed beak"
pair 43/1600; 5 good
yellow black body orange brown bodypointy beak shape beakshort tail long tailblack spot over head brown stripe over headshort leg long leg"
Images from CUB 200 dataset
![Page 14: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/14.jpg)
Analyzing the text: instance specific properties
Different properties are revealed for different instances
bird familyfur color
beak shapetail size
beak color
beak sizetail size
wing colorhead colorleg color
beak colorsitting vs. flyingfeather color
tail sizeleg color
![Page 15: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/15.jpg)
Analyzing the text: instance specific properties
Frequency of usage is a measure of its discriminability
![Page 16: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/16.jpg)
Analyzing the text: instance specific properties
Frequency of usage is a measure of its discriminability
![Page 17: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/17.jpg)
red rudder vs. white rudderpointy nose vs. round nose
sentence pairs
Analyzing the text : discovering a lexicon
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
![Page 18: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/18.jpg)
red rudder vs. white rudderpointy nose vs. round nose
sentence pairs
Analyzing the text : discovering a lexicon
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
ruddernose
parts
{red, white}{pointy, round}
modifiers
red rudder vs. white rudderpointy nose vs. round nose
sentence pairs
![Page 19: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/19.jpg)
red rudder vs. white rudderpointy nose vs. round nose
sentence pairs
Analyzing the text : discovering a lexicon
!"#$
• %&'()$• *'"$)(+,()$• -).$/0&0-$• *'"$-1..)-$
!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$
$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$
-01(.$-1..)-$!"#$%0,(45$-1..)-$
!"#$%+",)')-.)#%
ruddernose
parts
{red, white}{pointy, round}
modifiers
red rudder vs. white rudderpointy nose vs. round nose
sentence pairs
Key ideanouns : words that repeatmodifiers : words that are differentEach sentence has only one noun and modifier
![Page 20: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/20.jpg)
Analyzing the text : discovering a lexicon
Sentence alignment
yellowbeak
blackandwhitebeak
Used in NLP to initialize translation tables (IBM models)
![Page 21: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/21.jpg)
Analyzing the text : discovering a lexicon
Sentence alignment
yellowbeak
blackandwhitebeak
Used in NLP to initialize translation tables (IBM models)
GoalsDiscover a lexicon of parts, modifiers and part-modifier relations
Modifiers should be shared across attributes Estimate the frequency of each attribute
![Page 22: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/22.jpg)
A generative model of sentence pairs
![Page 23: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/23.jpg)
A generative model of sentence pairs
z
a
⇡
f
e
N
J
tI
✓
⌦ �
![Page 24: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/24.jpg)
A generative model of sentence pairs
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
![Page 25: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/25.jpg)
A generative model of sentence pairs
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...
part-modifier topic pairs
![Page 26: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/26.jpg)
A generative model of sentence pairs
zbeak-color
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...
part-modifier topic pairs
![Page 27: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/27.jpg)
A generative model of sentence pairs
zbeak-color
ttopic:colortopic:beak
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...
part-modifier topic pairs
![Page 28: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/28.jpg)
A generative model of sentence pairs
yellowbeak
NULL
ezbeak-color
ttopic:colortopic:beak
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...
part-modifier topic pairs
![Page 29: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/29.jpg)
A generative model of sentence pairs
yellowbeak
NULL
ezbeak-color
ttopic:colortopic:beak
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...
part-modifier topic pairs a
![Page 30: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/30.jpg)
A generative model of sentence pairs
yellowbeak
NULL
eblackandwhitebeak
fzbeak-color
ttopic:colortopic:beak
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...
part-modifier topic pairs a
![Page 31: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/31.jpg)
A generative model of sentence pairs
yellowbeak
NULL
eblackandwhitebeak
fzbeak-color
ttopic:colortopic:beak
z
a
⇡
f
e
N
J
tI
✓
⌦ �
yellow beak vs.
black and white beak
beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...
part-modifier topic pairs a
Initialize part and modifier topics using word alignments
![Page 32: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/32.jpg)
Parts, modifiers and attributes of airplanes
wheelwheels plane
engineengines rudder
wingswing
frontback nose facing body tail GLOBAL
onetwono
singlethreedoublefour
colorblackskylight
whiteblueordinarycoloreddark
whitegreenwhitered
pointyroundflat
pointedsharppointsquare
propellerpassenger
jetonly
militarycargo
smallbiglarge
medium
whiteredbluegreenyellowgrayorangebrown
topbottommiddledown
openclosedopenedclose
rightleft
slightly
onnearoff
200 images, 1000 random pairsImages from airliners.net
![Page 33: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/33.jpg)
Parts, modifiers and attributes of airplanes
wheelwheels plane
engineengines rudder
wingswing
frontback nose facing body tail GLOBAL
onetwono
singlethreedoublefour
colorblackskylight
whiteblueordinarycoloreddark
whitegreenwhitered
pointyroundflat
pointedsharppointsquare
propellerpassenger
jetonly
militarycargo
smallbiglarge
medium
whiteredbluegreenyellowgrayorangebrown
topbottommiddledown
openclosedopenedclose
rightleft
slightly
onnearoff
200 images, 1000 random pairsImages from airliners.net
1
![Page 34: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/34.jpg)
Parts, modifiers and attributes of airplanes
wheelwheels plane
engineengines rudder
wingswing
frontback nose facing body tail GLOBAL
onetwono
singlethreedoublefour
colorblackskylight
whiteblueordinarycoloreddark
whitegreenwhitered
pointyroundflat
pointedsharppointsquare
propellerpassenger
jetonly
militarycargo
smallbiglarge
medium
whiteredbluegreenyellowgrayorangebrown
topbottommiddledown
openclosedopenedclose
rightleft
slightly
onnearoff
200 images, 1000 random pairsImages from airliners.net
12
![Page 35: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/35.jpg)
Parts, modifiers and attributes of airplanes
wheelwheels plane
engineengines rudder
wingswing
frontback nose facing body tail GLOBAL
onetwono
singlethreedoublefour
colorblackskylight
whiteblueordinarycoloreddark
whitegreenwhitered
pointyroundflat
pointedsharppointsquare
propellerpassenger
jetonly
militarycargo
smallbiglarge
medium
whiteredbluegreenyellowgrayorangebrown
topbottommiddledown
openclosedopenedclose
rightleft
slightly
onnearoff
200 images, 1000 random pairsImages from airliners.net
12 3
![Page 36: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/36.jpg)
Parts, modifiers and attributes of airplanes
wheelwheels plane
engineengines rudder
wingswing
frontback nose facing body tail GLOBAL
onetwono
singlethreedoublefour
colorblackskylight
whiteblueordinarycoloreddark
whitegreenwhitered
pointyroundflat
pointedsharppointsquare
propellerpassenger
jetonly
militarycargo
smallbiglarge
medium
whiteredbluegreenyellowgrayorangebrown
topbottommiddledown
openclosedopenedclose
rightleft
slightly
onnearoff
200 images, 1000 random pairsImages from airliners.net
12 3localized vs. global
![Page 37: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/37.jpg)
Parts, modifiers and attributes of birds
bird wingsfeatherfeathers tail beak like body leg legs eyes neck head in fur GLOBAL
longshortsmalllargebig
vpointypointedpointbendbendedpointlyslightly
sparrowducksparowcroweagledovepigeonhumming
kiteparrot
brownblueyellowgrayredgreenspottedashlight
blackwhiteorange
fatslimsilmlean
sharproundflat
normalshapedcurvedbluntlittle
roundedordinaryoval
200 images, 1600 pairs1 image per category
CUB 200 dataset
beak-size, wing-color, tail-size, body color, bird type
![Page 38: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/38.jpg)
Parts, modifiers and attributes of people
handhands hair facing
snapsnape
thepicture spectacles in glass bag watch glasses GLOBAL
towardsleftright
backwardsforwardsidewards
manlady
womanboygirlladies
babyadultchildkid
childrenadults
shirttshirtjacketdresscoattshirts
wearingnot
smilinghaving
asiancaucasianafricansasianslatin
western
sidebackfront
backsidefrontaltowardturnturnedrear
sidewaysuprightus
blackwhitebluebrownblonderedgreengrayyellowcoloredpinkorange
singlecouple2non
doublegroupcouplesmany3
darklightfairteeskyshowbrightlighterdesignmediumthick
fatslim
normalthinleanfit
averageskinny
femalemale
fullhalfof
sleeveonlyfully
sleevelessjeensclosepartialrain
thinningtorsowaist
indooroutdoordoorhomelystage
insideoutsideout
handed
longshortsmallshot
womenmen
womens
youngoldoldermiddlematureelderly
twoonethreeboth
somebodyweight
sittingstandingwalkingridingcyclingsleepingdancingdriving
nowith
withoutalonebald
400 images, 1600 pairsRandom images from
PASCAL VOC 11
location, shirt vs tshirt, hair length, gender
![Page 39: Second’Internaonal’Workshop’on’Parts’and’A5ributes …smaji/presentations/lexicon... · 2019-12-21 · down open closed opened close right left slightly on near off 200](https://reader034.vdocuments.net/reader034/viewer/2022050509/5f99f3b10faaf36d3f0d92b9/html5/thumbnails/39.jpg)
Conclusions
• Discriminative description is an effective way to elicit a lexicon of attributes that are useful for fine-grained distinction
• Simple analysis of sentence pairs can help discover
• a lexicon of parts
• a lexicon of modifiers
• relationships between parts and modifiers (attributes)
• the relative frequency of these in a dataset
• A useful tool to bootstrap annotation collection process