Multimodal Deep Learning
Zeynep Akata
Zero-Shot Learning
Latent Embeddings for Zero-Shot Image ClassificationXian et.al., CVPR’16 & CVPR’17
WLarge
error
W₁
W₂
Linear compatibility function: large errors (left).
Piecewise-linear: significantly improves results (right).
Multi-Cue Zero-Shot Learning with Strong SupervisionAkata et.al., CVPR’16
Class
Embedding
F
Image
Embedding
Blue Jay Albatross
black
long tailblue
cone beak
Attributes: costly but good, W2V: cheap but weak.
Strong visual supervision: to compensate weak W2V.
Learning Deep Representations of Fine-Grained Visual
Descriptions Reed et.al., CVPR’16
2 4 6 8 10
# of train sentences per image
30
35
40
45
50
55
60
To
p-1
Acc.
(in
%)
Zero-Shot in CUB
Ours (word)Ours (char)LSTMTCNNAttributes
CNN-RNN: fast + models sequence of words or characters
With >4 sentences: outperforms SoA with attributes
Gaze Embeddings for Zero-Shot Image Classification
Karessli et.al., CVPR’17
Original image Gaze pointsGaze Features
with Grid (GFG)
Gaze histogram (GH)
0
0
0
0
0
02
13
22Raw gaze data
0
22
0
0
13
0
0
2
0
Gaze Features
without Grid (GFS)
GH Embedding per class
GFS Embedding per class
Gaze heatmap
Outlier
removal
Gaze data
collection
GFG Embedding per class+
x
y
�₁1
2
3
�₂
d
...
x 9
+x
y
d
...
x 3
�₁1
2
3
�₂
Generating: Vision + Language
Generarive Adversarial Text to Image SynthesisReed et.al. ICML’16
GAN conditioned on sentences: real/fake, matching/not
a tiny bird, with a
tiny beak, tarsus and
feet, a blue crown,
blue coverts, and
black cheek patch
this small bird has
a yellow breast,
brown crown, and
black superciliary
an all black bird
with a distinct
thick, rounded bill.
this bird is different
shades of brown all
over with white and
black spots on its
head and back
GAN - CLS
GAN - INT
GAN
GAN - INT
- CLS
the gray bird has a
light grey head and
grey webbed feetGT
Generates pixels from characters: intuitive
Language compensates lack of large # training images
Learning What and Where to Draw Reed et.al. NIPS’16
This bird has a yellow head, black eyes, a gray pointy beak and orange lines on its breast.
This water bird has a long white neck, black body, yellow beak and black head.
This bird is large, completely black, with a long pointy beak and black eyes.
This bird is completely red with a red and cone-shaped beak, black face and a red nape.
This white bird has gray wings, red webbed feet and a long, curved and yellow beak.
This small bird has a blue and gray head, pointy beak and a white belly.
GT
GT
GT
GT
GT
GT
Generating Visual Explanations Hendricks et.al. ECCV’16
D: this bird has a white breast
black wings and a red spot on its
head.
E: this is a white bird with a
black wing and a black and white
striped head.
D: this bird has a white breast
black wings and a red spot on its
head.
E: this is a black and white bird
with a red spot on its crown.
This is a Downy Woodpecker because... This is a Downy Woodpecker because...
Class + image conditional LSTM & Reinforcement Loss
Learns to mention class-specific and visible properties
