Zero-Shot Object Recognition Using Semantic Label Vectors

Traditional Object Recognition

Lion classifier

LionNot Lion

3

Traditional Object Detection

Learns a weight vector to distinguish between “Lion” and others

Limitation of Traditional Object Recognition

• Too many objects in the nature• Humans can recognize between 5,000 and

30,000 object categories• Collecting training images for all these object

categories is tedious and expensive

Transfer Learning

• Transfer the knowledge of known objects to recognize previously unseen objects

• Use shared properties such as physical attributes

• Use contextual information from textual knowledge base

Motivation: Finding images based on textual description

The Bengal Tiger The Bengal tiger's has zebra like

stripes ranging from dark brown to black

In comparison, a weight range of 150 to 189 kg (331 to 417 lb) is considered fairly average for a male African lion

The biggest and perhaps most fearsome of the world's big cats, the tiger shares 95.6 percent of its DNA with humans' cute and furry companions, domestic cats.

Can you find the “Tiger” based on the existing knowledge?

The Bengal Tiger The Bengal tiger's coat is yellow

to light orange, with stripes ranging from dark brown to black

In comparison, a weight range of 150 to 189 kg (331 to 417 lb) is considered fairly average for a male African lion

The biggest and perhaps most fearsome of the world's big cats, the tiger shares 95.6 percent of its DNA with humans' cute and furry companions, domestic cats.

8

Word Vector Representation

Attributes

10

Knowledge Transfer Using Wordvector

11

Transferring knowledge

Unknown classes Known classes

12

Baselines

WordVector DistanceWordNet Distance

Word vector for “Lion”

Word vector for “Tiger”

Predicted known class

Closest unknown class

13

Zero-Shot Object Recognition

Unknown classes Known classes