visual sense disambiguation: a multimodal approach phd thesis by kate saenko computer science and ai...
TRANSCRIPT
Visual Sense Disambiguation: A Multimodal Approach
PhD thesis by Kate Saenko
Computer Science and AI Lab
Massachusetts Institute of Technology
Advisor: Trevor Darrell
PhD defense by Kate Saenko 2
The challenge of large scale object recognition
• How to get examples of 10,000+ categories?– Collection of training images is time-
consuming, subjective– But the Web has billions of images!
• How to build precise models based on unlabeled image data?
• How to learn visual models on the fly, based on user input?
PhD defense by Kate Saenko 3
Multimodal context
speech text
image
collective knowledge
PhD defense by Kate Saenko 4
Main Contributions
• Proposed a method that combines images and spoken utterances to learn object models
• Developed an unsupervised approach that learns visual models from unlabeled images, text, and dictionaries
This is a bag…
… The Tote is the perfect example of two handbag design principles that ... The lines of this tote are incredibly sleek, but ... The semi buckles that form the handle attachments are ...
PhD defense by Kate Saenko 5
This is a bag
PhD defense by Kate Saenko 6
Noun• bag, container (a flexible container with a single opening)• bag, handbag, pocketbook, purse (a container used for carrying money
and small personal items or accessories (especially by women))• bag, travelling bag, suitcase (a rectangular container for carrying clothes)
bag
bag
bag
bag
… The Tote is the perfect example of two handbag design principles that ... The lines of this tote are incredibly sleek, but ... The semi buckles that form the handle attachments are ...
PhD defense by Kate Saenko 7
Outline
• Audio-visual object recognition– Related work– Fusion model and experiments*
• Unsupervised text and image models– Related work– WISDOM: probabilistic dictionary-based image sense model– Concrete WISDOM: identifying tangible objects
*see Saenko and Darrell. Object category recognition using probabilistic fusion of speech and image classifiers. MLMI, 2007.
PhD defense by Kate Saenko 8
Audio-visual object recognition
speech text
image
dictionary
PhD defense by Kate Saenko 9
Task: object recognition with audio-visual input*
Speechrecognizer
Speech DB
*e.g. BIRON robot, see S. Li and B.Wrede. “Why and how to model multi-modal interaction for a mobile robot companion," In Proc. AAAI, 2007.
lamp lamplamplamp
label <lamp>+
PhD defense by Kate Saenko 10
Speech, image can be ambiguous…
a pan...
That’s a pen!
Copy machine..
ant → fan
face → bass
piano → cannon
PhD defense by Kate Saenko 11
Proposal: use both channels to help disambiguate underlying word
objectrecognizer
PhD defense by Kate Saenko 12
Fusion of speech and image classifiers
ObjectClassifier
Speechrecognizer
Speech DB
ImageClassifier
Image DB
lamp
• Improve existing method by using both modalities
• Explore late fusion of classifier outputs
– Mean rule– Product rule
PhD defense by Kate Saenko 13
Experiments with 101 objects
• Asked users to speak object name for Caltech101, added noise• Plot shows benefit from fusion across noise levels
PhD defense by Kate Saenko 14
Remaining issues…
objectrecognizer
PhD defense by Kate Saenko 15
Unsupervised object models
speech text
image
dictionary
PhD defense by Kate Saenko 16
Next
• Audio-visual object recognition– Related work– Fusion model and experiments
• Unsupervised text and image models– Related work– WISDOM: probabilistic dictionary-based image sense model– Concrete WISDOM: identifying tangible objects
PhD defense by Kate Saenko 17
How can we learn a rich variety of visual concepts?
PhD defense by Kate Saenko 18
Image Sense Disambiguation
Would rather watch… Suicide watch
Hurricane, tornado watch
Watch out!
Celebrity watch
PhD defense by Kate Saenko 19
Text contexts
icrystal rfid wrist watch features watch masterpiece innovative watch making craftsmanship absolute precision fine charm high scratch resistance anti-
allergenic characteristics make chronometer true jewel s wrist water proof sleek stylish wrist watch solar powered available watch ticket key purse identity card special offer place order rfid wrist watch absolutely free rfid watch black
wrist strap rfid watch orange wrist strap rfid watch stainless steel privacy
disclaimer copyright icrystal pty website
PhD defense by Kate Saenko 20
Topic 1rolex
service repair battery omega replica
tag heuer breitling swiss
replace gucci button price band
…
Topic 2new world media right said
house april
obama islam march bush war
american time
…
Latent Dirichlet allocation (LDA) (Blei et al. ‘03)
• One of several techniques for discovering latent dimensions in bag-of-words data
α θ
z
w
β
φ
K M
Nd
d
word
P(w|z)
topic
document
P(z|d)
PhD defense by Kate Saenko 21
Latent Topics
icrystal rfid wrist watch features watch masterpiece innovative watch making craftsmanship absolute precision fine
charm high scratch resistance anti-allergenic characteristics make
chronometer true jewel wrist water proof sleek stylish wrist watch solar powered available watch ticket key
purse identity card special offer place order rfid wrist watch absolutely free rfid watch black wrist strap rfid watch
orange wrist strap rfid watch stainless steel privacy disclaimer copyright
icrystal pty website
PhD defense by Kate Saenko 22
Overview of approaches to web-based object model learning
• Some learn only from image features– (Li et al.07) bootstrap from labeled images– (Fergus et al.05) select correct image topic
• Some incorporate text features– (Schroff et al.07) use a category-independent text classifier– (Berg and Forsyth 06) ask user to sort text topics
• None address polysemy directly– (Loeff et al.06) do image sense discrimination, not
identification
• All rely on labeled images of correct sense
PhD defense by Kate Saenko 23
Next
• Audio-visual object recognition– Related work– Fusion model and experiments
• Unsupervised text and image models– Related work– WISDOM: probabilistic dictionary-based image sense
model*– Concrete WISDOM: identifying tangible objects
*see Saenko and Darrell. Unsupervised learning of visual sense models for polysemous words. NIPS 2008.
PhD defense by Kate Saenko 24
How can we ground image senses in the absence of labeled examples?
WORDNET: Noun•S: (n) watch, ticker (a small portable timepiece) •S: (n) watch (a period of time (4 or 2 hours) during which some of a ship's crew are on duty) •S: (n) watch, vigil (a purposeful surveillance to guard or observe) •S: (n) watch (the period during which someone (especially a guard) is on duty) •S: (n) lookout, lookout man, sentinel, sentry, watch, spotter, scout, picket (a person employed to keep watch for some anticipated event) •S: (n) vigil, watch (the rite of staying awake for devotional purposes (especially on the eve of a religious festival))
WIKIPEDIA: Watch may also refer to:•Watch system, a period of work duty •Tropical cyclone warnings and watches, alerts issued to coastal areas threatened by severe storms •Watch (Unix), a Unix command •Watch (TV channel) a TV station launching in Autumn 2008 •Watch (computer programming) •Help:Watching pages on Wikipedia •Watch (dog), name of the pet dog in the the Boxcar Children
D. Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. ACL, 1995.
PhD defense by Kate Saenko 25
Sense-specific classifier
training images
Web Image Sense DictiOnary Model
Search Engine WatchSearch Engine Watch is the authoritative guide to search engine marketing (SEM) and search engine optimization (SEO), offering the latest news about search ...searchenginewatch.com/ - 38k - Cached - Similar pages - Note thiswatch - MDCWatches for assignment to a property named prop in this object, calling handler(prop, oldval, newval) whenever prop is set and storing the return value in ...developer.mozilla.org/en/Core_JavaScript_1.5_Reference/Global_Objects/Object/watch - 30k - Cached - Similar pages - Note this
Search Engine WatchSearch Engine Watch is the authoritative guide to search engine marketing (SEM) and search engine optimization (SEO), offering the latest news about search ...searchenginewatch.com/ - 38k - Cached - Similar pages - Note thiswatch - MDCWatches for assignment to a property named prop in this object, calling handler(prop, oldval, newval) whenever prop is set and storing the return value in ...developer.mozilla.org/en/Core_JavaScript_1.5_Reference/Global_Objects/Object/watch - 30k - Cached - Similar pages - Note this
dictionary definitions
unlabeled text
dictionary model P( sense | data)
WISDOM does:1. image sense
disambiguation
2. dataset collection
3. classification of unseen images
noun
web images
fosil wrist watch a800 x 628 - 107k - jpg
amgmedia.com
watch-1(ticker)
PhD defense by Kate Saenko 26
WISDOM: Using dictionary entries to ground senses
• Use entry text to learn a probability distribution over words for that sense
• Problem: entries contain very little text– Expand by adding synonyms, example sentences, etc.– Still, very few words are covered!
•S: (n) mouse (any of numerous small rodents typically resembling diminutive rats having pointed snouts and small ears on elongated bodies with slender usually hairless tails) •S: (n) mouse (any of numerous small rodents typically resembling diminutive rats having pointed snouts and small ears on elongated bodies with slender usually hairless tails) •direct hyponym / full hyponym
•S: (n) house mouse, Mus musculus (brownish-grey Old World mouse now a common household pest worldwide) •S: (n) harvest mouse, Micromyx minutus (small reddish-brown Eurasian mouse inhabiting e.g. cornfields) •S: (n) field mouse, fieldmouse (any nocturnal Old World mouse of the genus Apodemus inhabiting woods and fields and gardens) •S: (n) nude mouse (a mouse with a genetic defect that prevents them from growing hair and also prevents them from immunologically rejecting human cells and tissues; widely used in preclinical trials) •S: (n) wood mouse (any of various New World woodland mice)
•direct hypernym / inherited hypernym / sister term •S: (n) rodent, gnawer (relatively small placental mammals having a single pair of constantly growing incisor teeth specialized for gnawing)
PhD defense by Kate Saenko 27
WISDOM: Probabilistic dictionary-based model
• Main idea:
– Using LDA, learn latent sense-like dimensions on a large amount of related text,
– Model dictionary senses in LDA space:
• Map image contexts to topics• Map topics to senses
Search Engine WatchSearch Engine Watch is the authoritative guide to search engine marketing (SEM) and search engine optimization (SEO), offering the latest news about search ...searchenginewatch.com/ - 38k - Cached - Similar pages - Note thiswatch - MDCWatches for assignment to a property named prop in this object, calling handler(prop, oldval, newval) whenever prop is set and storing the return value in ...developer.mozilla.org/en/Core_JavaScript_1.5_Reference/Global_Objects/Object/watch - 30k - Cached - Similar pages - Note this
Search Engine WatchSearch Engine Watch is the authoritative guide to search engine marketing (SEM) and search engine optimization (SEO), offering the latest news about search ...searchenginewatch.com/ - 38k - Cached - Similar pages - Note thiswatch - MDCWatches for assignment to a property named prop in this object, calling handler(prop, oldval, newval) whenever prop is set and storing the return value in ...developer.mozilla.org/en/Core_JavaScript_1.5_Reference/Global_Objects/Object/watch - 30k - Cached - Similar pages - Note this
unlabeled text
LDA
PhD defense by Kate Saenko 28
WISDOM sense model
• Given a query word with sense s with values in set {1,…,S}, and a text document d, the probability of sense is
d
z
N
s
• Define the likelihood of topic z given sense s with entry words es= w1,…,wEs as
• To compute probability of sense given topic
PhD defense by Kate Saenko 29
WISDOM: Incorporating Image Features
• Use LDA to discover visual topics v=1,…,L,
• Then estimate the conditional probability P(s|v)
• Given a test image di*, we can compute
• Combine contributions of image and text:
PhD defense by Kate Saenko 30
WISDOM classifier
SVM classifier
training images
Search Engine WatchSearch Engine Watch is the authoritative guide to search engine marketing (SEM) and search engine optimization (SEO), offering the latest news about search ...searchenginewatch.com/ - 38k - Cached - Similar pages - Note thiswatch - MDCWatches for assignment to a property named prop in this object, calling handler(prop, oldval, newval) whenever prop is set and storing the return value in ...developer.mozilla.org/en/Core_JavaScript_1.5_Reference/Global_Objects/Object/watch - 30k - Cached - Similar pages - Note this
Search Engine WatchSearch Engine Watch is the authoritative guide to search engine marketing (SEM) and search engine optimization (SEO), offering the latest news about search ...searchenginewatch.com/ - 38k - Cached - Similar pages - Note thiswatch - MDCWatches for assignment to a property named prop in this object, calling handler(prop, oldval, newval) whenever prop is set and storing the return value in ...developer.mozilla.org/en/Core_JavaScript_1.5_Reference/Global_Objects/Object/watch - 30k - Cached - Similar pages - Note this
dictionary definitions
unlabeled text
dictionary model P( sense | data)
noun
web images
fosil wrist watch a800 x 628 - 107k - jpg
amgmedia.com
watch-1(ticker)
PhD defense by Kate Saenko 31
Evaluation datasets
• Collected by querying Image Search – MIT-ISD: bass, face, mouse, speaker, watch
– MIT-OFFICE: cellphone, fork, hammer, keyboard, mug, pliers, scissors, stapler, telephone, watch
– UIUC-ISD: bass, crane, squash
core relatedcore relatedunrelated ???
PhD defense by Kate Saenko 32
Experimental Setup
1. Task: Image sense disambiguation (ISD) in search results– Separate images according to visual sense
– “core” labels are positive class, “related” and “unrelated” negative
– Metrics: true positives vs. false positives (ROC), recall-precision curve (RPC)
2. Task: object classification in a novel image– Classify image as having correct object category or not
– “core” labels are positive class, other keyword’s “core” senses are negative class
– Metric: percent correct
PhD defense by Kate Saenko 33
ISD example results
squash: sports
squash: vegetable
bass: musical instrument
bass: fish
bass: raw web image data
squash: raw web image data
PhD defense by Kate Saenko 34
yahoo
musical range
polyph. range
male singer
sea bass
freshwater bass
basso, voice
instrument
spiny fish
yahoo
musical range
polyph. range
male singer
sea bass
freshwater bass
basso, voice
instrument
spiny fish
ISD Results: ROC using each WordNet sense for BASS
BASSTr
ue p
ositi
ve ra
te
False positive rate
PhD defense by Kate Saenko 35
ISD Results: RPC using true sense
yahoo wisdom
Retrieval of core senses on UIUC-ISD
PhD defense by Kate Saenko 36
Results: object classification
• Baseline approach: – Automatically generate sense-specific keywords from WordNet– Append word to synonyms and direct hypernyms– Limit queries to 3 terms– Example: mouse + computer, mouse + electronic device
• Plot shows average accuracy across five objects in the MIT-ISD dataset (each is a two-class problem with chance performance of 50%)
85%
75%
65%
55%
50 100 150 200 250 300
Number of training images
baselinewisdom
Accu
racy
PhD defense by Kate Saenko 37
Next
• Audio-visual object recognition– Related work– Fusion model and experiments
• Unsupervised text and image models– Related work– WISDOM: probabilistic dictionary-based image sense model– Concrete WISDOM: identifying tangible objects*
*see Saenko and Darrell, Filtering Abstract Senses From Image Search Results, NIPS 2009.
PhD defense by Kate Saenko 38
Query Word: “cup”
Online DictionaryWord to search for:Noun
Online DictionaryWord to search for:Noun
cup Search Dictionary
• cup (a small open container usually used for drinking; usually has a handle) "he put the cup back in the saucer"; "the handle of the cup was missing"
• cup, loving cup (a large metal vessel with two handles that is awarded as a trophy to the winner of a competition) "the school kept the cups is a special glass case”
• a major sporting event or competition “the world cup”, “the Stanley cup”
Concrete WISDOM
Object Sense: drinking container
Abstract Sense: sporting event
Object Sense: loving cup (trophy)
Removing Abstract Senses
PhD defense by Kate Saenko 39
mouse
rodent
beaver
mammal
cow…
…
How can we identify abstract senses?
Mouse: Noun•<noun.animal>S: (n) mouse (any of numerous small rodents typically resembling diminutive rats having pointed snouts and small ears on elongated bodies with slender usually hairless tails) •<noun.state>S: (n) shiner, black eye, mouse (a swollen bruise caused by a blow to the eye) •<noun.person>S: (n) mouse (person who is quiet or timid) •<noun.artifact>S: (n) mouse, computer mouse (a hand-operated electronic device that controls the coordinates of a cursor on your computer screen…)
• Idea: use the ontological information available via WordNet– semantic relations between concepts (hypernym, part, etc.)– lexical tags:
PhD defense by Kate Saenko 40
Experimental Setup
Table: Concrete Senses Identified by WISDOM
• Task: ISD using concrete-sense WISDOM– all “core” and “related” labels of keyword are positive class,
“unrelated” labels are negative class
PhD defense by Kate Saenko 41
Results: Filtering visual senses
Yahoo Search: “telephone”DICTIONARY
1: (n) telephone, phone, telephone set (electronic equipment that converts sound into electrical signals that can be transmitted over distances and then converts received signals back into sounds)
2: (n) telephone, telephony (transmitting speech at a distance)
PhD defense by Kate Saenko 42
Results: Filtering visual senses
Artifact sense: “telephone”DICTIONARY
1: (n) telephone, phone, telephone set (electronic equipment that converts sound into electrical signals that can be transmitted over distances and then converts received signals back into sounds)
2: (n) telephone, telephony (transmitting speech at a distance)
PhD defense by Kate Saenko 43
Results: RPC of all concrete senses
Retrieval of core+related concrete senses on UIUC-ISD
yahoo wisdom
PhD defense by Kate Saenko 44
Further Improvement: Topic adaptation
• Original LDA topics are learned on text-only unlabeled data• Adapt to image-text data via semi-supervised Gibbs sampling• E.g.: one of “fork” topics:
product bike null tool tube seal set price oil
knife spoon spring ship use item accessory
handle shop order remove store custom
home weight steel supply cap clamp fit false
. . .
cutlery knife spoon product set price handle
steel tool item
stainless null bike
tube seal oil knive
kitchen utensil ship
order use table sp ring
supply design piece
carve weight shop . . .
PhD defense by Kate Saenko 45
“fork”: using original topics
unrelated:fork lift
road forkbike fork
knife…
PhD defense by Kate Saenko 46
“fork”: using adapted topics
unrelated:fork lift
road forkbike fork
knife…
PhD defense by Kate Saenko 47
Results on MIT-OFFICE
• The average area under the RPC improves from 0.47 to 0.57• Detailed RPCs:
yahoo wisdom
PhD defense by Kate Saenko 48
Conclusions
• Showed that combining speech with image input may be advantageous for object recognition
• Presented WISDOM, an unsupervised method to learn sense-specific object models from images and text harvested from the web
• Extended WISDOM to filter out non-physical word senses based on WordNet semantic structure
PhD defense by Kate Saenko 49
Future work: WISDOM-enabled interactive training
speech text
image
dictionary
supervised classifierWISDOM
PhD defense by Kate Saenko 50