a probabilistic approach to semantic representation
DESCRIPTION
A Probabilistic Approach to Semantic Representation. Tom Griffiths Mark Steyvers Josh Tenenbaum. How do we store the meanings of words? question of representation requires efficient abstraction. How do we store the meanings of words? question of representation - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/1.jpg)
A Probabilistic Approach to Semantic Representation
Tom Griffiths
Mark Steyvers
Josh Tenenbaum
![Page 2: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/2.jpg)
• How do we store the meanings of words?– question of representation– requires efficient abstraction
![Page 3: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/3.jpg)
• How do we store the meanings of words?– question of representation– requires efficient abstraction
• Why do we store this information?– function of semantic memory– predictive structure
![Page 4: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/4.jpg)
Latent Semantic Analysis(Landauer & Dumais, 1997)
1
…
6
…
11
…
spaces
…
6195semantic
2120in
3034words
Doc3 … Doc2Doc1
SVD words
in
semantic
spaces
X U D V T
co-occurrence matrix high dimensional space
![Page 5: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/5.jpg)
Mechanistic Claim
Some component of word meaning can be extracted from co-occurrence statistics
![Page 6: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/6.jpg)
Mechanistic Claim
Some component of word meaning can be extracted from co-occurrence statistics
But…– Why should this be true?– Is the SVD the best way to treat these data?– What assumptions are we making about meaning?
![Page 7: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/7.jpg)
Mechanism and Function
Some component of word meaning can be extracted from co-occurrence statistics
Semantic memory is structured to aid retrieval via context-specific prediction
![Page 8: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/8.jpg)
Functional Claim
Semantic memory is structured to aid retrieval via context-specific prediction
– Motivates sensitivity to co-occurrence statistics– Identifies how co-occurrence data should be used– Allows the role of meaning to be specified exactly,
and finds a meaningful decomposition of language
![Page 9: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/9.jpg)
A Probabilistic Approach
• The function of semantic memory– The psychological problem of meaning
– One approach to meaning
• Solving the statistical problem of meaning– Maximum likelihood estimation
– Bayesian statistics
• Comparisons with Latent Semantic Analysis– Quantitative
– Qualitative
![Page 10: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/10.jpg)
A Probabilistic Approach
• The function of semantic memory– The psychological problem of meaning
– One approach to meaning
• Solving the statistical problem of meaning– Maximum likelihood estimation
– Bayesian statistics
• Comparisons with Latent Semantic Analysis– Quantitative
– Qualitative
![Page 11: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/11.jpg)
The Function of Semantic Memory
• To predict what concepts are likely to be needed in a context, and thereby ease their retrieval
• Similar to rational accounts of categorization and memory (Anderson, 1990)
• Same principle appears in semantic networks (Collins & Quillian, 1969; Collins & Loftus, 1975)
![Page 12: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/12.jpg)
The Psychological Problem of Meaning
• Simply memorizing whole word-document co-occurrence matrix does not help
• Generalization requires abstraction, and this abstraction identifies the nature of meaning
• Specifying a generative model for documents allows inference and generalization
![Page 13: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/13.jpg)
One Approach to Meaning
• Each document a mixture of topics
• Each word chosen from a single topic
• from parameters
• from parameters
![Page 14: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/14.jpg)
One Approach to Meaning
HEART 0.2 LOVE 0.2SOUL 0.2TEARS 0.2JOY 0.2SCIENTIFIC 0.0KNOWLEDGE 0.0WORK 0.0RESEARCH 0.0MATHEMATICS 0.0
HEART 0.0 LOVE 0.0SOUL 0.0TEARS 0.0JOY 0.0 SCIENTIFIC 0.2KNOWLEDGE 0.2WORK 0.2RESEARCH 0.2MATHEMATICS 0.2
topic 1 topic 2
w P(w|z = 1) = (1) w P(w|z = 2) = (2)
![Page 15: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/15.jpg)
Choose mixture weights for each document, generate “bag of words”
One Approach to Meaning
= {P(z = 1), P(z = 2)}
{0, 1}
{0.25, 0.75}
{0.5, 0.5}
{0.75, 0.25}
{1, 0}
MATHEMATICS KNOWLEDGE RESEARCH WORK MATHEMATICS RESEARCH WORK SCIENTIFIC MATHEMATICS WORK
SCIENTIFIC KNOWLEDGE MATHEMATICS SCIENTIFIC HEART LOVE TEARS KNOWLEDGE HEART
MATHEMATICS HEART RESEARCH LOVE MATHEMATICS WORK TEARS SOUL KNOWLEDGE HEART
WORK JOY SOUL TEARS MATHEMATICS TEARS LOVE LOVE LOVE SOUL
TEARS LOVE JOY SOUL LOVE TEARS SOUL SOUL TEARS JOY
![Page 16: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/16.jpg)
z
w
One Approach to Meaning
• Generative model for co-occurrence data
• Introduced by Blei, Ng, and Jordan (2002)
• Clarifies pLSI (Hofmann, 1999)
![Page 17: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/17.jpg)
Matrix Interpretationw
ords
documents
wor
ds
topics
topi
cs
documents
normalizedco-occurrence matrix
mixtureweights
mixturecomponents
A form of non-negative matrix factorization
![Page 18: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/18.jpg)
wor
ds
documents
U D V
wor
ds
vectors
vectorsve
ctor
s
vect
ors documents
wor
ds
documents
wor
ds
topics
topi
cs
documents
Matrix Interpretation
![Page 19: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/19.jpg)
The Function of Semantic Memory
• Prediction of needed concepts aids retrieval
• Generalization aided by a generative model
• One generative model: mixtures of topics
• Gives non-negative, non-orthogonal factorization of word-document co-occurrence matrix
![Page 20: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/20.jpg)
A Probabilistic Approach
• The function of semantic memory– The psychological problem of meaning
– One approach to meaning
• Solving the statistical problem of meaning– Maximum likelihood estimation
– Bayesian statistics
• Comparisons with Latent Semantic Analysis– Quantitative
– Qualitative
![Page 21: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/21.jpg)
The Statistical Problem of Meaning
• Generating data from parameters easy
• Learning parameters from data is hard
• Two approaches to this problem– Maximum likelihood estimation– Bayesian statistics
![Page 22: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/22.jpg)
Inverting the Generative Model
• Maximum likelihood estimation
• Variational EM (Blei, Ng & Jordan, 2002)
• Bayesian inference
WT + DT parameters
WT + T parameters
0 parameters
![Page 23: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/23.jpg)
Bayesian Inference
• Sum in the denominator over Tn terms
• Full posterior only tractable to a constant
![Page 24: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/24.jpg)
Markov Chain Monte Carlo
• Sample from a Markov chain which converges to target distribution
• Allows sampling from an unnormalized posterior distribution
• Can compute approximate statistics from intractable distributions
(MacKay, 2002)
![Page 25: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/25.jpg)
Gibbs Sampling
For variables x1, x2, …, xn
Draw xi(t) from P(xi|x-i)
x-i = x1(t), x2
(t),…, xi-1(t)
, xi+1(t-1)
, …, xn(t-1)
![Page 26: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/26.jpg)
Gibbs Sampling
(MacKay, 2002)
![Page 27: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/27.jpg)
Gibbs Sampling
• Need full conditional distributions for variables
• Since we only sample z we need
number of times word w assigned to topic j
number of times topic j used in document d
![Page 28: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/28.jpg)
Gibbs Sampling
i wi di zi123456789
101112...
50
MATHEMATICSKNOWLEDGE
RESEARCHWORK
MATHEMATICSRESEARCH
WORKSCIENTIFIC
MATHEMATICSWORK
SCIENTIFICKNOWLEDGE
.
.
.JOY
111111111122...5
221212212111...2
iteration1
![Page 29: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/29.jpg)
Gibbs Sampling
i wi di zi zi123456789
101112...
50
MATHEMATICSKNOWLEDGE
RESEARCHWORK
MATHEMATICSRESEARCH
WORKSCIENTIFIC
MATHEMATICSWORK
SCIENTIFICKNOWLEDGE
.
.
.JOY
111111111122...5
221212212111...2
?
iteration1 2
![Page 30: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/30.jpg)
Gibbs Sampling
i wi di zi zi123456789
101112...
50
MATHEMATICSKNOWLEDGE
RESEARCHWORK
MATHEMATICSRESEARCH
WORKSCIENTIFIC
MATHEMATICSWORK
SCIENTIFICKNOWLEDGE
.
.
.JOY
111111111122...5
221212212111...2
?
iteration1 2
![Page 31: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/31.jpg)
Gibbs Sampling
i wi di zi zi123456789
101112...
50
MATHEMATICSKNOWLEDGE
RESEARCHWORK
MATHEMATICSRESEARCH
WORKSCIENTIFIC
MATHEMATICSWORK
SCIENTIFICKNOWLEDGE
.
.
.JOY
111111111122...5
221212212111...2
?
iteration1 2
![Page 32: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/32.jpg)
Gibbs Sampling
i wi di zi zi123456789
101112...
50
MATHEMATICSKNOWLEDGE
RESEARCHWORK
MATHEMATICSRESEARCH
WORKSCIENTIFIC
MATHEMATICSWORK
SCIENTIFICKNOWLEDGE
.
.
.JOY
111111111122...5
221212212111...2
2?
iteration1 2
![Page 33: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/33.jpg)
Gibbs Sampling
i wi di zi zi123456789
101112...
50
MATHEMATICSKNOWLEDGE
RESEARCHWORK
MATHEMATICSRESEARCH
WORKSCIENTIFIC
MATHEMATICSWORK
SCIENTIFICKNOWLEDGE
.
.
.JOY
111111111122...5
221212212111...2
21?
iteration1 2
![Page 34: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/34.jpg)
Gibbs Sampling
i wi di zi zi123456789
101112...
50
MATHEMATICSKNOWLEDGE
RESEARCHWORK
MATHEMATICSRESEARCH
WORKSCIENTIFIC
MATHEMATICSWORK
SCIENTIFICKNOWLEDGE
.
.
.JOY
111111111122...5
221212212111...2
211?
iteration1 2
![Page 35: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/35.jpg)
Gibbs Sampling
i wi di zi zi123456789
101112...
50
MATHEMATICSKNOWLEDGE
RESEARCHWORK
MATHEMATICSRESEARCH
WORKSCIENTIFIC
MATHEMATICSWORK
SCIENTIFICKNOWLEDGE
.
.
.JOY
111111111122...5
221212212111...2
2112?
iteration1 2
![Page 36: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/36.jpg)
Gibbs Sampling
i wi di zi zi zi123456789
101112...
50
MATHEMATICSKNOWLEDGE
RESEARCHWORK
MATHEMATICSRESEARCH
WORKSCIENTIFIC
MATHEMATICSWORK
SCIENTIFICKNOWLEDGE
.
.
.JOY
111111111122...5
221212212111...2
211222212212...1
…
222122212222...1
iteration1 2 … 1000
![Page 37: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/37.jpg)
pixel = word image = document
sample each pixel froma mixture of topics
A Visual Example: Bars
![Page 38: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/38.jpg)
A Visual Example: Bars
![Page 39: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/39.jpg)
From 1000 Images
![Page 40: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/40.jpg)
Interpretable Decomposition
• SVD gives a basis for the data, but not an interpretable one
• The true basis is not orthogonal, so rotation does no good
![Page 41: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/41.jpg)
Application to Corpus Data
• TASA corpus: text from first grade to college
• Vocabulary of 26414 words
• Set of 36999 documents
• Approximately 6 million words in corpus
![Page 42: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/42.jpg)
THEORYSCIENTISTS
EXPERIMENTOBSERVATIONS
SCIENTIFICEXPERIMENTSHYPOTHESIS
EXPLAINSCIENTISTOBSERVED
EXPLANATIONBASED
OBSERVATIONIDEA
EVIDENCETHEORIESBELIEVED
DISCOVEREDOBSERVE
FACTS
SPACEEARTHMOON
PLANETROCKET
MARSORBIT
ASTRONAUTSFIRST
SPACECRAFTJUPITER
SATELLITESATELLITES
ATMOSPHERESPACESHIPSURFACE
SCIENTISTSASTRONAUT
SATURNMILES
ARTPAINT
ARTISTPAINTINGPAINTEDARTISTSMUSEUM
WORKPAINTINGS
STYLEPICTURES
WORKSOWN
SCULPTUREPAINTER
ARTSBEAUTIFUL
DESIGNSPORTRAITPAINTERS
STUDENTSTEACHERSTUDENT
TEACHERSTEACHING
CLASSCLASSROOM
SCHOOLLEARNING
PUPILSCONTENT
INSTRUCTIONTAUGHTGROUPGRADE
SHOULDGRADESCLASSES
PUPILGIVEN
BRAINNERVESENSE
SENSESARE
NERVOUSNERVES
BODYSMELLTASTETOUCH
MESSAGESIMPULSES
CORDORGANSSPINALFIBERS
SENSORYPAIN
IS
CURRENTELECTRICITY
ELECTRICCIRCUIT
ISELECTRICAL
VOLTAGEFLOW
BATTERYWIRE
WIRESSWITCH
CONNECTEDELECTRONSRESISTANCE
POWERCONDUCTORS
CIRCUITSTUBE
NEGATIVE
NATUREWORLDHUMAN
PHILOSOPHYMORAL
KNOWLEDGETHOUGHTREASONSENSEOUR
TRUTHNATURAL
EXISTENCEBEINGLIFE
MINDARISTOTLEBELIEVED
EXPERIENCEREALITY
A Selection of Topics
THIRDFIRST
SECONDTHREE
FOURTHFOUR
GRADETWO
FIFTHSEVENTH
SIXTHEIGHTH
HALFSEVEN
SIXSINGLENINTH
ENDTENTH
ANOTHER
![Page 43: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/43.jpg)
STORYSTORIES
TELLCHARACTER
CHARACTERSAUTHOR
READTOLD
SETTINGTALESPLOT
TELLINGSHORT
FICTIONACTION
TRUEEVENTSTELLSTALE
NOVEL
MINDWORLDDREAM
DREAMSTHOUGHT
IMAGINATIONMOMENT
THOUGHTSOWNREALLIFE
IMAGINESENSE
CONSCIOUSNESSSTRANGEFEELINGWHOLEBEINGMIGHTHOPE
WATERFISHSEA
SWIMSWIMMING
POOLLIKE
SHELLSHARKTANK
SHELLSSHARKSDIVING
DOLPHINSSWAMLONGSEALDIVE
DOLPHINUNDERWATER
DISEASEBACTERIADISEASES
GERMSFEVERCAUSE
CAUSEDSPREADVIRUSES
INFECTIONVIRUS
MICROORGANISMSPERSON
INFECTIOUSCOMMONCAUSING
SMALLPOXBODY
INFECTIONSCERTAIN
A Selection of Topics
FIELDMAGNETIC
MAGNETWIRE
NEEDLECURRENT
COILPOLESIRON
COMPASSLINESCORE
ELECTRICDIRECTION
FORCEMAGNETS
BEMAGNETISM
POLEINDUCED
SCIENCESTUDY
SCIENTISTSSCIENTIFIC
KNOWLEDGEWORK
RESEARCHCHEMISTRY
TECHNOLOGYMANY
MATHEMATICSBIOLOGY
FIELDPHYSICS
LABORATORYSTUDIESWORLD
SCIENTISTSTUDYINGSCIENCES
BALLGAMETEAM
FOOTBALLBASEBALLPLAYERS
PLAYFIELD
PLAYERBASKETBALL
COACHPLAYEDPLAYING
HITTENNISTEAMSGAMESSPORTS
BATTERRY
JOBWORKJOBS
CAREEREXPERIENCE
EMPLOYMENTOPPORTUNITIES
WORKINGTRAINING
SKILLSCAREERS
POSITIONSFIND
POSITIONFIELD
OCCUPATIONSREQUIRE
OPPORTUNITYEARNABLE
![Page 44: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/44.jpg)
STORYSTORIES
TELLCHARACTER
CHARACTERSAUTHOR
READTOLD
SETTINGTALESPLOT
TELLINGSHORT
FICTIONACTION
TRUEEVENTSTELLSTALE
NOVEL
MINDWORLDDREAM
DREAMSTHOUGHT
IMAGINATIONMOMENT
THOUGHTSOWNREALLIFE
IMAGINESENSE
CONSCIOUSNESSSTRANGEFEELINGWHOLEBEINGMIGHTHOPE
WATERFISHSEA
SWIMSWIMMING
POOLLIKE
SHELLSHARKTANK
SHELLSSHARKSDIVING
DOLPHINSSWAMLONGSEALDIVE
DOLPHINUNDERWATER
DISEASEBACTERIADISEASES
GERMSFEVERCAUSE
CAUSEDSPREADVIRUSES
INFECTIONVIRUS
MICROORGANISMSPERSON
INFECTIOUSCOMMONCAUSING
SMALLPOXBODY
INFECTIONSCERTAIN
A Selection of Topics
FIELDMAGNETIC
MAGNETWIRE
NEEDLECURRENT
COILPOLESIRON
COMPASSLINESCORE
ELECTRICDIRECTION
FORCEMAGNETS
BEMAGNETISM
POLEINDUCED
SCIENCESTUDY
SCIENTISTSSCIENTIFIC
KNOWLEDGEWORK
RESEARCHCHEMISTRY
TECHNOLOGYMANY
MATHEMATICSBIOLOGY
FIELDPHYSICS
LABORATORYSTUDIESWORLD
SCIENTISTSTUDYINGSCIENCES
BALLGAMETEAM
FOOTBALLBASEBALLPLAYERS
PLAYFIELD
PLAYERBASKETBALL
COACHPLAYEDPLAYING
HITTENNISTEAMSGAMESSPORTS
BATTERRY
JOBWORKJOBS
CAREEREXPERIENCE
EMPLOYMENTOPPORTUNITIES
WORKINGTRAINING
SKILLSCAREERS
POSITIONSFIND
POSITIONFIELD
OCCUPATIONSREQUIRE
OPPORTUNITYEARNABLE
![Page 45: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/45.jpg)
A Probabilistic Approach
• The function of semantic memory– The psychological problem of meaning
– One approach to meaning
• Solving the statistical problem of meaning– Maximum likelihood estimation
– Bayesian statistics
• Comparisons with Latent Semantic Analysis– Quantitative
– Qualitative
![Page 46: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/46.jpg)
Probabilistic Queries
• can be computed in different ways
• Fixed topic assumption:
• Multiple samples:
![Page 47: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/47.jpg)
Quantitative Comparisons
• Two types of task– general semantic tasks: dictionary, thesaurus– prediction of memory data
• All tests use LSA with 400 vectors, and a probabilistic model with 100 samples each using 500 topics
![Page 48: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/48.jpg)
Fill in the Blank
• 12856 sentences extracted from WordNet
• Overall performance– LSA gives median rank of 3393– Probabilistic model gives median rank of 3344
his cold deprived him of his sense of _silence broken by dogs barking _a _ hybrid accent
![Page 49: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/49.jpg)
Fill in the Blank
![Page 50: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/50.jpg)
Synonyms
• 280 sets of five synonyms from WordNet, ordered by number of senses
• Two tasks:– Predict first synonym– Predict last synonym
• Increasing number of synonyms
BREAK (78) EXPOSE (9) DISCOVER (8) DECLARE (7) REVEAL (3)
CUT (72) REDUCE (19) CONTRACT (12) SHORTEN (5) ABRIDGE (1)
RUN (53) GO (34) WORK (25) FUNCTION (9) OPERATE (7)
![Page 51: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/51.jpg)
First Synonym
![Page 52: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/52.jpg)
Last Synonym
![Page 53: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/53.jpg)
Synonyms and Word Frequency
![Page 54: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/54.jpg)
Synonyms and Word Frequency
Probabilistic
LSA
![Page 55: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/55.jpg)
Synonyms and Word Frequency
Probabilistic
LSA
![Page 56: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/56.jpg)
Word Frequency and Filling Blanks
Probabilistic LSA
![Page 57: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/57.jpg)
Performance on Semantic Tasks
• Performance comparable, neither great
• Difference in effects of word frequency due to treatment of co-occurrence data
• Probabilistic approach useful in addressing psychological data: frequency important
![Page 58: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/58.jpg)
Intrusions in Free Recall
• Intrusion rates from Deese (1959)
• Used average word vectors in LSA, P(word|list) in probabilistic model
• Favors LSA, since probabilistic combination can be multimodal
CHAIRFOODDESKTOPLEGEATCLOTHDISHWOODDINNERMARBLETENNIS
![Page 59: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/59.jpg)
Intrusions in Free Recall
![Page 60: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/60.jpg)
Intrusions in Free Recall
word frequencymodels
![Page 61: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/61.jpg)
Word Frequency is Not Enough
• An explanation needs to address two questions:– Why do these words intrude?– Why do other words not intrude?
![Page 62: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/62.jpg)
Word Frequency is Not Enough
• An explanation needs to address two questions:– Why do these words intrude?– Why do other words not intrude?
• Median word frequency rank: 1698.5
• Median rank in model: 21
![Page 63: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/63.jpg)
Word Association
• Word association norms from Nelson et al. (1998)
people
EARTH STARS SPACE
SUN MARS
UNIVERSE SATURN GALAXY
model
STARS STAR SUN
EARTH SPACE
SKY PLANET
UNIVERSE
PLANETS
associate number
12345678
![Page 64: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/64.jpg)
Word Association
![Page 65: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/65.jpg)
Performance on Memory Tasks
• Outperforms LSA on simple memory tasks, both far better at predicting memory data
• Improvement due to role of word frequency
• Not a complete account, but can form a part of more complex memory models
![Page 66: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/66.jpg)
Qualitative Comparisons
• Naturally deals with complications for LSA– Polysemy– Asymmetry
• Respects natural statistics of language
• Easily extends to other models of meaning
![Page 67: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/67.jpg)
Beyond the Bag of Words
z
w
zz
w w
![Page 68: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/68.jpg)
Beyond the Bag of Words
z
w
zz
w w
z
w
zz
w w
sss
![Page 69: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/69.jpg)
FOODFOODSBODY
NUTRIENTSDIETFAT
SUGARENERGY
MILKEATINGFRUITS
VEGETABLESWEIGHT
FATSNEEDS
CARBOHYDRATESVITAMINSCALORIESPROTEIN
MINERALS
MAPNORTHEARTHSOUTHPOLEMAPS
EQUATORWESTLINESEAST
AUSTRALIAGLOBEPOLES
HEMISPHERELATITUDE
PLACESLAND
WORLDCOMPASS
CONTINENTS
DOCTORPATIENTHEALTH
HOSPITALMEDICAL
CAREPATIENTS
NURSEDOCTORSMEDICINENURSING
TREATMENTNURSES
PHYSICIANHOSPITALS
DRSICK
ASSISTANTEMERGENCY
PRACTICE
BOOKBOOKS
READINGINFORMATION
LIBRARYREPORT
PAGETITLE
SUBJECTPAGESGUIDE
WORDSMATERIALARTICLE
ARTICLESWORDFACTS
AUTHORREFERENCE
NOTE
GOLDIRON
SILVERCOPPERMETAL
METALSSTEELCLAYLEADADAM
OREALUMINUM
MINERALMINE
STONEMINERALS
POTMININGMINERS
TIN
BEHAVIORSELF
INDIVIDUALPERSONALITY
RESPONSESOCIAL
EMOTIONALLEARNINGFEELINGS
PSYCHOLOGISTSINDIVIDUALS
PSYCHOLOGICALEXPERIENCES
ENVIRONMENTHUMAN
RESPONSESBEHAVIORSATTITUDES
PSYCHOLOGYPERSON
CELLSCELL
ORGANISMSALGAE
BACTERIAMICROSCOPEMEMBRANEORGANISM
FOODLIVINGFUNGIMOLD
MATERIALSNUCLEUSCELLED
STRUCTURESMATERIAL
STRUCTUREGREENMOLDS
Semantic categories
PLANTSPLANT
LEAVESSEEDSSOIL
ROOTSFLOWERS
WATERFOOD
GREENSEED
STEMSFLOWER
STEMLEAF
ANIMALSROOT
POLLENGROWING
GROW
![Page 70: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/70.jpg)
GOODSMALL
NEWIMPORTANT
GREATLITTLELARGE
*BIG
LONGHIGH
DIFFERENTSPECIAL
OLDSTRONGYOUNG
COMMONWHITESINGLE
CERTAIN
THEHIS
THEIRYOURHERITSMYOURTHIS
THESEA
ANTHATNEW
THOSEEACH
MRANYMRSALL
MORESUCHLESS
MUCHKNOWN
JUSTBETTERRATHER
GREATERHIGHERLARGERLONGERFASTER
EXACTLYSMALLER
SOMETHINGBIGGERFEWERLOWER
ALMOST
ONAT
INTOFROMWITH
THROUGHOVER
AROUNDAGAINSTACROSS
UPONTOWARDUNDERALONGNEAR
BEHINDOFF
ABOVEDOWN
BEFORE
SAIDASKED
THOUGHTTOLDSAYS
MEANSCALLEDCRIED
SHOWSANSWERED
TELLSREPLIED
SHOUTEDEXPLAINEDLAUGHED
MEANTWROTE
SHOWEDBELIEVED
WHISPERED
ONESOMEMANYTWOEACHALL
MOSTANY
THREETHIS
EVERYSEVERAL
FOURFIVEBOTHTENSIX
MUCHTWENTY
EIGHT
HEYOU
THEYI
SHEWEIT
PEOPLEEVERYONE
OTHERSSCIENTISTSSOMEONE
WHONOBODY
ONESOMETHING
ANYONEEVERYBODY
SOMETHEN
Syntactic categories
BEMAKE
GETHAVE
GOTAKE
DOFINDUSESEE
HELPKEEPGIVELOOKCOMEWORKMOVELIVEEAT
BECOME
![Page 71: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/71.jpg)
Sentence generationRESEARCH:[S] THE CHIEF WICKED SELECTION OF RESEARCH IN THE BIG MONTHS[S] EXPLANATIONS[S] IN THE PHYSICISTS EXPERIMENTS[S] HE MUST QUIT THE USE OF THE CONCLUSIONS[S] ASTRONOMY PEERED UPON YOUR SCIENTISTS DOOR[S] ANATOMY ESTABLISHED WITH PRINCIPLES EXPECTED IN BIOLOGY[S] ONCE BUT KNOWLEDGE MAY GROW[S] HE DECIDED THE MODERATE SCIENCE
LANGUAGE:[S] RESEARCHERS GIVE THE SPEECH[S] THE SOUND FEEL NO LISTENERS[S] WHICH WAS TO BE MEANING[S] HER VOCABULARIES STOPPED WORDS[S] HE EXPRESSLY WANTED THAT BETTER VOWEL
![Page 72: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/72.jpg)
Sentence generationLAW:[S] BUT THE CRIME HAD BEEN SEVERELY POLITE OR CONFUSED[S] CUSTODY ON ENFORCEMENT RIGHTS IS PLENTIFUL
CLOTHING:[S] WEALTHY COTTON PORTFOLIO WAS OUT OF ALL SMALL SUITS[S] HE IS CONNECTING SNEAKERS[S] THUS CLOTHING ARE THOSE OF CORDUROY[S] THE FIRST AMOUNTS OF FASHION IN THE SKIRT[S] GET TIGHT TO GET THE EXTENT OF THE BELTS[S] ANY WARDROBE CHOOSES TWO SHOES
THE ARTS:[S] SHE INFURIATED THE MUSIC[S] ACTORS WILL MANAGE FLOATING FOR JOY[S] THEY ARE A SCENE AWAY WITH MY THINKER[S] IT MEANS A CONCLUSION
![Page 73: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/73.jpg)
Conclusion
Taking a probabilistic approach can clarify some of the central issues in semantic representation
– Motivates sensitivity to co-occurrence statistics– Identifies how co-occurrence data should be used– Allows the role of meaning to be specified exactly,
and finds a meaningful decomposition of language
![Page 74: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/74.jpg)
![Page 75: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/75.jpg)
Probabilities and Inner Products
• Single word:
• List of words:
w
![Page 76: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/76.jpg)
![Page 77: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/77.jpg)
Model Selection
• How many topics does a language contain?
• Major issue for parametric models
• Not so much for non-parametric models– Dirichlet process mixtures– Expect more topics than tractable– Choice of number is choice of scale
![Page 78: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/78.jpg)
![Page 79: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/79.jpg)
![Page 80: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/80.jpg)
Gibbs Sampling and EM
• How many topics does a language contain?
• EM finds fixed set of topics, single estimate
• Sampling allows for multiple sets of topics, and multimodal posterior distributions
![Page 81: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/81.jpg)
![Page 82: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/82.jpg)
![Page 83: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/83.jpg)
Natural Statistics
• Treating co-occurrence data as frequencies preserves the natural statistics of language
• Word frequency
• Zipf’s Law of Meaning
![Page 84: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/84.jpg)
Natural Statistics
![Page 85: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/85.jpg)
Natural Statistics
![Page 86: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/86.jpg)
Natural Statistics
![Page 87: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/87.jpg)
![Page 88: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/88.jpg)
Word Association
people
KING JEWEL QUEEN HEAD HAT TOP
ROYAL THRONE
model
KING TEETH HAIR
TOOTH ENGLAND
MOUTH QUEEN PRINCE
CROWN
![Page 89: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/89.jpg)
Word Association
people
CHRISTMAS TOYS
LIE
model
MEXICO SPANISH
CALIFORNIA
SANTA
![Page 90: A Probabilistic Approach to Semantic Representation](https://reader036.vdocuments.net/reader036/viewer/2022081515/56813691550346895d9e1bf3/html5/thumbnails/90.jpg)