introduction to artificial intelligence
DESCRIPTION
INTRODUCTION TO ARTIFICIAL INTELLIGENCE. Massimo Poesio LECTURE 16: Unsupervised methods, IR, and lexical acquisition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . FEATURE-DEFINED DATA SPACE. . UNSUPERVISED MACHINE LEARNING. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/1.jpg)
INTRODUCTION TO ARTIFICIAL INTELLIGENCE
Massimo Poesio
LECTURE 16: Unsupervised methods, IR, and lexical acquisition
![Page 2: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/2.jpg)
2
.
FEATURE-DEFINED DATA SPACE
.. ..
. .. ..
..
....
.. ..
. .. ..
. ....
.
![Page 3: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/3.jpg)
UNSUPERVISED MACHINE LEARNING
• In many cases, what we want to learn is not a target function from examples to classes, but what the classes are– I.e., learn without being told
![Page 4: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/4.jpg)
EXAMPLE: TEXT CLASSIFICATION
• Consider clustering a large set of computer science documents
NLP
Graphics
AI
Theory
Arch.
![Page 5: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/5.jpg)
6
CLUSTERING• Partition unlabeled examples into disjoint
subsets of clusters, such that:– Examples within a cluster are very similar– Examples in different clusters are very different
• Discover new categories in an unsupervised manner (no sample category labels provided).
![Page 6: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/6.jpg)
Deciding what a new doc is about
• Check which region the new doc falls into– can output “softer” decisions as well.
NLP
Graphics
AI
Theory
Arch.
= AI
![Page 7: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/7.jpg)
8
Hierarchical Clustering
• Build a tree-based hierarchical taxonomy (dendrogram) from a set of unlabeled examples.
• Recursive application of a standard clustering algorithm can produce a hierarchical clustering.
animal
vertebrate
fish reptile amphib. mammal worm insect crustacean
invertebrate
![Page 8: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/8.jpg)
9
Agglomerative vs. Divisive Clustering
• Agglomerative (bottom-up) methods start with each example in its own cluster and iteratively combine them to form larger and larger clusters.
• Divisive (partitional, top-down) separate all examples immediately into clusters.
![Page 9: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/9.jpg)
10
Direct Clustering Method• Direct clustering methods require a
specification of the number of clusters, k, desired.
• A clustering evaluation function assigns a real-value quality measure to a clustering.
• The number of clusters can be determined automatically by explicitly generating clusterings for multiple values of k and choosing the best result according to a clustering evaluation function.
![Page 10: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/10.jpg)
11
Hierarchical Agglomerative Clustering (HAC)
• Assumes a similarity function for determining the similarity of two instances.
• Starts with all instances in a separate cluster and then repeatedly joins the two clusters that are most similar until there is only one cluster.
• The history of merging forms a binary tree or hierarchy.
![Page 11: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/11.jpg)
12
Cluster Similarity• Assume a similarity function that determines the
similarity of two instances: sim(x,y).– Cosine similarity of document vectors.
• How to compute similarity of two clusters each possibly containing multiple instances?– Single Link: Similarity of two most similar members.– Complete Link: Similarity of two least similar members.– Group Average: Average similarity between members.
![Page 12: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/12.jpg)
13
Non-Hierarchical Clustering
• Typically must provide the number of desired clusters, k.
• Randomly choose k instances as seeds, one per cluster.
• Form initial clusters based on these seeds.• Iterate, repeatedly reallocating instances to different
clusters to improve the overall clustering.• Stop when clustering converges or after a fixed
number of iterations.
![Page 13: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/13.jpg)
CLUSTERING METHODS IN NLP
• Unsupervised techniques are heavily used in :– Text classification– Information retrieval – Lexical acquisition
![Page 14: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/14.jpg)
CLUSTERING METHODS IN NLP
• Unsupervised techniques are heavily used in :– Text classification– Information retrieval – Lexical acquisition
![Page 15: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/15.jpg)
2004/05 ANLE 16
Feature-based lexical semantics
• Very old idea in lexical semantics: the meaning of a word can be specified in terms of the values of certain `features’ (`DECOMPOSITIONAL SEMANTICS’)– dog : ANIMATE= +, EAT=MEAT, SOCIAL=+– horse : ANIMATE= +, EAT=GRASS, SOCIAL=+– cat : ANIMATE= +, EAT=MEAT, SOCIAL=-
![Page 16: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/16.jpg)
Feb 21st Cog/Comp Neuroscience 17
FEATURE-BASED REPRESENTATIONS IN PSYCHOLOGY
• Feature-based concept representations assumed by many cognitive psychology theories (Smith and Medin, 1981, McRae et al, 1997)
• Underpin development of prototype theory (Rosch et al)• Used, e.g., to account for semantic priming (McRae et al,
1997; Plaut, 1995)• Underlie much work on category-specific defects (Warrington
and Shallice, 1984; Caramazza and Shelton, 1998; Tyler et al, 2000; Vinson and Vigliocco, 2004)
![Page 17: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/17.jpg)
Feb 21st Cog/Comp Neuroscience 18
SPEAKER-GENERATED FEATURES (VINSON AND VIGLIOCCO)
![Page 18: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/18.jpg)
2004/05 ANLE 19
Vector-based lexical semantics
• If we think of the features as DIMENSIONS we can view these meanings as VECTORS in a FEATURE SPACE– (An idea introduced by Salton in Information
Retrieval, see below)
![Page 19: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/19.jpg)
2004/05 ANLE 20
Vector-based lexical semantics
DOG
CAT
HORSE
![Page 20: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/20.jpg)
2004/05 ANLE 21
General characterization of vector-based semantics (from Charniak)
• Vectors as models of concepts• The CLUSTERING approach to lexical semantics:
1. Define properties one cares about, and give values to each property (generally, numerical)
2. Create a vector of length n for each item to be classified3. Viewing the n-dimensional vector as a point in n-space,
cluster points that are near one another• What changes between models:
1. The properties used in the vector2. The distance metric used to decide if two points are
`close’3. The algorithm used to cluster
![Page 21: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/21.jpg)
2004/05 ANLE 22
Using words as features in a vector-based semantics
• The old decompositional semantics approach requires i. Specifying the featuresii. Characterizing the value of these features for each lexeme
• Simpler approach: use as features the WORDS that occur in the proximity of that word / lexical entry– Intuition: “You can tell a word’s meaning from the company it keeps”
• More specifically, you can use as `values’ of these features – The FREQUENCIES with which these words occur near the words whose meaning we
are defining– Or perhaps the PROBABILITIES that these words occur next to each other
• Alternative: use the DOCUMENTS in which these words occur (e.g., LSA)• Some psychological results support this view. Lund, Burgess, et al (1995, 1997):
lexical associations learned this way correlate very well with priming experiments. Landauer et al: good correlation on a variety of topics, including human categorization & vocabulary tests.
![Page 22: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/22.jpg)
Using neighboring words to specify lexical meanings
![Page 23: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/23.jpg)
Learning the meaning of DOG from text
![Page 24: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/24.jpg)
Learning the meaning of DOG from text
![Page 25: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/25.jpg)
Learning the meaning of DOG from text
![Page 26: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/26.jpg)
Learning the meaning of DOG from text
![Page 27: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/27.jpg)
Learning the meaning of DOG from text
![Page 28: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/28.jpg)
The lexicon we acquire
![Page 29: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/29.jpg)
Meanings in word space
![Page 30: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/30.jpg)
2004/05 ANLE 32
Acquiring lexical vectors from a corpus(Schuetze, 1991; Burgess and Lund, 1997)• To construct vectors C(w) for each word w:
1. Scan a text2. Whenever a word w is encountered, increment all cells of
C(w) corresponding to the words v that occur in the vicinity of w, typically within a window of fixed size
• Differences among methods:– Size of window– Weighted or not– Whether every word in the vocabulary counts as a
dimension (including function words such as the or and) or whether instead only some specially chosen words are used (typically, the m most common content words in the corpus; or perhaps modifiers only). The words chosen as dimensions are often called CONTEXT WORDS
– Whether dimensionality reduction methods are applied
![Page 31: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/31.jpg)
2004/05 ANLE 33
Variant: using probabilities (e.g., Dagan et al, 1997)
• E.g., for house
• Context vector (using probabilities)– 0.001394 0.016212 0.003169 0.000734 0.001460 0.002901 0.004725 0.000598 0
0 0.008993 0.008322 0.000164 0.010771 0.012098 0.002799 0.002064 0.007697 0 0 0.001693 0.000624 0.001624 0.000458 0.002449 0.002732 0 0.008483 0.007929 0 0.001101 0.001806 0 0.005537 0.000726 0.011563 0.010487 0 0.001809 0.010601 0.000348 0.000759 0.000807 0.000302 0.002331 0.002715 0.020845 0.000860 0.000497 0.002317 0.003938 0.001505 0.035262 0.002090 0.004811 0.001248 0.000920 0.001164 0.003577 0.001337 0.000259 0.002470 0.001793 0.003582 0.005228 0.008356 0.005771 0.001810 0 0.001127 0.001225 0 0.008904 0.001544 0.003223 0
![Page 32: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/32.jpg)
2004/05 ANLE 34
Variant: using modifiers to specify the meaning of words
• …. The Soviet cosmonaut …. The American astronaut …. The red American car …. The old red truck … the spacewalking cosmonaut … the full Moon …
cosmonaut
astronaut moon
car truck
Soviet 1 0 0 1 1American 0 1 0 1 1spacewalking
1 1 0 0 0
red 0 0 0 1 1full 0 0 1 0 0old 0 0 0 1 1
![Page 33: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/33.jpg)
2004/05 ANLE 35
Another variant: word / document matrices
d1 d2 d3 d4 d5 d6
cosmonaut 1 0 1 0 0 0astronaut 0 1 0 0 0 0moon 1 1 0 0 0 0car 1 0 0 1 1 0truck 0 0 0 1 0 1
![Page 34: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/34.jpg)
2004/05 ANLE 36
Measures of semantic similarity• Euclidean distance:
• Cosine:
• Manhattan Metric:
n
i ii yxd1
n
i in
i i
n
i ii
yx
yx
12
12
1)cos(
n
i ii yxd1
2
![Page 35: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/35.jpg)
SIMILARITY IN VECTOR SPACE MODELS: THE COSINE MEASURE
kj
kj
qdqd *
cos
θ
dj
qk
N
i ijN
i ik
N
iijik
jkww
wwdqsim
12,1
2,
1,,
,
![Page 36: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/36.jpg)
EVALUATION
• Synonymy identification• Text coherence• Semantic priming
![Page 37: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/37.jpg)
SYNONYMY: THE TOEFL TEST
![Page 38: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/38.jpg)
TOEFL TEST: RESULTS
![Page 39: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/39.jpg)
2004/05 ANLE 41
Some psychological evidence for vector-space representations
• Burgess and Lund (1996, 1997): the clusters found with HAL correlate well with those observed using semantic priming experiments.
• Landauer, Foltz, and Laham (1997): scores overlap with those of humans on standard vocabulary and topic tests; mimic human scores on category judgments; etc.
• Evidence about `prototype theory’ (Rosch et al, 1976)– Posner and Keel, 1968
• subjects presented with patterns of dots that had been obtained by variations from single pattern (`prototype’)
• Later, they recalled prototypes better than samples they had actually seen
– Rosch et al, 1976: `basic level’ categories (apple, orange, potato, carrot) have higher `cue validity’ than elements higher in the hierarchy (fruit, vegetable) or lower (red delicious, cox)
![Page 40: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/40.jpg)
2004/05 ANLE 42
The HAL model (Burgess and Lund, 1995, 1996, 1997)• A 160 million words corpus of articles
extracted from all newsgroups containing English dialogue
• Context words: the 70,000 most frequently occurring symbols within the corpus
• Window size: 10 words to the left and the right of the word
• Measure of similarity: cosine
![Page 41: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/41.jpg)
HAL AND SEMANTIC PRIMING
![Page 42: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/42.jpg)
INFORMATION RETRIEVAL
• GOAL: Find the documents most relevant to a certain QUERY
• Latest development: WEB SEARCH– Use the Web as the collection of documents
• Related: – QUESTION-ANSWERING– DOCUMENT CLASSIFICATION
![Page 43: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/43.jpg)
DOCUMENTS AS BAGS OF WORDS
broad tech stock rally may signal trend - traders.
technology stocks rallied on tuesday, with gains scored broadly across many sectors, amid what some traders called a recovery from recent doldrums.
broadmay rallyralliedsignal stockstocks techtechnology traderstraders trend
DOCUMENTINDEX
![Page 44: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/44.jpg)
THE VECTOR SPACE MODEL
• Query and documents represented as vectors of index terms, assigned non-binary WEIGHTS
• Similarity calculated using vector algebra: COSINE (cfr. lexical similarity models)– RANKED similarity
• Most popular of all models (cfr. Salton and Lesk’s SMART)
![Page 45: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/45.jpg)
TERM WEIGHTING IN VECTOR SPACE MODELS: THE TF.IDF MEASURE
ikiki df
Nftfidf log*,,
FREQUENCY of term i in document k Number of documents
with term i
![Page 46: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/46.jpg)
2004/05 ANLE 49
VECTOR-SPACE MODELS WITH SYNTACTIC INFORMATION
• Pereira and Tishby, 1992: two words are similar if they occur as objects of the same verbs– John ate POPCORN– John ate BANANAS
• C(w) is the distribution of verbs for which w served as direct object.– First approximation: just counts– In fact: probabilities
• Similarity: RELATIVE ENTROPY
![Page 47: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/47.jpg)
Feb 21st Cog/Comp Neuroscience 50
(SYNTACTIC) RELATION-BASED VECTOR MODELS
attackedattacked foxfox dogdog
<subj,fox><subj,fox> <det,the><det,the> <det,the><det,the>
<obj,dog><obj,dog> <mod,red><mod,red> <mod,lazy><mod,lazy>
attacked
fox dog
the red the lazy
subj obj
det detmod mod
E.g., Grefenstette, 1994; Lin, 1998; Curran and Moens, 2002
![Page 48: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/48.jpg)
2004/05 ANLE 51
SEXTANT (Grefenstette, 1992)
It was concluded that the carcinoembryonic antigens represent cellular constituents which are repressed during the course of differentiation the normal digestive system epithelium and reappear in the corresponding malignant cells by a process of derepressive dedifferentiation
antigen carcinoembryonic-ADJantigen repress-DOBJantigen represent-SUBJconstituent cellular-ADJconstituent represent-DOBJcourse repress-IOBJ……..
![Page 49: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/49.jpg)
2004/05 ANLE 52
SEXTANT: Similarity measure
dog pet-DOBJdog eat-SUBJ dog shaggy-ADJdog brown-ADJdog leash-NN
cat pet-DOBJcat pet-DOBJ cat hairy-ADJcat leash-NN
CATDOG
B andA by possessed attributes Unique
B andA by shared AttributesCount
CountJaccard:
6
2ADJ}-shaggyDOBJ,-petNN,-leashADJ,-hairySUBJ,-eatADJ,-{brown
DOBJ}-pet NN,-{leash
CountCount
![Page 50: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/50.jpg)
MULTIDIMENSIONAL SCALING
• Many models (included HAL) apply techniques for REDUCING the number of dimensions
• Intuition: many features express a similar property / topic
![Page 51: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/51.jpg)
MULTIDIMENSIONAL SCALING
![Page 52: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/52.jpg)
2004/05 ANLE 55
Latent Semantic Analysis (LSA) (Landauer et al, 1997)
• Goal: extract relatons of expected contextual usage from passages
• Two steps:1. Build a word / document cooccurrence matrix2. `Weigh’ each cell 3. Perform a DIMENSIONALITY REDUCTION
• Argued to correlate well with humans on a number of tests
![Page 53: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/53.jpg)
2004/05 ANLE 56
LSA: the method, 1
![Page 54: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/54.jpg)
2004/05 ANLE 57
LSA: Singular Value Decomposition
![Page 55: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/55.jpg)
2004/05 ANLE 58
LSA: Reconstructed matrix
![Page 56: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/56.jpg)
2004/05 ANLE 59
Topic correlations in `raw’ and `reconstructed’ data
![Page 57: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/57.jpg)
2004/05 ANLE 60
Some caveats• Two senses of `similarity’
– Schuetze: two words are similar if one can replace the other– Brown et al: two words are similar if they occur in similar
contexts• What notion of `meaning’ is learned here?
– “One might consider LSA’s maximal knowledge of the world to be analogous to a well-read nun’s knowledge of sex, a level of knowledge often deemed a sufficient basis for advising the young” (Landauer et al, 1997)
• Can one do semantics with these representations?– Our own experience: using HAL-style vectors for resolving
bridging references– Very limited success– Applying dimensionality reduction didn’t seem to help
![Page 58: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/58.jpg)
REMAINING LECTURES
DAY HOUR TOPICWed 25/11 12-14 Text classification with
Artificial Neural Nets
Tue 1/12 10-12 Lab: Supervised ML with Weka
Fri 4/12 10-12 Unsupervised methods & their application in lexical acq and IR
Wed 9/12 10-12 Lexical acquisition by clustering
Thu 10/12 10-12 Psychological evidence on learning
Fri 11/12 10-12 Psychological evidence on language processing
Mon 14/12 10-12 Intro to NLP
![Page 59: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/59.jpg)
REMAINING LECTURES
DAY HOUR TOPICTue 15/12 10-12 Machine learning for
anaphora
Tue 15/12 14-16 Lab: Clustering
Wed 16/12 14-16 Lab: BART
Thu 17/12 10-12 Ling. & psychological evidence on anaphora
Fri 18/12 10-12 Corpora for anaphora
Mon 21/12 10-12 Lexical & commons. knowledge for anaphora
Tue 22/12 10-12 Salience
Tue 22/12 14-16 Discourse new detection
![Page 60: INTRODUCTION TO ARTIFICIAL INTELLIGENCE](https://reader035.vdocuments.net/reader035/viewer/2022062316/56816792550346895ddcc88d/html5/thumbnails/60.jpg)
ACKNOWLEDGMENTS
• Some of the slides come from – Ray Mooney’s Utexas AI course– Marco Baroni