lecture 21 computational lexical semantics topics features in nltk iii computational lexical...

Lecture 21Computational Lexical Semantics

Lecture 21Computational Lexical Semantics

Topics Topics Features in NLTK III Computational Lexical Semantics Semantic Web USC

Readings:Readings: NLTK book Chapter 10

Text Chapters 20

April 3, 2013

CSCE 771 Natural Language Processing

– 2 – CSCE 771 Spring 2013

OverviewOverviewLast Time (Programming)Last Time (Programming)

Features in NLTK NL queries SQL NLTK support for Interpretations and Models Propositional and predicate logic support Prover9

TodayToday Last Lectures slides 25-29 Features in NLTK Computational Lexical Semantics

Readings: Readings: Text 19,20 NLTK Book: Chapter 10

Next Time: Computational Lexical Semantics IINext Time: Computational Lexical Semantics II

– 3 – CSCE 771 Spring 2013

Model Building in NLTK - Chapter 10continuedModel Building in NLTK - Chapter 10continued

Mace model builderMace model builder

lp = nltk.LogicParser()lp = nltk.LogicParser()

# install Mace4# install Mace4

config_mace4('c:\Python26\Lib\site-packages\prover9')config_mace4('c:\Python26\Lib\site-packages\prover9')

a3 = lp.parse('exists x.(man(x) & walks(x))')a3 = lp.parse('exists x.(man(x) & walks(x))')

c1 = lp.parse('mortal(socrates)')c1 = lp.parse('mortal(socrates)')

c2 = lp.parse('-mortal(socrates)')c2 = lp.parse('-mortal(socrates)')

mb = nltk.Mace(5)mb = nltk.Mace(5)

print mb.build_model(None, [a3, c1])print mb.build_model(None, [a3, c1])

TrueTrue

print mb.build_model(None, [a3, c2])print mb.build_model(None, [a3, c2])

TrueTrue

print mb.build_model(None, [c1, c2])print mb.build_model(None, [c1, c2])

FalseFalse

– 4 – CSCE 771 Spring 2013

>>> a4 = lp.parse('exists y. (woman(y) & all x. (man(x) -> >>> a4 = lp.parse('exists y. (woman(y) & all x. (man(x) -> love(x,y)))')love(x,y)))')

>>> a5 = lp.parse('man(adam)')>>> a5 = lp.parse('man(adam)')

>>> a6 = lp.parse('woman(eve)')>>> a6 = lp.parse('woman(eve)')

>>> g = lp.parse('love(adam,eve)')>>> g = lp.parse('love(adam,eve)')

>>> mc = nltk.MaceCommand(g, assumptions=[a4, a5, a6])>>> mc = nltk.MaceCommand(g, assumptions=[a4, a5, a6])

>>> mc.build_model()>>> mc.build_model()

TrueTrue

– 5 – CSCE 771 Spring 2013

10.4   The Semantics of English Sentences10.4   The Semantics of English SentencesPrinciple of compositionality --Principle of compositionality --

– 6 – CSCE 771 Spring 2013

Representing the λ-Calculus in NLTKRepresenting the λ-Calculus in NLTK

(33)(33)

a.a. (walk(x) chew_gum(x))∧(walk(x) chew_gum(x))∧

b.b. λλx.(walk(x) chew_gum(x))∧x.(walk(x) chew_gum(x))∧

c.c. \x.(walk(x) & chew_gum(x)) -- \x.(walk(x) & chew_gum(x)) -- the NLTK way!the NLTK way!

– 7 – CSCE 771 Spring 2013

Lambda0.pyLambda0.py

import nltkimport nltk

from nltk import load_parserfrom nltk import load_parser

lp = nltk.LogicParser()lp = nltk.LogicParser()

e = lp.parse(r'\x.(walk(x) & chew_gum(x))')e = lp.parse(r'\x.(walk(x) & chew_gum(x))')

print eprint e

\x.(walk(x) & chew_gum(x))\x.(walk(x) & chew_gum(x))

e.free()e.free()

print lp.parse(r'\x.(walk(x) & chew_gum(y))')print lp.parse(r'\x.(walk(x) & chew_gum(y))')

\x.(walk(x) & chew_gum(y))\x.(walk(x) & chew_gum(y))

– 8 – CSCE 771 Spring 2013

Simple β-reductionsSimple β-reductions

>>> e = lp.parse(r'\x.(walk(x) & chew_gum(x))(gerald)')>>> e = lp.parse(r'\x.(walk(x) & chew_gum(x))(gerald)')

>>> print e>>> print e

\x.(walk(x) & chew_gum(x))(gerald)\x.(walk(x) & chew_gum(x))(gerald)

>>> print e.simplify() [1]>>> print e.simplify() [1]

(walk(gerald) & chew_gum(gerald))(walk(gerald) & chew_gum(gerald))

– 9 – CSCE 771 Spring 2013

Predicate reductionsPredicate reductions

>>> e3 = lp.parse('\P.exists x.P(x)(\y.see(y, x))')>>> e3 = lp.parse('\P.exists x.P(x)(\y.see(y, x))')

>>> print e3>>> print e3

(\P.exists x.P(x))(\y.see(y,x))(\P.exists x.P(x))(\y.see(y,x))

>>> print e3.simplify()>>> print e3.simplify()

exists z1.see(z1,x)exists z1.see(z1,x)

– 10 – CSCE 771 Spring 2013

Figure 19.7 Inheritance of PropertiesFigure 19.7 Inheritance of Properties

Exists e,x,y Eating(e) ^ Agent(e, x) ^ Theme(e, y)Exists e,x,y Eating(e) ^ Agent(e, x) ^ Theme(e, y)

““hamburger edible?” from wordnethamburger edible?” from wordnet

Copyright ©2009 by Pearson Education, Inc.Upper Saddle River, New Jersey 07458

All rights reserved.

Speech and Language Processing, Second EditionDaniel Jurafsky and James H. Martin

– 11 – CSCE 771 Spring 2013

Figure 20.1 Possible sense tags for bassFigure 20.1 Possible sense tags for bassChapter 20 – Word Sense disambiguation (WSD)Chapter 20 – Word Sense disambiguation (WSD)

Machine translationMachine translation

Supervised vs unsupervised learningSupervised vs unsupervised learning

Semantic concordance – corpus with words tagged Semantic concordance – corpus with words tagged with sense tags with sense tags

– 12 – CSCE 771 Spring 2013

Feature Extraction for WSDFeature Extraction for WSD

Feature vectorsFeature vectors

CollocationCollocation

[w[wi-2i-2, POS, POSi-2i-2, w, wi-1i-1, POS, POSi-1i-1, w, wii, POS, POSii, w, wi+1i+1, POS, POSi+1i+1, w, wi+2i+2, POS, POSi+2i+2]]

Bag-of-words – unordered set of neighboring wordsBag-of-words – unordered set of neighboring words

Represent sets of most frequent content words with Represent sets of most frequent content words with membership vectormembership vector

[0,0,1,0,0,0,1] – set of 3[0,0,1,0,0,0,1] – set of 3rdrd and 7 and 7thth most freq. content word most freq. content word

Window of nearby words/featuresWindow of nearby words/features

– 13 – CSCE 771 Spring 2013

Naïve Bayes ClassifierNaïve Bayes Classifier

w – word vectorw – word vector

s – sense tag vectors – sense tag vector

f – feature vector [wf – feature vector [wii, POS, POSii ] for i=1, …n ] for i=1, …n

Approximate by frequency countsApproximate by frequency counts

But how practical?But how practical?

)|(maxarg^

fsPsSs

– 14 – CSCE 771 Spring 2013

Looking for Practical formulaLooking for Practical formula

..

Still not practicalStill not practical

)(

)()|(maxarg

)|(maxarg^

fP

sPsfP

fsPs

Ss

Ss

– 15 – CSCE 771 Spring 2013

Naïve == Assume IndependenceNaïve == Assume Independence

n

jj sfPsfP

1

)|()|(

Now practical, but realistic?Now practical, but realistic?

n

jj

SssfPs

1

^

)|(maxarg

– 16 – CSCE 771 Spring 2013

Training = count frequenciesTraining = count frequencies

..

Maximum likelihood estimator (20.8)Maximum likelihood estimator (20.8)

)(

),()|(

scount

sfcountsfP jij

)(

),()(

j

jii wcount

wscountsP

– 17 – CSCE 771 Spring 2013

Decision List ClassifiersDecision List Classifiers

Naïve Bayes Naïve Bayes hard for humans to examine decisions hard for humans to examine decisions and understandand understand

Decision list classifiers - like “case” statementDecision list classifiers - like “case” statement

sequence of (test, returned-sense-tag) pairssequence of (test, returned-sense-tag) pairs

– 18 – CSCE 771 Spring 2013

Figure 20.2 Decision List Classifier RulesFigure 20.2 Decision List Classifier Rules

– 19 – CSCE 771 Spring 2013

WSD Evaluation, baselines, ceilingsWSD Evaluation, baselines, ceilings

Extrinsic evaluation - evaluating embedded NLP in Extrinsic evaluation - evaluating embedded NLP in end-to-end applications (in vivo)end-to-end applications (in vivo)

Intrinsic evaluation – WSD evaluating by itself (in vitro)Intrinsic evaluation – WSD evaluating by itself (in vitro)

Sense accuracy Sense accuracy

Corpora – SemCor, SENSEVAL, SEMEVALCorpora – SemCor, SENSEVAL, SEMEVAL

Baseline - Most frequent sense (wordnet sense 1)Baseline - Most frequent sense (wordnet sense 1)

Ceiling – Gold standard – human experts with Ceiling – Gold standard – human experts with discussion and agreement discussion and agreement

– 20 – CSCE 771 Spring 2013

Figure 20.3 Simplified Lesk AlgorithmFigure 20.3 Simplified Lesk Algorithm

gloss/sentence overlap

– 21 – CSCE 771 Spring 2013

Simplified Lesk exampleSimplified Lesk example

The bank can guarantee deposits will eventually cover The bank can guarantee deposits will eventually cover future tuition costs because it invests in adjustable future tuition costs because it invests in adjustable rate mortgage securities.rate mortgage securities.

– 22 – CSCE 771 Spring 2013

SENSEVAL competitions SENSEVAL competitions

http://www.senseval.org/

Check the Senseval-3 website.

– 23 – CSCE 771 Spring 2013

Corpus LeskCorpus Lesk

weights applied to overlap wordsweights applied to overlap words

inverse document frequencyinverse document frequency

idfidfii = log (N = log (Ndocsdocs / num docs containing w / num docs containing wii))

– 24 – CSCE 771 Spring 2013

20.4.2 Selectional Restrictions and Preferences20.4.2 Selectional Restrictions and Preferences

– 25 – CSCE 771 Spring 2013

Wordnet Semantic classes of ObjectsWordnet Semantic classes of Objects

– 26 – CSCE 771 Spring 2013

Minimally Supervised WSD: BootstrappingMinimally Supervised WSD: Bootstrapping

Yarowsky algorithmYarowsky algorithm

Heuritics:Heuritics:

1.1. one sense per collocationsone sense per collocations

2.2. one sense per discourseone sense per discourse

– 27 – CSCE 771 Spring 2013

Figure 20.4 Two senses of plantFigure 20.4 Two senses of plant

– 28 – CSCE 771 Spring 2013

Figure 20.5Figure 20.5

– 29 – CSCE 771 Spring 2013

Figure 20.6 Path Based SimilarityFigure 20.6 Path Based Similarity

– 30 – CSCE 771 Spring 2013

Figure 20.6 Path Based SimilarityFigure 20.6 Path Based Similarity

..

\\

simsimpathpath(c(c11, c, c22)= 1/pathlen(c)= 1/pathlen(c11, c, c22) (length + 1)) (length + 1)

– 31 – CSCE 771 Spring 2013

Information Content word similarityInformation Content word similarity

– 32 – CSCE 771 Spring 2013

Figure 20.7 Wordnet with P(c) valuesFigure 20.7 Wordnet with P(c) values

– 33 – CSCE 771 Spring 2013

Figure 20.8Figure 20.8

– 34 – CSCE 771 Spring 2013

– 35 – CSCE 771 Spring 2013

Figure 20.9Figure 20.9

– 36 – CSCE 771 Spring 2013

Figure 20.10Figure 20.10

– 37 – CSCE 771 Spring 2013

Figure 20.11Figure 20.11

– 38 – CSCE 771 Spring 2013

Figure 20.12Figure 20.12

– 39 – CSCE 771 Spring 2013

Figure 20.13Figure 20.13

– 40 – CSCE 771 Spring 2013

Figure 20.14Figure 20.14

– 41 – CSCE 771 Spring 2013

Figure 20.15Figure 20.15

– 42 – CSCE 771 Spring 2013

Figure 20.16Figure 20.16