CS388:NaturalLanguageProcessingLecture13:Seman8csI

GregDurrett

SlidesadaptedfromDanKlein,UCBerkeley

Administrivia‣Mini2due*today*at5pm

Recall:Dependencies

DT NNTOVBDDT NNthe housetoranthe dog

‣ Dependencysyntax:syntac8cstructureisdefinedbydependencies‣ Head(parent,governor)connectedtodependent(child,modifier)‣ EachwordhasexactlyoneparentexceptfortheROOTsymbol‣ Dependenciesmustformadirectedacyclicgraph

ROOT

Recall:ShiX-ReduceParsing

IatesomespagheZbolognese

ROOT

‣ State:Stack:[ROOTIate]Buffer:[somespagheZbolognese]

‣ LeX-arc(reduceopera8on):Letdenotethestack�‣ “Poptwoelements,addanarc,putthembackonthestack”

‣ State:Stack:[ROOTate]Buffer:[somespagheZbolognese]

I

�|w�2, w�1 ! �|w�1 w�1w�2 isnowachildof,

Wherearewenow?‣ Earlyintheclass:sentencesarejustsequencesofwords

‣Whyisthisuseful?Whatdoesthisallowustodo?

‣ Nowwecanunderstandthemintermsoftreestructuresaswell

‣We’regoingtoseehowparsingcanbeasteppingstonetowardsmoreformalrepresenta8onsoflanguagemeaning

Today‣ First-orderlogic

‣ CCGparsingfordatabasequeries

‣ Composi8onalseman8cswithfirst-orderlogic

‣ Lambda-DCSforques8onanswering

First-OrderLogic

First-orderLogic

‣ singsisapredicate(withoneargument),func8onf:en8ty=>true/false

‣ Powerfullogicformalismincludingthingslikeen88es,rela8ons,andquan8fica8ons

‣ [[sings]]=denota,on,setofen88eswhichsing(sortoflikeexecu8ngthispredicateontheworld—we’llcomebacktothis)

‣ Proposi8ons:leta=Itisday,b=Itisnight‣ a∨b=eitheraistrueorbistrue,a=>¬b=aimpliesnotb

‣Morecomplexstatements:“LadyGagasings”

‣ sings(LadyGaga)=trueorfalse,havetoexecutethisagainstsomedatabase(calledaworld)

Quan8fica8on

‣ ∀xsings(x)∨dances(x)=>performs(x)

‣ ∃y∀xfriend(x,y)

‣ Universalquan8fica8on:“forall”operator

‣ Existen8alquan8fica8on:“thereexists”operator

‣ ∀x∃yfriend(x,y)‣ Sourceofambiguity!“Everyoneisfriendswithsomeone”

“Everyonewhosingsordancesperforms”

“Someonesings”‣ ∃xsings(x)

LogicinNLP‣ Ques8onanswering:

‣ Informa8onextrac8on: LadyGagaandEminemarebothmusicians

∀xmusician(x)=>performer(x)musician(LadyGaga)∧musician(Eminem)

Then:performer(LadyGaga)∧performer(Eminem)

WhoarealltheAmericansingersnamedAmy?

λx.na8onality(x,USA)∧sings(x)∧firstName(x,Amy)

‣ Func8onthatmapsfromxtotrue/false,likefilter.Executethisontheworldtoanswertheques8on

‣ Cannowdoreasoning.Maybeknow:

‣ Lambdacalculus:powerfulsystemforexpressingthesefunc8ons

Composi8onalSeman8cswithFirst-OrderLogic

Truth-Condi8onalSeman8csId Name Alias Birthdate Sings?

e470 StefaniGermanooa LadyGaga 3/28/1986 Te728 MarshallMathers Eminem 10/17/1972 T

‣ Databasecontainingen88es,predicates,etc.

‣ Truth-condi8onalseman8cs:sentenceexpressessomethingabouttheworldwhichiseithertrueorfalse

NP VP

NNP NNP

S

VBPLadyGaga sings

‣ Denota8on:evalua8onofsomeexpressionagainstthisdatabase

‣[[LadyGaga]] = e470denota8onofthisstringisanen8ty

‣[[sings(e470)]] = True

denota8onofthisexpressionisT/F

ParsestoLogicalForms

NP VP

NNP NNP

S

VBPLadyGaga sings

e470

λy. sings(y)takesoneargument(y,theen8ty)andreturnsalogicalformsings(y)

λy. sings(y)

sings(e470)

‣Wecanusethesyntac8cparseasabridgetothelambda-calculusrepresenta8on,buildupalogicalformcomposi,onally

func8onapplica8on:applythistoe470ID

ParsestoLogicalForms

NP

VPNNP NNP

S

VBPLadyGaga

sings

e470

λy. sings(y)

sings(e470) ∧ dances(e470)

VP

CC VP

VBPdancesλy. dances(y)

and

VP:λy.a(y)∧b(y)->VP:λy.a(y)CCVP:λy.b(y)

λy. sings(y) ∧ dances(y)

‣ Generalrules:S:f(x)->NP:xVP:f

ParsestoLogicalForms

NP

NNP NNP

S

VBDLadyGaga was

e470

λx.λy. born(y,x)

born(e470,3/28/1986)

VP

NP

March28,1986born

λy. born(y, 3/28/1986)

VBN

VP

λy. born(y, 3/28/1986)

‣ Howtohandletense:shouldweindicatethatthishappenedinthepast?‣ Func8ontakestwoarguments:firstx(date),theny(en8ty)

3/28/1986

Trickythings‣ Adverbs/temporality:LadyGagasangwellyesterday

∃e. type(e,sing) ∧ agent(e,e470) ∧ manner(e,well) ∧ time(e,…)‣ “Neo-Davidsonian”viewofevents:thingswithmanyproper8es:

‣ Quan8fica8on:Everyoneisfriendswithsomeone

‣ Generic:Catseatmice(allcatseatmice?mostcats?somecats?)

∀x ∃y friend(x,y)∃y ∀x friend(x,y)(differentfriends)(onefriend)

‣ Samesyntac8cparseforboth!Sosyntaxdoesn'tresolveallambigui8es

sings(Lady Gaga, time=yesterday, manner=well)

‣ Indefinite:Amyateawaffle ∃w. waffle(w) ∧ ate(Amy,w)

QAfromParsing

LadyGagaborn

NP VP

WHADVP

VBD

SQ

WRB

NNP NNP VBN

SBARQ

When was

λx. born(e470,x)‣ Executethisfunc8onagainstaknowledgebasetoanswertheques8on

‣ Trickytoparseduetowh-movement…wouldbeeasierifwesaidLadyGagawasbornwhen

Seman8cParsing‣ Forques8onanswering,syntac8cparsingdoesn’ttellyoueverythingyouwanttoknow,butindicatestherightstructure

‣ Solu8on:seman,cparsing:manyformsofthistaskdependingonseman8cformalisms

‣ Twotoday:CCG(lookslikewhatwe’vebeendoing)andlambda-DCS

CCGParsing

CombinatoryCategorialGrammar‣ Steedman+Szabolcsi1980s:formalismbridgingsyntaxandseman8cs

‣ Syntac8ccategories(forthislecture):S,NP,“slash”categories

‣ S\NP:“ifIcombinewithanNPonmyleXside,Iformasentence”—verb

NP S\NP

Eminem singse728 λy. sings(y)

Ssings(e728)

‣ Parallelderiva8onsofsyntac8cparseandlambdacalculusexpression

‣Whenyouapplythis,therehastobeaparallelinstanceoffunc8onapplica8onontheseman8csside

CombinatoryCategorialGrammar‣ Steedman+Szabolcsi1980s:formalismbridgingsyntaxandseman8cs‣ Syntac8ccategories(forthislecture):S,NP,“slash”categories‣ S\NP:“ifIcombinewithanNPonmyleXside,Iformasentence”—verb‣ (S\NP)/NP:“IneedanNPonmyrightandthenonmyleX”—verbwithadirectobject

NP S\NP

Eminem singse728 λy. sings(y)

Ssings(e728)

NP (S\NP)/NP

Oklahoma borderse101

Texase89NP

λx.λy borders(y,x)

S\NPλy borders(y,e89)

Sborders(e101,e89)

CCGParsing

ZeolemoyerandCollins(2005)

‣ “What”isaverycomplextype:needsanounandneedsaS\NPtoformasentence.S\NPisbasicallyaverbphrase(borderTexas)

‣ Lexiconishighlyambiguous—allthechallengeofCCGparsingisinpickingtherightlexiconentries

CCGParsing

Slidecredit:DanKlein

‣ “to”needsanNP(des8na8on)andN(parent)

BuildingCCGParsers

ZeolemoyerandCollins(2005)

‣Model:log-linearmodeloverderiva8onswithfeaturesonrules:

P (d|x) / expw

>

X

r2d

f(r, x)

!

‣ CanparsewithavariantofCKYEminem sings

NP S\NPe728 λy. sings(y)

Ssings(e728)

f

f

f =Indicator(S\NP->sings)

=Indicator(S->NPS\NP)

BuildingCCGParsers

ZeolemoyerandCollins(2005)

‣ Trainingdatalookslikepairsofsentencesandlogicalforms

WhatstatesborderTexas λx. state(x) ∧ borders(x, e89)

‣ TexascorrespondstoNP|e89inthelogicalform(easytofigureout)

(S/(S\NP))/N|λf.λg.λx. f(x) ∧ g(x)‣Whatcorrespondsto

‣ Howdoweinferthatwithoutbeingtoldit?

‣ Problem:wedon’tknowthederiva8on

Lexicon

WhatstatesborderTexas λx. state(x) ∧ borders(x, e89)

‣ Anysubstringcanparsetoanyoftheseinthelexicon

‣ Chunksinferredfromthelogicformbasedonrules:

‣ GENLEX:takessentenceSandlogicalformL.BreakuplogicalformintochunksC(L),assumeanysubstringofSmightmaptoanychunk

‣ Texas->NP:e89iscorrect‣ borderTexas->NP:e89‣WhatstatesborderTexas->NP:e89… ZeolemoyerandCollins(2005)

‣ NP:e89 ‣ (S\NP)/NP:λx.λy. borders(x,y)

GENLEX

‣ Verycomplexandhand-engineeredwayoftakinglambdacalculusexpressionsand“backsolving”forthederiva8on

ZeolemoyerandCollins(2005)

Learning

ZeolemoyerandCollins(2005)

‣ Itera8veprocedureliketheEMalgorithm:es8mate“best”parsesthatderiveeachlogicalform,retraintheparserusingtheseparseswithsupervisedlearning

‣We’lltalkaboutasimplerformofthisinafewslides

Applica8ons

‣ GeoQuery:answeringques8onsaboutstates(~80%accuracy)

‣ Jobs:answeringques8onsaboutjobpos8ngs(~80%accuracy)

‣ ATIS:flightsearch

‣ CandowellonallofthesetasksifyouhandcraXsystemsanduseplentyoftrainingdata:thesedomainsaren’tthatrich

‣WhataboutbroaderQA?

Lambda-DCS

Lambda-DCS

Liangetal.(2011),Liang(2013)

‣ Dependency-basedcomposi8onalseman8cs—originalversionwaslesspowerfulthanlambdacalculus,lambda-DCSisaspowerful

‣ DesignedinthecontextofbuildingaQAsystemfromFreebase

‣ Freebase:setofen88esandrela8ons

AliceSmith

BobCooper

Seaole

March15,1961 WashingtonDateOfBirth

PlaceOfBirth

PlaceOfBirth

CapitalOf

‣ [[PlaceOfBirth]]=setofpairsof(person,loca8on)

Lambda-DCS

Liangetal.(2011),Liang(2013)

Lambda-DCS Lambdacalculus

Seattle λx. x = SeattlePlaceOfBirth λx.λy. PlaceOfBirth(x,y)

PlaceOfBirth.Seattle λx. PlaceOfBirth(x,Seattle)

‣ LookslikeatreefragmentoverFreebase

SeaolePlaceOfBirth???

Profession.Scientist ∧ PlaceOfBirth.Seattle

λx. Profession(x,Scientist) ∧ PlaceOfBirth(x,Seattle)

Lambda-DCS

Liangetal.(2011),Liang(2013)

AliceSmith

BobCooper

Seaole

March15,1961 WashingtonDateOfBirth

PlaceOfBirth

PlaceOfBirth

CapitalOf

Profession

Scien8st

Profession.Scientist ∧ PlaceOfBirth.Seattle“listofscien8stsborninSeaole”

‣ ExecutethisfragmentagainstFreebase,returnsAliceSmith(andothers)

???

SeaolePlaceOfBirthProfession

Scien8st

ParsingintoLambda-DCS

Berantetal.(2013)

‣ Buildingthelexicon:moresophis8catedprocessthanGENLEX,butcanhandlethousandsofpredicates

‣ Log-linearmodelwithfeaturesonrules: P (d|x) / expw

>

X

r2d

f(r, x)

!

‣ Deriva8ondonsentencex:

‣ SimilartoCRFparsers

‣ Nomoreexplicitsyntax inthesederiva8ons likewehadinCCG

ParsingwithLambda-DCS

Berantetal.(2013)

‣ Learnjustfromques8on-answerpairs:maximizethelikelihoodoftherightdenota8onywiththederiva8ondmarginalizedout

Foreachexample:Runbeamsearchtogetasetofderiva8ons

Letd*=highest-scoringderiva8oninthebeamwithcorrectdenota,onDoastructuredperceptronupdatetowardsd*awayfromd

Letd=highest-scoringderiva8oninthebeam

sumoverderiva8onsdsuchthatthedenota8onofdonknowledgebaseKisyi

Learning

Berantetal.(2013)

‣ Onlyasmallnumberofques8onsareevenreachablebybeamsearchini8ally(butsomeques8onsareveryeasysoevenatotallyuntrainedmodelcananswerthem)‣ Duringtraining,moreandmore“good”deriva8onssurfaceandwillresultinmodelupdates

‣ Eachver8calsliceisthebeamforoneexample.Green=correctdenota8on

Takeaways‣ Canrepresentmeaningwithfirstorderlogicandlambdacalculus

‣ Usefulforqueryingdatabases,ques8onanswering,etc.

‣ Canbridgesyntaxandseman8csandcreateseman8cparsersthatcaninterpretlanguageintolambda-calculusexpressions

‣ Next8me:neuralnetmethodsfordoingthisthatrelylessonhavingexplicitgrammars