ttic 31190: natural language processing › ~kgimpel › teaching › 31190 › lectures ›...

TTIC31190:NaturalLanguageProcessing

KevinGimpelWinter2016

Lecture16:MachineTranslation

andotherNLPApplications

Announcements• presentationswillactuallybe9minutesbecausewehavesomanytofitin

• Iwillpostguidelinesonthefinalprojectreport– thinkofitasashort(4-page)paper

• Iwillsendyouyourmidtermandassignment2gradestomorrow

Roadmap• classification• words• lexicalsemantics• languagemodeling• sequencelabeling• neuralnetworkmethodsinNLP• syntaxandsyntacticparsing• computationalsemantics• machinetranslation• otherNLPapplications

model score

BLEUscore

AfricanNationalCongress opposition sanction Zimbabwe

非国大 反对 制裁 津巴布韦

Goldstandard:AfricanNationalCongressopposes

sanctionsagainstZimbabwe

model score

BLEUscore

AfricanNationalCongress opposition sanction Zimbabwe

非国大 反对 制裁 津巴布韦

opposition tosanctionsagainstZimbabweAfricanNationalCongress

predictedtranslation

Goldstandard:AfricanNationalCongressopposes

sanctionsagainstZimbabwe

model score

BLEUscore

AfricanNationalCongress opposition sanction Zimbabwe

非国大 反对 制裁 津巴布韦

Goldstandard:AfricanNationalCongressopposes

sanctionsagainstZimbabwe

learningmovestranslationsleftorrightinthisplot

AfricanNationalCongress opposition sanction Zimbabwe

非国大 反对 制裁 津巴布韦

model score

BLEUscore

“ideal”model

Goldstandard:AfricanNationalCongressopposes

sanctionsagainstZimbabwe

model score

BLEUscore

AfricanNationalCongress opposition sanction Zimbabwe

非国大 反对 制裁 津巴布韦

Issue:goldstandardtranslationisoften

unreachable bythemodel

Goldstandard:AfricanNationalCongressopposes

sanctionsagainstZimbabwe

Why?limitedtranslationrules,

freetranslations,noisydata

model score

BLEUscore

PerceptronLossgoldstandard

model score

BLEUscore

PerceptronLoss

modelprediction

goldstandard

model score

BLEUscore

HingeLossgoldstandard

cost-augmentedprediction

model score

BLEUscore

PerceptronLossforMT?(Collins,2002)

reference

modelprediction

model score

BLEUscore

RampLossMinimization

model score

BLEUscore

RampLossMinimization

modelprediction

model score

BLEUscore

RampLossMinimization

modelprediction

“fear”translation

model score

BLEUscore

“Fear”RampLoss(Doetal.,2008)

modelprediction

“fear”translation

model score

BLEUscore

“Hope”RampLoss(McAllester &Keshet,2011; Liangetal.,2006)

modelprediction

model score

BLEUscore

modelprediction

“hope”translation

“Hope”RampLoss(McAllester &Keshet,2011; Liangetal.,2006)

model score

BLEUscore

“Hope-Fear”RampLoss(Chiangetal.,2008;2009;Cherry&Foster,2012;

Chiang,2012;Gimpel &Smith,2012)

“hope”translation

“fear”translation

Experiments(Gimpel,2012)

Moses%BLEU Hiero %BLEU

MERT 35.9 37.0

FearRamp(awayfrombad) 34.9 34.2

HopeRamp(towardgood) 35.2 36.0

Hope-FearRamp(towardgood+awayfrombad) 35.7 37.0

averagesover8 testsetsacross3languagepairs

Whydoyouthinkthathoperampworksbetterthanfearramp?

Ithink:goingawayfromsomethingbaddoesnotnecessarilymeanthatyouaregoingtowardsomethinggood.

youmightbegoingtowardsomethingelsethat’sbad!

ClassificationFrameworkforMachineTranslation

• wehavealatentvariable,sothisbecomes:

• wemaximizeoverthelatentvariableANDtheoutput!• h couldbewordalignments,phrasesegmentations/alignments,synchronousCFGderivations,etc.

inference:solve_

• Forphrase-basedtranslation,searchover:– Segmentationsintophrases– Translationsforeachphrase– Orderingsofthetranslatedphrases

zimbabwe african national congresssanctions againstopposition to

ANC opposition sanction Zimbabwe

非国大 反对 制裁 津巴布韦

african national congress opposes sanctions against zimbabweReference:

• Forphrase-basedtranslation,searchover:– Segmentationsintophrases– Translationsforeachphrase– Orderingsofthetranslatedphrases

zimbabwe african national congresssanctions againstopposition to

ANC opposition sanction Zimbabwe

非国大 反对 制裁 津巴布韦

african national congress opposes sanctions against zimbabweReference:

ThissearchproblemisNP-hard(Knight,1999)Approximatebeamsearchisusedinpractice

AfricanNationalCongress opposition sanction Zimbabwe

非国大 反对 制裁 津巴布韦

African National Congress opposessanctions against Zimbabwe

Reference translation:

Koehn et al. (2003)

Phrase-Based Machine Translation

AfricanNationalCongress opposition sanction Zimbabwe

非国大 反对 制裁 津巴布韦

African National Congress opposessanctions against Zimbabwe

Reference translation:

1 非国大 / African National Congress 2 反对 / opposition to 3 反对 / is opposed to 4 制裁 / sanctions 5 制裁 津巴布韦 /

sanctions against Zimbabwe...

Phrase TableKoehn et al. (2003)

Phrase-Based Machine Translation

AfricanNationalCongress opposition sanction Zimbabwe

非国大 反对 制裁 津巴布韦

African National Congress opposessanctions against Zimbabwe

Reference translation:

1 非国大 / African National Congress 2 反对 / opposition to 3 反对 / is opposed to 4 制裁 / sanctions 5 制裁 津巴布韦 /

sanctions against Zimbabwe...

Phrase Table

opposition to

African National Congress

2

1

Koehn et al. (2003)

Phrase-Based Machine Translation

AfricanNationalCongress opposition sanction Zimbabwe

非国大 反对 制裁 津巴布韦

African National Congress opposessanctions against Zimbabwe

Reference translation:

1 非国大 / African National Congress 2 反对 / opposition to 3 反对 / is opposed to 4 制裁 / sanctions 5 制裁 津巴布韦 /

sanctions against Zimbabwe...

Phrase Table

opposition toopposition to sanctions

against Zimbabwe

African National Congress

African National Congressis opposed to

2 5

3

1

Koehn et al. (2003)

Phrase-Based Machine Translation

AfricanNationalCongress opposition sanction Zimbabwe

非国大 反对 制裁 津巴布韦

African National Congress opposessanctions against Zimbabwe

Reference translation:

1 非国大 / African National Congress 2 反对 / opposition to 3 反对 / is opposed to 4 制裁 / sanctions 5 制裁 津巴布韦 /

sanctions against Zimbabwe...

Phrase Table

opposition to sanctionsagainst Zimbabwe

African National Congress

1opposition to

opposition to sanctionsagainst Zimbabwe

African National Congress

African National Congressis opposed to

2 5

3

1

Koehn et al. (2003)

Phrase-Based Machine Translation

zimbabwe african national congresssanctions againstopposition to

zimbabwe african national congresssanctions onopposition to

zimbabwe african national congress sanctionsopposition to

zimbabweafrican national congress sanctions againstopposition to

zimbabweafrican national congress sanctions againstoppose

1

2

3

4

5

otherusefulinferencetasks:• findk-besttranslations

-11.8

-12.1

-12.4

-12.9

-13.5

Rank Score

typicallatticescontainupto1080 paths!(butnotallareuniquetranslations)

zimbabwe

zimbabwe

zimbabwe

african national congress

african national congress

african national congress

african national assemblyafricannationalcongress

sanctions against

sanctions on

sanctions against

sanctions

sanctions against

opposition to

opposition to

opposition tozimbabwe

is opposed to

otherusefulinferencetasks:• findphraselatticeoftranslations

NeuralNetworksandMachineTranslation

• currenttrendinMTresearchistouseneuralnetworksforeverything

• “neuralMT”typicallyreferstoapproachesthatonlyuseneuralnetworks

• butmostMTsystemscombinetraditionalphrase-basedmodelswithfeaturesbasedonneuralnetworks

ACL2014(bestpaperaward)

ACL2014

ACL2014

NeuralMT

EMNLP2013

EMNLP2013

EMNLP2014

EMNLP2014

NIPS2014

NIPS2014

NIPS2014

ICLR2015

ICLR2015

ICLR2015

OtherNLPTasksandApplications• coreference resolution• questionanswering• summarization• dialoguesystems

OtherNLPTasksandApplications• coreference resolution• questionanswering• summarization• dialoguesystems

Coreference Resolution• determinewhichpiecesoftextrefertothesamereferent:– PresidentObamaselectedtendelegatesafterreceivingrecommendationsfromhis cabinetmembers.They spentalldaySaturdayworkingontheir recommendationsforhim.

OtherNLPTasksandApplications• coreference resolution• questionanswering

– factoidquestionanswering– machinecomprehension

• summarization• dialoguesystems

IBM’sWatson

IBM’sWatson

ClassifyingQuestionsinto“LexicalAnswerTypes”

OtherNLPTasksandApplications• coreference resolution• questionanswering• summarization• dialoguesystems

AutomaticSummarization• givenadocument,produceasummaryofaprovidedlength

• vastmajorityofsystemsareextractive:theyextractcontentfromthedocument– thisissafer,sincethedocumentispresumablygrammatical

– butthislimitsapplicability• somework,especiallyrecently,thattriestodoabstractivesummarization– typicallybasedonintermediatesemanticrepresentationsorneuralnetworks

baseline=takefirst100wordsofdocument

regarding thefirsttwoyearsofDUC:

AAAI2005

MachineComprehensionCanamachinereadadocumentand

answerquestionsaboutit?

57

58

• 660fictionalstories,writtenata4th gradereadinglevel

• 4multiplechoicequestionsperstory

OncetherewasaboynamedFritzwholovedtodraw.Hedreweverything.Inthemorning,hedrewapictureofhiscerealwithmilk.Hispapasaid,“Don’tdrawyourcereal.Eatit!”Afterschool,Fritzdrewapictureofhisbicycle.Hisunclesaid,“Don'tdrawyourbicycle.Rideit!”…

59

OncetherewasaboynamedFritzwholovedtodraw.Hedreweverything.Inthemorning,hedrewapictureofhiscerealwithmilk.Hispapasaid,“Don’tdrawyourcereal.Eatit!”Afterschool,Fritzdrewapictureofhisbicycle.Hisunclesaid,“Don'tdrawyourbicycle.Rideit!”…

WhatdidFritzdrawfirst?A)thetoothpasteB)hismamaC)cerealandmilkD)hisbicycle

60

OncetherewasaboynamedFritzwholovedtodraw.Hedreweverything.Inthemorning,hedrewapictureofhiscerealwithmilk.Hispapasaid,“Don’tdrawyourcereal.Eatit!”Afterschool,Fritzdrewapictureofhisbicycle.Hisunclesaid,“Don'tdrawyourbicycle.Rideit!”…

WhatdidFritzdrawfirst?A)thetoothpasteB)hismamaC)cerealandmilkD)hisbicycle

61

OncetherewasaboynamedFritzwholovedtodraw.Hedreweverything.Inthemorning,hedrewapictureofhiscerealwithmilk.Hispapasaid,“Don’tdrawyourcereal.Eatit!”Afterschool,Fritzdrewapictureofhisbicycle.Hisunclesaid,“Don'tdrawyourbicycle.Rideit!”…

WhatdidFritzdrawfirst?A)thetoothpasteB)hismamaC)cerealandmilkD)hisbicycleE)everything

62

63

JamestheTurtlewasalwaysgettingintrouble.…

Whatisthenameofthetroublemakingturtle?A)FriesB)PuddingC)JamesD)Jane

• Somequestionsaremucheasier

• Simplewordoverlapbaselinegets63%correct

64

institution year accuracy(%)

TTI-Chicago 2015 69.9

CarnegieMellon 2015 67.8

UniversityCollegeLondon 2015 66.0

MIT 2015 63.8

Microsoft Research 2013 63.3

MCTest Leaderboard

Oursystemusesseveraltypesofautomaticlinguisticanalysis:

– dependencyparsing– framesemanticparsing– coreference– wordembeddings

65

Oursystemusesseveraltypesofautomaticlinguisticanalysis:

– dependencyparsing

66

Oursystemusesseveraltypesofautomaticlinguisticanalysis:

– dependencyparsing

67

outputofStanforddependencyparser

Oursystemusesseveraltypesofautomaticlinguisticanalysis:

– dependencyparsing

68

FritzdrawXfirst

Oursystemusesseveraltypesofautomaticlinguisticanalysis:

– dependencyparsing

69

FritzdrawXfirst

FritzdrawthetoothpastefirstFritz draw his mama firstFritz draw cereal and milk firstFritz draw his bicycle first

Oursystemusesseveraltypesofautomaticlinguisticanalysis:

– dependencyparsing– framesemanticparsing

70

Oursystemusesseveraltypesofautomaticlinguisticanalysis:

– dependencyparsing– framesemanticparsing

71

outputofCarnegieMellonframesemanticparser

Oursystemusesseveraltypesofautomaticlinguisticanalysis:

– dependencyparsing– framesemanticparsing

72

Oursystemusesseveraltypesofautomaticlinguisticanalysis:

– dependencyparsing– framesemanticparsing

73

Oursystemusesseveraltypesofautomaticlinguisticanalysis:

– dependencyparsing– framesemanticparsing– coreference

74

Oursystemusesseveraltypesofautomaticlinguisticanalysis:

– dependencyparsing– framesemanticparsing– coreference

75

outputofStanfordcoreference resolutionsystem

Oursystemusesseveraltypesofautomaticlinguisticanalysis:

– dependencyparsing– framesemanticparsing– coreference– wordembeddings

76

OncetherewasaboynamedFritz wholovedtodraw.Hedreweverything.Inthemorning,hedrewapictureofhiscerealwithmilk.Hispapasaid,“Don’tdrawyourcereal.Eatit!”…WhatdidFritzdrawfirst?

77

OncetherewasaboynamedFritz wholovedtodraw.Hedreweverything.Inthemorning,hedrewapictureofhiscerealwithmilk.Hispapasaid,“Don’tdrawyourcereal.Eatit!”…WhatdidFritzdrawfirst?

transformedquestion(usingdependencyparsing):Fritzdrawcerealandmilkfirst

Fritz≈he (coreference,framesemantics)draw≈drew (wordembeddings,framesemantics)withmilk≈andmilk (wordembeddings)

78

RemovingFeaturesOneataTime

69.9

67.667.9

68.4 68.3

64

65

66

67

68

69

70

71

72

allfeatures

removedependencyparsing

removeframesemantics

removecoreference

removeembeddings

Accuracy