LanguageModeling

Prof.SameerSinghCS295:STATISTICALNLP

WINTER2017

January24,2017

BasedonslidesfromDanJurafsky,NoahSmith,andeveryoneelsetheycopiedfrom.

Outline

CS295:STATISTICALNLP(WINTER2017) 2

Wrapup WordEmbeddings

IntroductiontoLanguageModels

N-GramBasedLanguageModels

SmoothingLanguageModels

Outline

CS295:STATISTICALNLP(WINTER2017) 3

Wrapup WordEmbeddings

IntroductiontoLanguageModels

N-GramBasedLanguageModels

SmoothingLanguageModels

Predictsurroundingwords

CS295:STATISTICALNLP(WINTER2017) 4

Abottleoftezguino isonthetable.

u

v

NegativeSampling

CS295:STATISTICALNLP(WINTER2017) 5

NeuralViewofEmbeddings

CS295:STATISTICALNLP(WINTER2017) 6

Wordembeddings

CS295:STATISTICALNLP(WINTER2017) 7

Variations

• Skip-gram:predictcontextfromword• CBOW:predictwordfromcontextbagofwords• Dependencies:abetterdescriptionofcontext

Uses

• Similarity:• Grammar:• Analogies

• Gender:• Facts:

Outline

CS295:STATISTICALNLP(WINTER2017) 8

Wrapup WordEmbeddings

IntroductiontoLanguageModels

N-GramBasedLanguageModels

SmoothingLanguageModels

LanguageModels

CS295:STATISTICALNLP(WINTER2017) 9

ProbabilityofaSentence

• Isagivensentencesomethingyouwouldexpecttosee?• Syntactically(grammar)andSemantically(meaning)

ProbabilityoftheNextWord

• Predictwhatcomesnextforagivensequenceofwords.• ThinkofitasV-wayclassification

Task:SpeechRecognition

CS295:STATISTICALNLP(WINTER2017) 10

“eyesaweofan”

“Isawavan”

OR

Task:MachineTranslation

CS295:STATISTICALNLP(WINTER2017) 11

Task:HandwritingRecognition

CS295:STATISTICALNLP(WINTER2017) 12

http://www.cedar.buffalo.edu/handwriting/HRoverview.html

Task:ImageCaptioning

CS295:STATISTICALNLP(WINTER2017) 13

Task:SpellingCorrection

CS295:STATISTICALNLP(WINTER2017) 14

Theofficeisaboutfifteenminuetsfrommyhouse

P(aboutfifteenminutesfrom)>>P(aboutfifteenminuetsfrom)

OtherApplications

CS295:STATISTICALNLP(WINTER2017) 15

Summarization

QuestionAnswering

DialogSystems

EvaluatingLanguageModels

CS295:STATISTICALNLP(WINTER2017) 16

Bestchoice:Extrinsic

2ndchoice:Intrinsic

Perplexity

CS295:STATISTICALNLP(WINTER2017) 17

GeneratingTextfromanLM

CS295:STATISTICALNLP(WINTER2017) 18

Outline

CS295:STATISTICALNLP(WINTER2017) 19

Wrapup WordEmbeddings

IntroductiontoLanguageModels

N-GramBasedLanguageModels

SmoothingLanguageModels

DirectLanguageModeling

CS295:STATISTICALNLP(WINTER2017) 20

P(“Idonotlikegreeneggsandham”)

P(w|“Idonotlikegreeneggsand”)

ApplyingtheChainRule

CS295:STATISTICALNLP(WINTER2017) 21

MarkovAssumption

CS295:STATISTICALNLP(WINTER2017) 22

UnigramLanguageModel

CS295:STATISTICALNLP(WINTER2017) 23

BigramLanguageModel

CS295:STATISTICALNLP(WINTER2017) 24

BerkeleyRestaurantProject

CS295:STATISTICALNLP(WINTER2017) 25

BerkeleyRestaurantProject

CS295:STATISTICALNLP(WINTER2017) 26

N-GramLanguageModels

CS295:STATISTICALNLP(WINTER2017) 27

“ThecomputerwhichIhadjustputintothediningroomonthefifthfloorcrashed.”

“ThecomputerwhichIhadjustputintothediningroomonthefifthfloorhadlunch.”

Shakespeare

CS295:STATISTICALNLP(WINTER2017) 28

WallStreetJournal

CS295:STATISTICALNLP(WINTER2017) 29

ImplementationTips

CS295:STATISTICALNLP(WINTER2017) 30

UseLogs

• Preventunderflow• Sums,insteadofproducts

Filteroutn-grams

• Raren-gramsarenoisy/havelowprob• Useunigramstofilterbigrams…

Outline

CS295:STATISTICALNLP(WINTER2017) 31

Wrapup WordEmbeddings

IntroductiontoLanguageModels

N-GramBasedLanguageModels

SmoothingLanguageModels

ZeroProbabilityProblem

CS295:STATISTICALNLP(WINTER2017) 32

• Truthiness• #letalonethehashtags• bigly

NewwordsRarewords/combinations Mispellings

• “minuets”• Becausecorpusisfinite..

Trainingset:…deniedtheallegations…deniedthereports…deniedtheclaims…deniedtherequest

P(“offer”|deniedthe)=0

• Testset…deniedtheoffer…deniedtheloan

LaplaceSmoothing

CS295:STATISTICALNLP(WINTER2017) 33

IntuitionBehindSmoothing

CS295:STATISTICALNLP(WINTER2017) 34

Whenwehavesparsestatistics:

P(w|deniedthe)3allegations2reports1claims1request7total

allegatio

ns

repo

rts

claims

attack

requ

est

man

outcome

…

P(w|deniedthe)2.5allegations1.5reports0.5claims0.5request2other7total

allegatio

ns

attack

man

outcome

…allegatio

ns

repo

rts

claims

requ

est

Stealprobabilitymasstogeneralizebetter

BerkeleyRestaurantProject

CS295:STATISTICALNLP(WINTER2017) 35

BerkeleyRestaurantProject

CS295:STATISTICALNLP(WINTER2017) 36

Backoff andInterpolation

CS295:STATISTICALNLP(WINTER2017) 37

• Usetrigram,unlessrare• Thenusebigram,unlessrare• Thenuseunigram..

Backoff

• Combineallthree!• Linearfunctionwithparameters• Learnonheldoutdata

Interpolation

Upcoming…

CS295:STATISTICALNLP(WINTER2017) 38

• Homework1isdue:January26,2017• Write-up,data,andcodeforHomework2isup• Homework2isdue:February9,2017

Homework

• Proposalisdue:February7,2017(~2weeks)• Makethingsmoreconcrete:approach,metrics,baselines• Mentionprogress,andaddressmyconcerns,ifany• Only2pages

Project