language models - university of virginia school of ...hw5x/course/ir2015/_site/docs/pdfs...•allows...

Language Models

Hongning Wang

CS@UVa

Recap: document generation model

CS@UVa CS 4501: Information Retrieval

)0,|(

)1,|(

)0|()0,|(

)1|()1,|(

)0|,(

)1|,(),|1(

RQDP

RQDP

RQPRQDP

RQPRQDP

RDQP

RDQPDQROdd

Model of relevant docs for Q

Model of non-relevant docs for Q

Assume independent attributes of A1…Ak ….(why?)Let D=d1…dk, where dk {0,1} is the value of attribute Ak (Similarly Q=q1…qk )

k

di i

ik

di i

i

k

i ii

ii

iiRQAP

RQAP

RQAP

RQAP

RQdAP

RQdAPDQROdd

0,11,1

1

)0,|0(

)1,|0(

)0,|1(

)1,|1(

)0,|(

)1,|(),|1(

Terms occur in doc

Terms do not occur in doc

document relevant(R=1) nonrelevant(R=0)

term present Ai=1 pi ui

term absent Ai=0 1-pi 1-ui

Ignored for ranking

2

Recap: document generation model


k

qi i

ik

qdi ii

ii

k

qdi i

ik

qdi i

i

k

di i

ik

di i

i

k

i ii

ii

iii

iiii

ii

u

p

pu

up

u

p

u

p

RQAP

RQAP

RQAP

RQAP

RQdAP

RQdAPDQROdd

1,11,1

1,0,11,1

0,11,1

1

1

1

)1(

)1(

1

1

)0,|0(

)1,|0(

)0,|1(

)1,|1(

)0,|(

)1,|(),|1(

Terms occur in doc

Terms do not occur in doc

document relevant(R=1) nonrelevant(R=0)

term present Ai=1 pi ui

term absent Ai=0 1-pi 1-ui

Assumption: terms not occurring in the query are equally likely to occur in relevant and nonrelevantdocuments, i.e., pt=ut

Important tricks

3

Recap: Maximum likelihood vs. Bayesian

• Maximum likelihood estimation– “Best” means “data likelihood reaches maximum”

– Issue: small sample size

• Bayesian estimation – “Best” means being consistent with our “prior”

knowledge and explaining data well

– A.k.a, Maximum a Posterior estimation

– Issue: how to define prior?

CS@UVa CS 4501: Information Retrieval 4

𝜽 = 𝐚𝐫𝐠𝐦𝐚𝐱𝜽𝐏(𝐗|𝜽)

𝜽 = 𝐚𝐫𝐠𝐦𝐚𝐱𝜽𝑷 𝜽 𝑿 = 𝐚𝐫𝐠𝐦𝐚𝐱𝜽𝐏 𝐗 𝜽 𝐏(𝜽)

ML: Frequentist’s point of view

MAP: Bayesian’s point of view

Recap: Robertson-Sparck Jones Model(Robertson & Sparck Jones 76)


Two parameters for each term Ai: pi = P(Ai=1|Q,R=1): prob. that term Ai occurs in a relevant doc ui = P(Ai=1|Q,R=0): prob. that term Ai occurs in a non-relevant doc

k

qdi i

i

i

ik

qdi ii

iiRank

iiiiu

u

p

p

pu

upDQRO

1,11,1

1log

1log

)1(

)1(log),|1(log (RSJ model)

How to estimate these parameters?Suppose we have relevance judgments,

1).(#

5.0).(#ˆ

1).(#

5.0).(#ˆ

docnonrel

Awithdocnonrelu

docrel

Awithdocrelp i

ii

i

• “+0.5” and “+1” can be justified by Bayesian estimation as priors

Per-query estimation!

5

Recap: the BM25 formula

• A closer look

– 𝑏 is usually set to [0.75, 1.2]

– 𝑘1is usually set to [1.2, 2.0]

– 𝑘2 is usually set to (0, 1000]


i

i

i

in

i

iqtfk

kqtf

Davg

Dbbktf

ktfqIDFDqrel

2

2

1

1

1

)1(

)||

||1(

)1()(),(

TF-IDF component for document TF component for query

Vector space model with TF-IDF schema!

Notion of Relevance

Relevance

(Rep(q), Rep(d))

Similarity

P(r=1|q,d) r {0,1}Probability of Relevance

P(d q) or P(q d)

Probabilistic inference

Different

rep & similarity

Vector space

model

(Salton et al., 75)

Prob. distr.

model

(Wong & Yao, 89)

…

Generative ModelRegression

Model

(Fox 83)

Classical

prob. Model

(Robertson &

Sparck Jones, 76)

Doc

generation

Query

generation

LM approach

(Ponte & Croft, 98)

(Lafferty & Zhai, 01a)

Prob. concept

space model

(Wong & Yao, 95)

Different

inference system

Inference

network

model

(Turtle & Croft, 91)


What is a statistical LM?

• A model specifying probability distribution over word sequences

– p(“Today is Wednesday”) 0.001

– p(“Today Wednesday is”) 0.0000000000001

– p(“The eigenvalue is positive”) 0.00001

• It can be regarded as a probabilistic mechanism for “generating” text, thus also called a “generative” model


Why is a LM useful?

• Provides a principled way to quantify the uncertainties associated with natural language

• Allows us to answer questions like:– Given that we see “John” and “feels”, how likely will we see

“happy” as opposed to “habit” as the next word? (speech recognition)

– Given that we observe “baseball” three times and “game” once in a news article, how likely is it about “sports”? (text categorization, information retrieval)

– Given that a user is interested in sports news, how likely would the user use “baseball” in a query? (information retrieval)


Source-Channel framework [Shannon 48]

SourceTransmitter

(encoder)Destination

Receiver

(decoder)

Noisy

Channel

P(X)P(Y|X)

X Y X’

P(X|Y)=?

)()|(maxarg)|(maxargˆ XpXYpYXpXXX

When X is text, p(X) is a language model

(Bayes Rule)

Many Examples: Speech recognition: X=Word sequence Y=Speech signal

Machine translation: X=English sentence Y=Chinese sentence

OCR Error Correction: X=Correct word Y= Erroneous word

Information Retrieval: X=Document Y=Query

Summarization: X=Summary Y=Document


Language model for text

• Probability distribution over word sequences

– 𝑝 𝑤1 𝑤2 … 𝑤𝑛 =𝑝 𝑤1 𝑝 𝑤2 𝑤1 𝑝 𝑤3 𝑤1, 𝑤2 …𝑝 𝑤𝑛 𝑤1, 𝑤2, … , 𝑤𝑛−1

– Complexity - 𝑂(𝑉𝑛∗)

• 𝑛∗ - maximum document length


Chain rule: from conditional probability to joint probability

sentence

• A rough estimate: 𝑂(47500014)

• Average English sentence length is 14.3 words

• 475,000 main headwords in Webster's Third New International Dictionary

47500014

8𝑏𝑦𝑡𝑒𝑠 × 1024 4≈ 3.38𝑒66𝑇𝐵How large is this?

We need independence assumptions!

Unigram language model

• Generate a piece of text by generating each word independently

– 𝑝(𝑤1𝑤2 … 𝑤𝑛) = 𝑝(𝑤1)𝑝(𝑤2)…𝑝(𝑤𝑛)

– 𝑠. 𝑡. 𝑝 𝑤𝑖 𝑖=1𝑁 , 𝑖 𝑝 𝑤𝑖 = 1 , 𝑝 𝑤𝑖 ≥ 0

• Essentially a multinomial distribution over the vocabulary

0

0.02

0.04

0.06

0.08

0.1

0.12

w1

w2

w3

w4

w5

w6

w7

w8

w9

w1

0

w1

1

w1

2

w1

3

w1

4

w1

5

w1

6

w1

7

w1

8

w1

9

w2

0

w2

1

w2

2

w2

3

w2

4

w2

5

A Unigram Language Model


The simplest and most popular choice!

More sophisticated LMs

• N-gram language models– In general, 𝑝(𝑤1𝑤2 …𝑤𝑛) =𝑝(𝑤1)𝑝(𝑤2|𝑤1)…𝑝(𝑤𝑛|𝑤1…𝑤𝑛−1)

– N-gram: conditioned only on the past N-1 words

– E.g., bigram: 𝑝(𝑤1 … 𝑤𝑛) =𝑝(𝑤1)𝑝(𝑤2|𝑤1) 𝑝(𝑤3|𝑤2) …𝑝(𝑤𝑛|𝑤𝑛−1)

• Remote-dependence language models (e.g., Maximum Entropy model)

• Structured language models (e.g., probabilistic context-free grammar)


Why just unigram models?

• Difficulty in moving toward more complex models

– They involve more parameters, so need more data to estimate

– They increase the computational complexity significantly, both in time and space

• Capturing word order or structure may not add so much value for “topical inference”

• But, using more sophisticated models can still be expected to improve performance ...


Generative view of text documents(Unigram) Language Model

p(w| )

…text 0.2mining 0.1assocation 0.01clustering 0.02…food 0.00001…

Topic 1:Text mining

…text 0.01food 0.25nutrition 0.1healthy 0.05diet 0.02…

Topic 2:Health

Document

Text miningdocument

Food nutritiondocument

Sampling


Sampling with replacement

• Pick a random shape, then put it back in the bag


How to generate text document from an N-gram language model?

• Sample from a discrete distribution 𝑝(𝑋)

– Assume 𝑛 outcomes in the event space 𝑋

1. Divide the interval [0,1] into 𝑛 intervals according to the probabilities of the outcomes

2. Generate a random number 𝑟 uniformlybetween 0 and 1

3. Return 𝑥𝑖 where 𝑟 falls into


apple

arm

banana

bikebird

book

chin

clamclass

A UNIGRAM LANGUAGE MODEL

Generating text from language models


Under a unigram language model:

Generating text from language models


The same likelihood!Under a unigram language model:

N-gram language models will help

• Unigram– Months the my and issue of year foreign new exchange’s

september were recession exchange new endorsed a q acquire to six executives.

• Bigram– Last December through the way to preserve the Hudson

corporation N.B.E.C. Taylor would seem to complete the major central planners one point five percent of U.S.E. has already told M.X. corporation of living on information such as more frequently fishing to keep her.

• Trigram– They also point to ninety nine point six billon dollars from two

hundred four oh six three percent of the rates of interest stores as Mexico and Brazil on market conditions.


Generated from language models of New York Times

Turing test: generating Shakespeare


A

B

C

D

SCIgen - An Automatic CS Paper Generator

http://pdos.csail.mit.edu/scigen/

Recap: what is a statistical LM?

• A model specifying probability distribution over word sequences

– p(“Today is Wednesday”) 0.001

– p(“Today Wednesday is”) 0.0000000000001

– p(“The eigenvalue is positive”) 0.00001

• It can be regarded as a probabilistic mechanism for “generating” text, thus also called a “generative” model


Recap: Source-Channel framework [Shannon 48]

SourceTransmitter

(encoder)Destination

Receiver

(decoder)

Noisy

Channel

P(X)P(Y|X)

X Y X’

P(X|Y)=?

)()|(maxarg)|(maxargˆ XpXYpYXpXXX

When X is text, p(X) is a language model

(Bayes Rule)

Many Examples: Speech recognition: X=Word sequence Y=Speech signal

Machine translation: X=English sentence Y=Chinese sentence

OCR Error Correction: X=Correct word Y= Erroneous word

Information Retrieval: X=Document Y=Query

Summarization: X=Summary Y=Document


Recap: language model for text

• Probability distribution over word sequences

– 𝑝 𝑤1 𝑤2 … 𝑤𝑛 =𝑝 𝑤1 𝑝 𝑤2 𝑤1 𝑝 𝑤3 𝑤1, 𝑤2 …𝑝 𝑤𝑛 𝑤1, 𝑤2, … , 𝑤𝑛−1

– Complexity - 𝑂(𝑉𝑛∗)

• 𝑛∗ - maximum document length


Chain rule: from conditional probability to joint probability

sentence

• A rough estimate: 𝑂(47500014)

• Average English sentence length is 14.3 words

• 475,000 main headwords in Webster's Third New International Dictionary

47500014

8𝑏𝑦𝑡𝑒𝑠 × 1024 4≈ 3.38𝑒66𝑇𝐵How large is this?

We need independence assumptions!

Recap: how to generate text document from an N-gram language model?

• Sample from a discrete distribution 𝑝(𝑋)

– Assume 𝑛 outcomes in the event space 𝑋

1. Divide the interval [0,1] into 𝑛 intervals according to the probabilities of the outcomes

2. Generate a random number 𝑟 uniformlybetween 0 and 1

3. Return 𝑥𝑖 where 𝑟 falls into


Estimation of language models


Document

text 10mining 5association 3database 3algorithm 2…query 1efficient 1

Unigram Language Model p(w| )=?

…text ?mining ?assocation ?database ?…query ?…

Estimation

A “text mining” paper(total #words=100)

Sampling with replacement

• Pick a random shape, then put it back in the bag


Estimation of language models

• Maximum likelihood estimation

Unigram Language Model p(w| )=?

Document

text 10mining 5association 3database 3algorithm 2…query 1efficient 1

…text ?mining ?assocation ?database ?…query ?…

Estimation

A “text mining” paper(total #words=100)

10/1005/1003/1003/100

1/100


Maximum likelihood estimation

• For N-gram language models

– 𝑝 𝑤𝑖 𝑤𝑖−𝑛+1, … , 𝑤𝑖−1 =𝑐(𝑤𝑖−𝑛+1,…,𝑤𝑖−1,𝑤𝑖)

𝑐(𝑤𝑖−𝑛+1,…,𝑤𝑖−1)

• 𝑐 ∅ = 𝑁


Length of document or total number of words in a corpus

Language models for IR[Ponte & Croft SIGIR’98]

Document

Text miningpaper

Food nutritionpaper

Language Model

…text ?mining ?assocation ?clustering ?…food ?…

…food ?nutrition ?healthy ?diet ?…

?Which model would most likely have generated this query?

“data mining algorithms”

Query


Ranking docs by query likelihood

d1

d2

dN

qp(q| d1

)

p(q| d2)

p(q| dN)

Query likelihood

……

d1

d2

dN

Doc LM

……

)1,|(),|1( RDQPDQRO

Justification: PRP


Justification from PRP

))0|()0,|(()0|(

)1|()1,|(

)0|()0,|(

)1|()1,|(

)0|,(

)1|,(),|1(

RQPRDQPAssumeRDP

RDPRDQP

RDPRDQP

RDPRDQP

RDQP

RDQPDQRO

Query likelihood p(q| d) Document prior

Assuming uniform document prior, we have

)1,|(),|1( RDQPDQRO


Query generation

Retrieval as language model estimation

• Document ranking based on query likelihood

– Retrieval problem Estimation of 𝑝(𝑤𝑖|𝑑)

– Common approach

• Maximum likelihood estimation (MLE)

n

i

i

wwwqwhere

dwpdqp

...,

)|(log)|(log

21

Document language model


Problem with MLE

• Unseen events

– There are 440K tokens on a larger collection of Yelp reviews, but:

• Only 30,000 unique words occurred

• Only 0.04% of all possible bigrams occurred

This means any word/N-gram that does not occur in the collection has zero probability with MLE!

No future documents can contain those unseen words/N-grams


A plot of word frequency in Wikipedia (Nov 27, 2006)

Wo

rd f

req

uen

cy

Word rank by frequency

𝑓 𝑘; 𝑠, 𝑁 ∝ 1/𝑘𝑠

Problem with MLE

• What probability should we give a word that has not been observed in the document?

– log0?

• If we want to assign non-zero probabilities to such words, we’ll have to discount the probabilities of observed words

• This is so-called “smoothing”


General idea of smoothing

• All smoothing methods try to

1. Discount the probability of words seen in a document

2. Re-allocate the extra counts such that unseen words will have a non-zero count


Illustration of language model smoothing

P(w|d)

w

Max. Likelihood Estimate

wordsallofcount

wofcount

ML wp )(

Smoothed LM

Assigning nonzero probabilities to the unseen words


Discount from the seen words

Smoothing methods

• Method 1: Additive smoothing

– Add a constant to the counts of each word

– Problems?

• Hint: all words are equally important?

( , ) 1( | )

| | | |

c w dp w d

d V

“Add one”, Laplace smoothing

Vocabulary size

Counts of w in d

Length of d (total counts)


Add one smoothing for bigrams


After smoothing

• Giving too much to the unseen events


Refine the idea of smoothing

• Should all unseen words get equal probabilities?

• We can use a reference model to discriminate unseen words

( | )( | )

( | )

seen

d

p w d if w is seen in dp w d

p w REF otherwise

Discounted ML estimate

Reference language model1 ( | )

( | )

seen

w is seen

d

w is unseen

p w d

p w REF


Smoothing methods

• Method 2: Absolute discounting – Subtract a constant from the counts of each

word

– Problems? • Hint: varied document length?

max( ( ; ) ,0) | | ( | )

| |( | ) uc w d d p w REF

dp w d

# uniq words


Smoothing methods

• Method 4: Dirichlet Prior/Bayesian

– Assume pseudo counts p(w|REF)

– Problems?

( ; ) ( | ) | |

| | | | | |

( , )( | ) ( | )

| |

c w d p w REF d

d d d

c w dp w d p w REF

d

parameter


Dirichlet prior smoothing

• Bayesian estimator

– Posterior of LM: 𝑝 𝜃 𝑑 ∝ 𝑝 𝑑 𝜃 𝑝(𝜃)

• Conjugate prior

• Posterior will be in the same form as prior

• Prior can be interpreted as “extra”/“pseudo” data

• Dirichlet distribution is a conjugate prior for multinomial distribution

1

11

11

)()(

)(),,|(

i

i

N

iN

NNDir

“extra”/“pseudo” word counts, we set i= p(wi|REF)

prior over models

likelihood of doc given the model


Some background knowledge

• Conjugate prior

– Posterior dist in the same family as prior

• Dirichlet distribution

– Continuous

– Samples from it will be the parameters in a multinomial distribution

Gaussian -> GaussianBeta -> BinomialDirichlet -> Multinomial


Dirichlet prior smoothing (cont)

))( , ,)( | () | ( 11 NNwcwcDirdp

Posterior distribution of parameters:

}{)E(then ),|(~ If :Property

i

iDir

The predictive distribution is the same as the mean:

|d|

)|()w(

|d|

)w(

)|()|p(w)ˆ|p(w

i

1i

i

ii

REFwpcc

dDir

i

N

i

i

Dirichlet prior smoothingCS@UVa CS 4501: Information Retrieval 47

Estimating using leave-one-out [Zhai & Lafferty 02]

P(w1|d- w1)

P(w2|d- w2)

N

i Vw i

ii

d

CwpdwcdwcCL

1

1 )1||

)|(1),(log(),()|(

log-likelihood

)C|(μLargmaxμ̂ 1μ

Maximum Likelihood Estimator

Leave-one-outw1

w2

P(wn|d- wn)

wn

...


Why would “leave-one-out” work?

abc abc ab c d d

abc cd d d

abd ab ab ab ab

cd d e cd e

20 word by author1

20 word by author2

abc abc ab c d d

abe cb e f

acf fb ef aff abef

cdc db ge f s

Now, suppose we leave “e” out…

1 20 1(" " | 1) (" " | 1) (" " | )

19 20 19 20

0 20 0(" " | 2) (" " | 2) (" " | )

19 20 19 20

ml smooth

ml smooth

p e author p e author p e REF

p e author p e author p e REF

must be big! more smoothing

doesn’t have to be big

The amount of smoothing is closely related to

the underlying vocabulary size


Recap: ranking docs by query likelihood

d1

d2

dN

qp(q| d1

)

p(q| d2)

p(q| dN)

Query likelihood

……

d1

d2

dN

Doc LM

……

)1,|(),|1( RDQPDQRO

Justification: PRP


Recap: retrieval as language model estimation

• Document ranking based on query likelihood

– Retrieval problem Estimation of 𝑝(𝑤𝑖|𝑑)

– Common approach

• Maximum likelihood estimation (MLE)

n

i

i

wwwqwhere

dwpdqp

...,

)|(log)|(log

21

Document language model


Recap: illustration of language model smoothing

P(w|d)

w

Max. Likelihood Estimate

wordsallofcount

wofcount

ML wp )(

Smoothed LM

Assigning nonzero probabilities to the unseen words


Discount from the seen words

Recap: smoothing methods

• Method 1: Additive smoothing

– Add a constant to the counts of each word

– Problems?

• Hint: all words are equally important?

( , ) 1( | )

| | | |

c w dp w d

d V

“Add one”, Laplace smoothing

Vocabulary size

Counts of w in d

Length of d (total counts)


Recap: refine the idea of smoothing

• Should all unseen words get equal probabilities?

• We can use a reference model to discriminate unseen words

( | )( | )

( | )

seen

d


p w REF otherwise

Discounted ML estimate

Reference language model1 ( | )

( | )

seen

w is seen

d

w is unseen

p w d

p w REF



• Method 2: Absolute discounting – Subtract a constant from the counts of each

word

– Problems? • Hint: varied document length?

max( ( ; ) ,0) | | ( | )

| |( | ) uc w d d p w REF

dp w d

# uniq words



• Method 4: Dirichlet Prior/Bayesian

– Assume pseudo counts p(w|REF)

– Problems?

( ; ) ( | ) | |

| | | | | |

( , )( | ) ( | )

| |

c w d p w REF d

d d d

c w dp w d p w REF

d

parameter


Two-stage smoothing [Zhai & Lafferty 02]

c(w,d)

|d|P(w|d) =

+p(w|C)

+

Stage-1

-Explain unseen words

-Dirichlet prior (Bayesian)

Collection LM

(1-) + p(w|U)

Stage-2

-Explain noise in query

-2-component mixture

User background model


Understanding smoothing

• Plug in the general smoothing scheme to the query likelihood retrieval formula, we obtain

qw

id

qdw id

iseen

ii

CwpqCwp

dwpdqp )|(loglog||]

)|(

)|([log)|(log

Ignore for rankingIDF weighting

TF weightingDoc length normalization(longer doc is expected to have a smaller d)

• Smoothing with p(w|C) TF-IDF + doc-

length normalization


1 ( | )

( | )

seen

w is seen

d

w is unseen

p w d

p w REF

, ( , ) 0

, ( , ) 0, , ( , ) 0, ( , ) 0( , ) 0

, ( , ) 0 , ( , )

log ( | ) ( , ) log ( | )

( , ) log ( | ) ( , ) log ( | )

( , ) log ( | ) ( , ) log ( | ) ( , ) log ( | )

w V c w q

Seen d

w V c w d w V c w q c w dc w q

Seen d d

w V c w d w V c w q

p q d c w q p w d

c w q p w d c w q p w C

c w q p w d c w q p w C c w q p w C

, ( , ) 0 0, ( , ) 0

, ( , ) 0 , ( , ) 0( , ) 0

( | )( , ) log | | log ( , ) ( | )

( | )

w V c w q c w d

Seend

w V c w d w V c w qdc w q

p w dc w q q c w q p w C

p w C

Smoothing & TF-IDF weighting

( | )( | )

( | )

Seen

d


p w C otherwise

1 ( | )

( | )

Seen

w is seen

d

w is unseen

p w d

p w C

Smoothed ML estimate

Reference language model

Retrieval formula using the

general smoothing scheme

Key rewriting step (where did we see it before?)


Similar rewritings are very common when

using probabilistic models for IR…

𝑐 𝑤, 𝑞 > 0

What you should know

• How to estimate a language model

• General idea and different ways of smoothing

• Effect of smoothing


Today’s reading

• Introduction to information retrieval

– Chapter 12: Language models for information retrieval


language models - university of virginia school of ...hw5x/course/ir2015/_site/docs/pdfs...•allows...

Documents