Probabilistic Context Free Grammars

Grant Schindler

8803-MDM

April 27, 2006

Problem

PCFGs can model a more powerful class of languages than HMMs. Can we take advantage of this property?

Regular Language

Context Free Language Probabilistic Context Free Grammar (PCFG)

Hidden Markov Model (HMM)

Context Sensitive Language

Unrestricted Language

PCFG Background

S N V (1.0)

N Bob (0.3) Jane (0.7)

<Left-Hand Side> <Right-Hand Side> (Probability)

V V N (0.4) loves (0.6)

Example Grammar:

Production Rule:

Jane loves Bob.

S

VN

V

N

Example Parse:

PCFG Applications

•Natural Language Processing: parsing written sentences

•BioInformatics: RNA sequences

•Stock Markets: model rise/fall of the Dow Jones (?)

•Computer Vision: parsing architectural scenes

PCFG Application: Architectural Facade Parsing

Goal: Inferring 3D Semantic Structure

Discrete vs. Continuous Observations

•Natural Language Processing: parsing written sentences

•BioInformatics: RNA sequences

•Stock Markets: model rise/fall of the Dow Jones (?)

Discrete Values

Continuous Values

How do we estimate the parameters of PCFGs with continuous observation densities (terminal nodes in the parse tree)?

PCFG Parameter Estimation

In the discrete case, there exists an Expectation Maximization (EM) algorithm:

E-Step: Compute expected number of times each rule (A-> BC) is used in generating a given set of observation sequences (based on previous parameter estimates).

M-Step: Update parameters as normalized counts computed in E-Step.

Essentially: P*(N Bob) = #Bobs / #Nouns

Gaussian Parameter Update Equations

NEW!

Probability that rule A was applied to generate the observed value at location i, computed from Inside-Outside Algorithm via CYK Algorithm

Significance

We can now begin applying probabilistic context-free grammars to problems with continuous data (e.g. stock market) rather than restricting ourselves to discrete outputs (e.g. natural language, RNA).

We hope to find problems for which PCFGs offer a better model than HMMs.

Questions

Open Problems

How do we estimate the parameters of PCFGs with:

A. continuous observation densities (terminal nodes in the parse tree)?

B. continuous values for both non-terminal and terminal nodes?

CYK Algorithm

Inside-Outside Probabilities