syntactic parsing introduction - cl.lingfil.uu.se

Syntactic Parsing

Introduction

Joakim Nivre and Daniel DakotaDepartment of Linguistics and Philology

Partly based on slides from Marco Kuhlmann and Sara Stymne

Today

• Introduction to syntactic parsing

• Course information

What is syntactic parsing?

• Given a natural language sentence, compute a representation of its syntactic structure

Why study syntactic parsing?

• Syntactic structure determines semantic interpretation

Disney acquired PixarPixar acquired DisneyPixar was acquired by Disney

• Natural language understanding applications

- Information extraction- Question answering

• Grammar checking

• Modeling human sentence processing

Parsing as Structured Prediction

Parsing is a structured prediction problem:

X → Y

• Input space X: sentences

• Output space Y: syntactic representations

• X and Y are infinite sets of structured objects

• Complexity requires specialised algorithms

Input Space

A sentence x ∈ X is a sequence of tokens x1, …, xn

• How do we delimit sentences?

• How do we split sentences into tokens?

- Different writing systems (white space or not)

- Different degrees of morphological complexity

• How do we preprocess tokens before parsing?

- Part-of-speech tagging

- Morphological analysis

- Lemmatization

Output Space

A syntactic representation y ∈ Y is a labeled graph

• Constituency-based representations

- Sentences decomposed into phrases and words

- Focus on internal structure of phrases

• Dependency-based representations

- Words connected by dependency relations

- Focus on functional role of words

• Hybrid representations exist

Syntactic Representations


Terminal nodes


Nonterminal nodes


Root node


Nodes = Terminal nodes


Labeled arcs


Root node

Mapping

What is the relation between inputs and outputs?

X → Y

• Grammar G generates x ∈ X with analysis y ∈ Y

- Recognition – does G generate x (yes/no)?

- Parsing – derive (all) y assigned to x by G

• Every instance of x has a single interpretation y

- Disambiguation – select the “best” y for x

- Parsing as optimization: y* = argmaxy f(x, y)

Ambiguity

I shot an elephant in my pajamas.

• This sentence is ambiguous. In what way?

• What should happen if we parse the sentence?

Ambiguity

Ambiguity

Recognition: Yes

Ambiguity

Recognition: Yes

Parsing with G

Ambiguity

Recognition: Yes

Parsing with G

Disambiguation

Parsing Models

A model for parsing (with disambiguation):

• GEN(x) ⊆ Y

- Defines the set of candidate analyses for x

• EVAL(x, y) ∈ R

- Scores an analysis y in relation to x

• Parsing algorithm

- Finds the highest scoring analysis y*

Parsing Models

PCFG Parsing Dependency Parsing

GEN(x) { y | G derives x with y } { y | y is a spanning tree for x }

EVAL(x, y) PG(x, y) = PG(y) S(x, y) = ∑ S(xi, xj) for (xi, xj) in y

Algorithm CKY, Earley, … Collins, Eisner, …

Parsing as Search

• Parsing algorithms have to search the space of candidate trees

• In order to search the space of trees we have to build them

• To do parsing efficiently, we have to use “smart” search strategies

How many trees are there?

0

400

800

1200

1600

1 2 3 4 5 6 7 8

linear cubic exponential

Search Strategies

• Naive enumeration takes exponential time: O(2n)

• Dynamic programming

- Solve each subproblem only once

- Parsing in polynomial time, often cubic: O(n3)

- Examples: CKY, Earley, Eisner

• Greedy search and beam search

- Best-first search of a (small) subspace

- Parsing in sub-cubic time, often linear: O(n)

- Example: Transition-based parsing

Structured Prediction Again

• The algorithmic challenges arising in parsing are typical of structured prediction problems.

• Many of the techniques can be applied directly to other problems, for example, sequence labeling.

• This parsing course has an algorithmic focus in order to teach these techniques.

Treebanks

Syntactically annotated corpora (treebanks) have two main uses in parser development:

• The parameters of the scoring function EVAL(x, y) can be estimated from a training set

- This may involve more or less advanced machine learning methods – not a focus of this course

- Grammars may also be extracted from treebanks

• Parsing accuracy can be evaluated on a test set

- Most evaluation metrics compute partial matches between parse trees and treebank trees

- Test set must be distinct from training set – why?

Questions?

Course Information

All course information is in Studium

(TimeEdit schedule is not up to date)

Intended Learning Outcomes

At the end of the course, you should be able to

• explain the standard models and algorithms used in phrase structure and dependency parsing

• implement and evaluate some of these techniques

• critically evaluate scientific publications in the field of syntactic parsing

• design, evaluate, or theoretically analyse the syntactic component of an NLP system

Examination

Examination is continuous and based on

• 3 assignments

- 2 programming assignments

- 1 literature review

• 2 literature seminars

• 1 project

Programming Assignments

Assignment 1: PCFG parsing

• Implement CKY algorithm

• Evaluate parser on treebank data

Assignment 3: Dependency parsing

• Implement arc-eager transition system

• Implement oracle for arc-eager transition system

• Evaluate oracle on treebank data

Literature Review

Assignment 2: Literature review

• Pick two research articles about parsing

- Journal, conference or workshop papers

- Main topic should be parsing methods

• Write a 3-page report

- Summarize, analyze, critically review

Literature Seminars

Seminars to discuss an article in smaller groups

• Preparation:

- Read article

- Go through questions and discussion points

• Active participation is obligatory

- If you miss a seminar or do not participate actively, you have to hand in a written report

Project

Organization:

• Define your own project – suggestions in Studium

• Work individually or in pairs

Activities:

• Project proposal – February 26

- Assignment of supervisor

• Oral discussion (only for pairs) – March 22

• Project report – March 26

Learning Outcomes and Examination












Ass 1–3Seminars







Ass 1–3Seminars

Ass 1, 3







Ass 1–3Seminars

Ass 1, 3

Ass 2Seminars







Ass 1–3Seminars

Ass 1, 3

Ass 2Seminars

Project

Grading

Grading scales:

• Assignments and project: U–G–VG

• Literature seminars: U–G

To achieve G on the course:

• G on all assignments, seminars and project

To achieve VG on the course:

• VG on 3 assignments or 1 assignment + project

Teachers

Joakim Nivre

• Examiner and course coordinator

• Lectures, seminars, project supervision

• Assignment 2 and 3

Daniel Dakota

• Lectures, seminars, project supervision

• Assignments 1 and 2

Teaching

All teaching and tutoring is on line

• Monday and Wednesday 10.15–12:

- 10 sessions (90 min, whole class)

• Live lecture on Zoom

• Recorded lecture + Q&A

- 2 literature seminars (45 min, small groups)

• Monday 13:15–14 (from February 1):

- Tutoring for assignments and project

Teaching

All teaching and tutoring is on line

• Monday and Wednesday 10.15–12:

- 10 sessions (90 min, whole class)

• Live lecture on Zoom

• Recorded lecture + Q&A

- 2 literature seminars (45 min, small groups)

• Monday 13:15–14 (from February 1):

- Tutoring for assignments and project

CHECK SCHEDULE

Lectures

• Lectures and course books cover basic parsing algorithms in detail

• They touch on more advanced material, but you will need to read up on that independently

• To get the most out of the lectures, prepare by watching recorded lectures and reading the literature in advance

Course Workload

7.5 hp means about 200 hours work:

~ 40 h lectures (including preparation)

2 h seminars

158 h work on your own

~ 80 h assignment work (including reading)

~ 10 h seminar preparation

~ 68 h project work

Deadlines

Assignment Deadline1: PCFG parsing February 19

2: Literature review March 123: Dependency parsing March 26Project proposal February 26Project report March 26

Backup April 23

Seminars Date

Seminar 1 February 8, 10Seminar 2 March 1, 3

Reading: Course Books

Daniel Jurafsky and James H. Martin. Speech and Language Processing. 3rd edition. 2020. Chapters 12–15. Available on line at: https://web.stanford.edu/~jurafsky/slp3/.

Sandra Kübler, Ryan McDonald, and Joakim Nivre. Dependency Parsing. Morgan and Claypool. 2009. Chapters 1–4, 6. Available as e-book at: ub.uu.se.

https://web.stanford.edu/~jurafsky/slp3/

http://ub.uu.se

Reading: Articles

Seminar 1:

• Mark Johnson. PCFG Models of Linguistic Tree Representations. Computational Linguistics 24(4), 613–632.

Seminar 2:

• Joakim Nivre and Jens Nilsson. Pseudo-Projective Dependency Parsing. Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, 99–106.

Individual selection of articles for literature review (Ass 2).

Evaluation 2020

• Overall score: 3.7 / 5 (Median 4)

• What did you find valuable to the degree that it should be retained:

- Assignments interesting and suitably challenging. Flexibility in choosing and executing final project was good. Programming assignments were useful (but hard).

- The combination of implementing basic algorithms such as CKY and talking about more advanced approaches was really helpful.

- Very well organised. Great slides and lectures.

• Which course components did you find most in need of revision?

- The workload felt a bit too high for a 7.5 credit course.

- Vague assignment instructions. The first lab was the most difficult and confusing.

Changes 2021

• One assignment less than in 2021.

• Assignment 1 redesigned and simplified.

Questions?

syntactic parsing introduction - cl.lingfil.uu.se

Documents