Advanced Lecture on Dependency Parsing:Dynamic Oracles and Online Reordering

Miryam de Lhoneux

14 December 2017

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 1

Outline

1 Arc-Hybrid

2 Dynamic Oracles

3 Reordering

4 Dynamic Oracles and Reordering

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 2

Outline for section 1

1 Arc-Hybrid

2 Dynamic Oracles

3 Reordering

4 Dynamic Oracles and Reordering

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 3

Transition-Based Parsing with Arc-Hybrid

STACK BUFFER

Configuration:

friend home rootDrive your

Transitions:

Kuhlmann et al. (2011)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 4

Transition-Based Parsing with Arc-Hybrid

LEFT−ARC

STACK BUFFER

Configuration:

friend home rootDrive your

Transitions:

Kuhlmann et al. (2011)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 4

Transition-Based Parsing with Arc-Hybrid

LEFT−ARC

STACK BUFFER

Configuration:

friend home rootDrive your

Transitions:

Kuhlmann et al. (2011)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 4

Transition-Based Parsing with Arc-Hybrid

LEFT−ARC

STACK BUFFER

Configuration:

friend home rootDrive your

Transitions:

Kuhlmann et al. (2011)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 4

Transition-Based Parsing with Arc-Hybrid

RIGHT−ARC

LEFT−ARC

STACK BUFFER

Configuration:

friend home rootDrive your

Transitions:

Kuhlmann et al. (2011)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 4

Transition-Based Parsing with Arc-Hybrid

RIGHT−ARC

LEFT−ARC

STACK BUFFER

Configuration:

friend home rootDrive your

Transitions:

Kuhlmann et al. (2011)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 4

Transition-Based Parsing with Arc-Hybrid

RIGHT−ARC

LEFT−ARC

STACK BUFFER

Configuration:

friend home rootDrive your

Transitions:

Kuhlmann et al. (2011)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 4

Transition-Based Parsing with Arc-Hybrid

SHIFT

RIGHT−ARC

LEFT−ARC

STACK BUFFER

Configuration:

friend home rootDrive your

Transitions:

Kuhlmann et al. (2011)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 4

Transition-Based Parsing with Arc-Hybrid

SHIFT

RIGHT−ARC

LEFT−ARC

STACK BUFFER

Configuration:

friend home rootDrive your

Transitions:

Kuhlmann et al. (2011)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 4

Training a Transition-Based Parser

Figure : Figure taken from Goldberg and Nivre (2012)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 5

Training a Transition-Based Parser

Figure : Figure taken from Goldberg and Nivre (2012)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 5

Static Oracle for Arc-Hybrid

Drive your friend home **root**

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 6

Static Oracle for Arc-Hybrid

Drive your friend home **root**

[ ] [Drive your friend home **root**]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 6

Static Oracle for Arc-Hybrid

SHIFT

Drive your friend home **root**

[ Drive ] [your friend home **root**]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 6

Static Oracle for Arc-Hybrid

SHIFT

Drive your friend home **root**

[ Drive your ] [friend home **root**]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 6

Static Oracle for Arc-Hybrid

LEFT-ARC

Drive your friend home **root**

[ Drive ] [friend home **root**]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 6

Static Oracle for Arc-Hybrid

SHIFT

Drive your friend home **root**

[ Drive friend] [home **root**]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 6

Static Oracle for Arc-Hybrid

RIGHT-ARC

Drive your friend home **root**

[ Drive ] [home **root**]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 6

Static Oracle for Arc-Hybrid

SHIFT

Drive your friend home **root**

[ Drive home] [**root**]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 6

Static Oracle for Arc-Hybrid

RIGHT-ARC

Drive your friend home **root**

[ Drive ] [**root**]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 6

Static Oracle for Arc-Hybrid

LEFT-ARC

Drive your friend home **root**

[ ] [**root**]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 6

Exercise!

Write the transition sequence for this tree:

I love parsing so much ! *root*

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 7

Outline for section 2

1 Arc-Hybrid

2 Dynamic Oracles

3 Reordering

4 Dynamic Oracles and Reordering

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 8

Static Oracle: problem 1

Figure : Figure taken from Goldberg and Nivre (2012)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 9

Static Oracle: problem 1

Figure : Figure taken from Goldberg and Nivre (2012)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 9

Spurious ambiguity

Send him a flower

[ Send him ] [ a flower ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 10

Spurious ambiguity

RIGHT-ARC

Send him a flower

[ Send ] [ a flower ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 10

Spurious ambiguity

Send him a flower

[ Send him ] [ a flower ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 10

Spurious ambiguity

SHIFT

Send him a flower

[ Send him a ] [ flower ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 10

Spurious ambiguity

LEFT-ARC

Send him a flower

[ Send him ] [ flower ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 10

Spurious ambiguity

RIGHT-ARC

Send him a flower

[ Send ] [ flower ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 10

Static Oracle: problem 2

Figure : Figure taken from Goldberg and Nivre (2012)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 11

Static Oracle: problem 2

Figure : Figure taken from Goldberg and Nivre (2012)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 11

Training with a Dynamic Oracle

Figure : Figure taken from Goldberg and Nivre (2013)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 12

Training with a Dynamic Oracle

Figure : Figure taken from Goldberg and Nivre (2013)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 12

Training with a Dynamic Oracle

Figure : Figure taken from Goldberg and Nivre (2013)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 12

Dynamic Oracle for Arc-Hybrid

Drive your friend home **root**

[ Drive your ] [ friend home **root**]

Goldberg and Nivre (2013)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 13

Dynamic Oracle for Arc-Hybrid

RIGHT-ARC

Drive your friend home **root**

[ Drive your ] [ friend home **root**]

Goldberg and Nivre (2013)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 13

Dynamic Oracle for Arc-Hybrid

RIGHT-ARC

Drive your friend home **root**

[ Drive ] [ friend home **root**]

Goldberg and Nivre (2013)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 13

Dynamic Oracle for Arc-Hybrid

RIGHT-ARC

Drive your friend home **root**

cost+=1

[ Drive ] [ friend home **root**]

Goldberg and Nivre (2013)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 13

Dynamic Oracle for Arc-Hybrid

SHIFT

Drive your friend home **root**

[ Drive your ] [ friend home **root**]

Goldberg and Nivre (2013)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 13

Dynamic Oracle for Arc-Hybrid

SHIFT

Drive your friend home **root**

[ Drive your friend ] [ home **root**]

Goldberg and Nivre (2013)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 13

Dynamic Oracle for Arc-Hybrid

SHIFT

Drive your friend home **root**

cost+=1

cost+=1

[ Drive your friend ] [ home **root**]

Goldberg and Nivre (2013)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 13

Dynamic Oracle for Arc-Hybrid

Figure : Cost function

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 14

Exercise!

For the 7 first transitions in the correct transition sequence wedefined in the previous exercise, compute the cost of the otherlegal transitions.

I love parsing so much ! *root*

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 15

Outline for section 3

1 Arc-Hybrid

2 Dynamic Oracles

3 Reordering

4 Dynamic Oracles and Reordering

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 16

Arc-Hybrid Parsing with Reordering

SHIFT

RIGHT−ARC

LEFT−ARC

STACK BUFFER

Configuration:

friend home rootDrive your

Transitions:

Nivre (2009); de Lhoneux et al. (2017)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 17

Arc-Hybrid Parsing with Reordering

SWAP

SHIFT

RIGHT−ARC

LEFT−ARC

STACK BUFFER

Configuration:

friend home rootDrive your

Transitions:

Nivre (2009); de Lhoneux et al. (2017)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 17

Arc-Hybrid Parsing with Reordering

SWAP

SHIFT

RIGHT−ARC

LEFT−ARC

STACK BUFFER

Configuration:

friend home rootDrive your

Transitions:

Nivre (2009); de Lhoneux et al. (2017)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 17

Arc-Hyrid Parsing with Reordering

found best example ever1 2 4 3

(Thanks Carlos Gomez-Rodriguez for the example!)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 18

Arc-Hyrid Parsing with Reordering

found best example ever1 2 4 3

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 18

Arc-Hyrid Parsing with Reordering

found best example ever1 2 4 3

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 18

Arc-Hyrid Parsing with Reordering

SHIFT

found best example ever1 2 4 3

[ ] [ found1 best2 example4 ever3 ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 18

Arc-Hyrid Parsing with Reordering

SHIFT

found best example ever1 2 4 3

[ found1 ] [ best2 example4 ever3 ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 18

Arc-Hyrid Parsing with Reordering

SHIFT

found best example ever1 2 4 3

[ found1 best2 ] [ example4 ever3 ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 18

Arc-Hyrid Parsing with Reordering

SHIFT

found best example ever1 2 4 3

[ found1 best2 example4 ] [ ever3 ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 18

Arc-Hyrid Parsing with Reordering

SHIFT

found best example ever1 2 4 3

[ found1 best2 example4 ] [ ever3 ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 18

Arc-Hyrid Parsing with Reordering

SWAP

found best example ever1 2 4 3

[ found1 best2 ] [ ever3 example4 ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 18

Arc-Hyrid Parsing with Reordering

SHIFT

found best example ever1 2 4 3

[ found1 best2 ever3 ] [ example4 ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 18

Arc-Hyrid Parsing with Reordering

RIGHT-ARC

found best example ever1 2 4 3

[ found1 best2 ] [ example4 ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 18

Arc-Hyrid Parsing with Reordering

LEFT-ARC

found best example ever1 2 4 3

[ found1 ] [ example4 ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 18

Arc-Hyrid Parsing with Reordering

SHIFT

found best example ever1 2 4 3

[ found1 example4 ] [ ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 18

Arc-Hyrid Parsing with Reordering

RIGHT-ARC

found best example ever1 2 4 3

[ found1 ] [ ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 18

Exercise!

Write the transition sequence for this tree:

Stay away as1 far as2 possible . **root**1 2 3 6 4 5 7 8

(Example taken from the UD English treebank.)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 19

Outline for section 4

1 Arc-Hybrid

2 Dynamic Oracles

3 Reordering

4 Dynamic Oracles and Reordering

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 20

Dynamic Oracles and Reordering

Dynamic oracle for parsing with reordering:Open research question!

Partial solution:A static-dynamic oracle (de Lhoneux et al., 2017)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 21

Dynamic Oracles and Reordering

Dynamic oracle for parsing with reordering:Open research question!

Partial solution:A static-dynamic oracle (de Lhoneux et al., 2017)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 21

Dynamic Oracles and Reordering

Dynamic oracle for parsing with reordering:Open research question!

Partial solution:A static-dynamic oracle (de Lhoneux et al., 2017)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 21

Dynamic Oracles and Reordering

Dynamic oracle for parsing with reordering:Open research question!

Partial solution:A static-dynamic oracle (de Lhoneux et al., 2017)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 21

Dynamic Oracles and Reordering

Dynamic oracle for parsing with reordering:Open research question!

Partial solution:A static-dynamic oracle (de Lhoneux et al., 2017)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 21

Why is it hard?

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 22

Why is it hard?

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 22

Why is it hard?

Using information about the position of words in stack and buffer

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 22

Why is it hard?

Using information about the position of words in stack and bufferBut now words can move!

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 22

A Static-Dynamic Oracle

Figure : Figure taken from Goldberg and Nivre (2013)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 23

A Static-Dynamic Oracle

Figure : Figure taken from Goldberg and Nivre (2013)

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 23

A Static-Dynamic Oracle

found best example ever1 2 4 3

[ found1 best2] [ example4 ever3 ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 24

A Static-Dynamic Oracle

RIGHT-ARC

found best example ever1 2 4 3

[ found1 ] [ example4 ever3 ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 24

A Static-Dynamic Oracle

RIGHT-ARC

found best example ever1 2 4 3

cost+=1

cost+=1

[ found1 ] [ example4 ever3 ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 24

A Static-Dynamic Oracle

LEFT-ARC

found best example ever1 2 4 3

[ found1 ] [ example4 ever3 ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 24

A Static-Dynamic Oracle

LEFT-ARC

found best example ever1 2 4 3

cost+=1

[ found1 ] [ example4 ever3 ]

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 24

References

Miryam de Lhoneux, Sara Stymne, and Joakim Nivre. 2017. Arc-hybrid non-projective dependency parsing with astatic-dynamic oracle. In Proceedings of the 15th International Conference on Parsing Technologies.Association for Computational Linguistics, Pisa, Italy, pages 99–104.http://www.aclweb.org/anthology/W17-6314.

Yoav Goldberg and Joakim Nivre. 2012. A dynamic oracle for arc-eager dependency parsing. In Proceedings of the24th International Conference on Computational Linguistics (COLING). pages 959–976.

Yoav Goldberg and Joakim Nivre. 2013. Training deterministic parsers with non-deterministic oracles. Transactionsof the Association for Computational Linguistics 1:403–414.

Marco Kuhlmann, Carlos Gómez-Rodríguez, and Giorgio Satta. 2011. Dynamic programming algorithms fortransition-based dependency parsers. In Proceedings of the 49th Annual Meeting of the Association forComputational Linguistics (ACL). pages 673–682.

Joakim Nivre. 2009. Non-projective dependency parsing in expected linear time. In Proceedings of the JointConference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on NaturalLanguage Processing of the AFNLP (ACL-IJCNLP). pages 351–359.

Miryam de Lhoneux Advanced Lecture on Dependency Parsing 25