Dependency Parser

자연언어처리

Dependency Structure

CFG

• Dependency structure– Head-dependent relations

– Functional categories

– 응용에직접적용하기쉬움!• Predicate-argument structure

• IE, QA, SMT 등

• Phrase structure (CFG)– Phrases

– Structural categories

최근동향

• Data-Driven Dependency Parsing

–기계학습(machine learning)을이용

• 문법대신 training data (tree bank)사용

–언어에독립적:영어 Parser를한국어에적용가능

• Grammar-driven: 언어에종속적

– Two models of dependency parsing

• Graph-based model

• Transition-based model

Graph-based Dependency Parsing

• Dependency structure를 Graph(directed Tree)로표현

– V: nodes (wi, 단어)집합

– A: arcs (wi, wj, l), l은 label: wi wj

• Dependency parsing Maximum Spanning Tree (MST) Problem : O(n2) or O(n3)

Minimum(or Maximum) Spanning Tree – 1

• Kruskal’s algorithm1 A = ∅

2 foreach v ∈ G.V:

3 MAKE-SET(v)

4 foreach (u, v) ordered by weight(u, v), increasing:

5 if FIND-SET(u) ≠ FIND-SET(v):

6 A = A ∪ {(u, v)}

7 UNION(u, v)

8 return A

Minimum(or Maximum) Spanning Tree – 2

• Prim’s algorithm

1. Initialize a tree with a single vertex, chosen arbitrarily from the graph.

2. Grow the tree by one edge: Of the edges that connect the tree to vertices not yet in the tree, find the minimum-weight edge, and transfer it to the tree.

3. Repeat step 2 (until all vertices are in the tree).

Arc weight: Machine Learning

• Arc features: f(i,j,k) : words wi and wj, label lk– head=saw & dependent=with– head-POS=Verb & dependent-POS=Preposition– in-between-POS=Noun, …– arc-distance=3– arc-direction=right– …

Learning Parameters (w)

• Averaged Perceptron, Structural SVM, …

Dependency parsing

Higher-order Model

• 성능향상

• Normal (arc-factored = first-order): O(n2) or O(n3)

• Second-order model: O(n3)

• Third-order model: O(n4)

Transition-based Dependency Parsing

• Shift-Reduce dependency parsing– Configuration: parser state– Transition: parsing action (parser state update)

• Ex. Arc-Eager parsing

Action

Example: Arc-Eager Parsing – 1

Shift

Left-Arc(nmod)

Shift

Left-Arc(sbj)

Right-Arc(pred)

Example: Arc-Eager Parsing – 2

Shift

Left-Arc(nmod)

Right-Arc(pred)

Right-Arc(obj)

Example: Arc-Eager Parsing – 3

Right-Arc(nmod)

Shift

Right-Arc(obj)

Left-Arc(nmod)

Example: Arc-Eager Parsing – 4

Right-Arc(pc)

Reduce

Left-Arc(nmod)

Reduce

Example: Arc-Eager Parsing – 5

Reduce

Reduce

Reduce

Example: Arc-Eager Parsing – 6

Right-Arc(p)

Classifier-Based Parsing

• Data-driven deterministic parsing– Oracle(Configuration) = Parser action (shift, …)– An oracle can be approximated by a classifier– A Classifier can be trained using treebank data

• SVM, Perceptron, …

– Complexity: O(n)– Feature:

Greedy Local Search

• 속도가매우빠름: O(n)• 앞단계에서오류가발생하면뒷단계로오류가전파

– 오류를복구할수있는방법이없음성능저하

• 이러한문제를해결하기위해 Beam Search 등장

Beam Search

• Beam Search를수행하여항상상위 b (beam size)개의 parsing state를유지– 앞단계에서오류(top이아닌경우)가발생 Beam에유지뒷단계에서점수상승 (다시 top) 오류복구성능향상

• 항상 b개의 Beam을유지: O(b*n)• Beam Search의학습

– Averaged Perceptron + early update

• 영어의존파서성능

– UAS 95.7%, LAS 94.1%

영어의존파서성능

UAS(test) Comp.

Charniak 00 (PCFG) 92.5 O(n5)

McDonald 06 (MST) 91.5 O(n3)

Zhang 08 combo (beam=64) 92.1 O(n2)

Koo 08 semi-sup 93.16 O(n4)

Semi-sup-SCM 09 93.79 O(n4)

Koo 10 third-order 93.04 O(n4)

Huang 10 DP (beam=16) 92.1 O(bn)

Zhang 11 rich-feat (beam=64) 92.9 O(bn)

Top-down TP 12 (beam=32) 92.6 O(bn2)

강원대: rich+semi-sup+WE feat(beam=32)

93.61 O(bn)

Deep Biaffine Attention ICLR17 95.74 O(n2)

한국어 의존 파서 예

전이기반의한국어의존구문분석: Forward

• Transition-based(Arc-Eager): O(N)

• 예: CJ그룹이1대한통운2인수계약을3체결했다4

– [root], [CJ그룹이1대한통운2 …], {}

• 1: Shift

– [root CJ그룹이1], [대한통운2인수계약을3 …], {}

• 2: Shift

– [root CJ그룹이1 대한통운2], [인수계약을3체결했다4], {}

• 3: Left-arc(NP_MOD)

– [root CJ그룹이1], [2인수계약을3체결했다4], {(인수계약을3대한통운2)}

• 4: Shift

– [root CJ그룹이1 2인수계약을3], [체결했다4], {(인수계약을3대한통운2)}

• 5: Left-arc(NP_OBJ)

– [root CJ그룹이1], [3체결했다4], {(체결했다4인수계약을3), …}

• 6: Left-arc(NP_SUB)

– [root], [(1,3)체결했다4], {(체결했다4CJ그룹이1), …}

• 7: Right-arc(VP)

– [root4 (1,3)체결했다4], [], {(root체결했다4), …}

전이기반의한국어의존구문분석: Backward

• Transition-based(Arc-Eager) + Backward: O(N)

• 예: CJ그룹이대한통운인수계약을체결했다– [root], [체결했다4인수계약을3대한통운2 CJ그룹이1], {}

• 1: Right-arc(VP)

– [root4 체결했다4], [인수계약을3 …], {(root체결했다4)}

• 2: Right-arc(NP_OBJ)

– [root4 체결했다43 인수계약을3], [대한통운2 …], {(체결했다4인수계약을3), …}

• 3: Right-arc(NP_MOD)

– [root4 체결했다43 인수계약을32 대한통운2], [CJ그룹이1], {(인수계약을3대한통운2), …}

• 4: Reduce

– [root4 체결했다43 인수계약을32], [CJ그룹이1], {(인수계약을3대한통운2), …}

• 5: Reduce

– [root4 체결했다43], [CJ그룹이1], {(인수계약을3대한통운2), …}

• 6: Right-arc(NP_SUB)

– [root4 체결했다4(1,3) CJ그룹이1], [], {(체결했다4CJ그룹이), …}

딥러닝기반한국어의존구문분석

• Transition-based + Backward

– O(N)

– 세종코퍼스의존구문변환

• Deep Learning 기반

– ReLU(> Sigmoid) + Dropout

– Korean Word Embedding

– Feature Embedding

포인터네트워크를이용한한국어의존구문분석(동계학술대회 16)

CJ그룹이 대한통운 인수계약을 체결했다

SBJOBJMOD

Deep Biaffine Attention for Neural Dependency Parsing(ICLR17 Stanford)

BERT 기반한국어의존구문분석(KCC 19)

• BERT + Bi-LSTM + Deep Biaffine attention

• 학습셋

– 세종구문분석코퍼스• 의존구조로변환

– 학습: 53,842문장

– 평가: 5,817 문장

세종코퍼스 UAS LAS

이창기[16] with MI 90.37 88.17

나승훈[6]: deep biaffine attention 91.78 89.76

박천음[5]: 포인터네트워크 92.16 89.88

안휘진[10]: deep biaffine + 스택포인터네트워크 92.17 90.08

박성식[11]: ELMo + 멀티헤드어텐션 92.85 90.65

BERT + LSTM deep biaffine (ours) 94.02 91.97

𝑠𝑗,𝑖𝑎𝑟𝑐

= 𝐡T𝑖

𝑑𝑒𝑝𝑈𝐡𝑗

ℎ𝑒𝑎𝑑, 𝑏𝑖𝑙𝑖𝑛𝑒𝑎𝑟

𝐡T𝑖𝑑𝑒𝑝

𝑈𝐡𝑗ℎ𝑒𝑎𝑑

+𝐰T𝐡𝑗ℎ𝑒𝑎𝑑

, 𝑏𝑖𝑎𝑓𝑓𝑖𝑛𝑒

𝐡𝑖𝑑𝑒𝑝

= elu FFNN 𝑑𝑒𝑝 𝐫𝑖

𝐡𝑖ℎ𝑒𝑎𝑑

= elu FFNN ℎ𝑒𝑎𝑑 𝐫𝑖