deep learning and lexical, syntactic and semantic...

Deep Learning and Lexical, Syntacticand Semantic Analysis

Wanxiang Che (HIT)Yue Zhang (SUTD)

CCL 2016 Tutorial 12016-10-14

Part 3: Greedy Decoding

CCL 2016 Tutorial 22016-10-14

Part 3.1: Greedy Decoding for Tagging

CCL 2016 Tutorial 32016-10-14

WindowApproach

• Tasks– POS tagging, Chunking, NER, SRL

• Tagonewordata time• Feedafixed-sizewindowoftextaroundeachwordtotag

• Features– Words, POS tags, Suffix, Cascading, …

RonanCollobert,JasonWeston,LéonBottou,MichaelKarlen,Koray Kavukcuoglu,andPavel Kuksa.2011.NaturalLanguageProcessing(Almost)fromScratch.J.Mach.Learn.Res.12,2493-2537.

CCL 2016 Tutorial 42016-10-14

WindowApproach

• Worksfineformosttasks• Howtodealwithlong-rangedependencies?

– E.g.inSRL,theverbofinterestmightbeoutsidethewindow!

RonanCollobert,JasonWeston,LéonBottou,MichaelKarlen,Koray Kavukcuoglu,andPavel Kuksa.2011.NaturalLanguageProcessing(Almost)fromScratch.J.Mach.Learn.Res.12,2493-2537.CCL 2016 Tutorial 52016-10-14

SentenceApproach

• Tagonewordata time– addextrarelative position features

• Feedthewholesentencetothenetwork• Convolutionstohandlevariable-lengthinputs• Maxovertimetocapturemostrelevantfeatures

– Outputsafixed-sizedfeaturevector


SentenceApproach


Results

• Window approach: POS, Chunking,NER• Sentence approach: SRL• WLL: Word-LevelLog-Likelihood


CCGSupertagging

Lewis, M., & Steedman, M. (2014). Improved CCG Parsing with Semi-supervised Supertagging. TACL.Xu, W., Auli, M., & Clark, S. (2015). CCG Supertagging with a Recurrent Neural Network. ACL.Lewis, M., Lee, Kenton., & Zettlemoyer, L. (2016). LSTM CCG Parsing. NAACL.

CCL 2016 Tutorial 92016-10-14

CCGSupertagging

• NoPOSfeature

• Reducewordsparsity

• Globalcontextinformation


CCL 2016 Tutorial 102016-10-14

CCGSupertagging


CCL 2016 Tutorial 112016-10-14

CCGSupertagging

• Parsingresults


CCL 2016 Tutorial 122016-10-14

Part 3.2: Greedy Search for ConstituentParsingwithRNN

CCL 2016 Tutorial 132016-10-14

Constituent Parsing with RecursiveNN

• Our goal

RichardSocher,CliffLin,AndrewY.Ng,andChristopherD.Manning.ParsingNaturalScenesandNaturalLanguagewithRecursiveNeuralNetworks.ICML2011CCL 2016 Tutorial 142016-10-14

RecursiveNN

• Inputs– Two candidate children’s representations

• Outputs– The semantic representation if the two nodes are merged– Score of how plausible the new node would be


RNN Definition

!score = 𝑈*𝑝

𝑝 = tanh(𝑊𝑐3𝑐4

+ 𝑏)

where W at all nodes of the tree are thesame


Parsing a sentence with an RNN

CCL 2016 Tutorial 172016-10-14


CCL 2016 Tutorial 182016-10-14


CCL 2016 Tutorial 192016-10-14


CCL 2016 Tutorial 202016-10-14

Discussion on Simple RNN

• The composition function with single weight matrix isthe same for all categories, punctuation, etc.

• It could capture some phenomena but not adequate formore complex, higher order composition

CCL 2016 Tutorial 212016-10-14

Solution: Syntactically-Untied RNN (SU-RNN)

• Intuition– Condition the composition function on the syntacticcategories

• Allows for different composition functions for pairs ofsyntactic categories, e.g. Adv + AdjP, VP + NP

RichardSocher,JohnBauer,ChristopherD.ManningandAndrewY.Ng.ParsingwithCompositionalVectorGrammars.ACL2013.CCL 2016 Tutorial 222016-10-14

Compositional Vector Grammars (CVG)

• PCFG + SU-RNN• PCFG

– Produce: k-best parsing trees

• SU-RNN– Re-ranking with SU-RNN

RichardSocher,JohnBauer,ChristopherD.ManningandAndrewY.Ng.ParsingwithCompositionalVectorGrammars.ACL2013.CCL 2016 Tutorial 232016-10-14

Part 3.3: Transition-basedDependencyParsingwithGreedy Search

CCL 2016 Tutorial 242016-10-14

DependencyParsing

•NeuralMaltParser

Chen, D., & Manning, C. D. (2014). A Fast and Accurate Dependency Parser using Neural Network. ACL.CCL 2016 Tutorial 252016-10-14

DependencyParsing


DependencyParsing

•ZPar features(ZhangandNivre,ACL2011)


DependencyParsing


DependencyParsing

•ChenandManningwithcombinedfeatures

Zhang, M., & Zhang, Y. (2015). Combining Discrete and Continuous Features for Deterministic Transition-based Dependency Parsing. EMNLP. CCL 2016 Tutorial 292016-10-14

DependencyParsing

•ChenandManningwithcombinedfeatures

Zhang, M., & Zhang, Y. (2015). Combining Discrete and Continuous Features for Deterministic Transition-based Dependency Parsing. EMNLP. CCL 2016 Tutorial 302016-10-14

DependencyParsing

•ChenandManningwithricherfeatures

•11 more LSTMs

Keperwasser, E., & Goldberg, Y. (2016). Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations. TACL.

DependencyParsing

Keperwasser, E., & Goldberg, Y. (2016). Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations. TACL. CCL 2016 Tutorial 322016-10-14

DependencyParsing

•Chen andManningwithless features

Dyer, C., Ballesteros, M., Ling, W., Matthews, A., & Smith, N. A. (2015). Transition-Based Dependency Parsing with Stack Long Short-Term Memory. ACL. CCL 2016 Tutorial 332016-10-14

DependencyParsing


DependencyParsing

Dyer, C., Ballesteros, M., Ling, W., Matthews, A., & Smith, N. A. (2015). Transition-Based Dependency Parsing with Stack Long Short-Term Memory. ACL.

Chinese parsing results (CTB5)English parsing results (SD)

CCL 2016 Tutorial 372016-10-14

DependencyParsing

•Dyeretal.withcharacterbasedwordvector

Ballesteros, M., Dyer, C., & Smith, N. A. (2015). Improved Transition-Based Parsing by Modeling Characters instead of Words with LSTMs. EMNLP. CCL 2016 Tutorial 382016-10-14

DependencyParsing


Dependency Parsing


Part 3.4: SequencetoSequencewithGreedy Search

CCL 2016 Tutorial 422016-10-14

ConstituentParsing

•Sequence tosequence

Vinyals, O., Kaiser, L., Koo, T., Petrov, S., & Sutskever, I. (2015). Grammar as a Foreign Language. ICLR.CCL 2016 Tutorial 432016-10-14

ConstituentParsing


deep learning and lexical, syntactic and semantic...

Documents