nlp for endangered languages: morphological analysis,...
TRANSCRIPT
-
Described our research on developing Ainu language analysis
toolkit. Described recent improvements to the previous system as
well as other enhancements made with an aim to help Ainu
language researchers.
In particular:
• enhanced POS tagger with analysis of morphological information,
• added translation support tool for Ainu language translators,
• created simple shallow parser (chunker).
In the near future we plan to:
- Compare different tokenization approaches (ex. Huang et al., 2007).
- Add other dictionaries (Nakagawa, 1995; Tamura, 1998).
- Perform a robust evaluation of the annotations with the help of
several experts and Ainu native speakers.
- Create Deep Syntactic Parser
- Add Named Entity Recognition
- Improve the system for even better performance.
- Apply to machine translation.
NLP for Endangered Languages: Morphological Analysis,
Translation Support and Shallow Parsing of Ainu Language
Michal Ptaszynski1 Kazuki Mukaichi2 and Yoshio Momouchi2
1) Kitami Institute of Technology, Department of Computer Science 2) Hokkai-Gakuen University, Department of Electronics and Information Engineering
In this research we perform computer processing of Ainu language with the use of various NLP
techniques. Ainu language is an endangered language close to extinction. At present linguists and
anthropologists make a great effort to preserve the language by analyzing and understanding it.
However, most of the work in this matter is done manually, which makes it an uphill task. Previously
we have presented POST-AL, a part-of-speech tagger for Ainu language. This paper describes recent
improvements to the system as well as other enhancements made with an aim to help Ainu language
researchers. In particular, we have enhanced the POS tagger with analysis of deeper morphological
information. We have added a translation support tool for Ainu language translators and made a first
step toward deeper syntactical analysis of Ainu language by creating a simple shallow parser.
Abstract
Tokenization DL-LSM: (Dictionary Lookup with Longest String Matching)
based on Longest Match Principle
POS Tagging CON-POST: (Contextual Part of Speech Tagging)
based on higher order HMM trained on dictionary examples (* HMM = bigrams, higher order HMM = bigrams, trigrams and longer)
Token Translation CON-ToT: (Contextual Token Translation)
translation selected specifically for the word selected in CON-POST
System Description
© Michał Ptaszyński 2013
Output Options
Linguistic Studies:
• collections of Ainu epic stories and myths (Chiri, 1978; Kayano, 1998; Piłsudski and Majewicz,2004)
• dictionaries and lexicons (Batchelor, 1905; Hattori, 1964; Chiri, 1975-1976; Nakagawa, 1995;
Kayano, 1996; Tamura, 1998; Kirikae, 2003)
• grammar descriptions (Chiri, 1974; Murasaki, 1979; Refsing, 1986; Kindaichi, 1993; Sato, 2008)
NLP-related Studies:
• attempt to transform Ainu language dictionary into an online database (Bugaeva, 2010)
• automatically gather word translations from texts (Echizen-ya et al., 2004)
• analysis / retrieval of hierarchical Ainu-Japanese translations (Azumi and Momouchi, 2009a,b)
• annotating Ainu “yukar” stories for machine translation system (Momouchi et al. 2008)
• a system for translation of Ainu topological names (Momouchi and Kobayashi 2010)
Our previous work:
• created POST-AL, a simple POS tagger for Ainu language (Ptaszynski and Momouchi, 2012)
Previous Research on Ainu language
• [default] Vertical
(typical for POS taggers) Additional options:
• -o Additional information
View Selection
References 1. Skye Hohmann. 2008. The Ainu’s modern struggle. In Worlds Watch, Vol 21., No. 6. 2. Christopher Moseley (ed.). 2010. Atlas of the World? Languages in Danger, 3rd ed. Paris,
UNESCO Publishing. Online version:http://www.unesco.org/culture/languages-atlas/ 3. Yukie Chiri. 1978. Ainu shin-yoshu. Tokyo, Iwanami Shoten. 4. Shigeru Kayano. 1998. Kayano no ainu shinwa shuusei [A collection of Ainu myths by
Kayano]. vol. 1-10, Tokyo, Heibonsha. 5. Bronisław Piłsudski (Author), Alfred F. Majewicz (Editor). 2004. The Collected Works of
Bronislaw Pilsudski: Materials for the Study of the Ainu Language and Folklore, v.3, Pt. 2: Materials for the Study of the Ainu, (Trends in Linguistics: Documentation). Mouton de Gruyter (Oct 2004)
6. Shiroo Hattori (ed.). 1964. An Ainu dialect dictionary. Tokyo, Iwanami Shoten. 7. Mashiho Chiri. 1975-1976. Bunrui ainugo jiten [A classificational dictionary of the Ainu
language], vol. 1-3, Tokyo, Heibonsha. Reprint of works from 1953, 1954 and 1962. 8. Hiroshi Nakagawa. 1995. Ainugo Chitose Hogen Jiten: The Ainu-Japanese Dictionary:
Chitose Dialect [In Japanese]. Sofukan. 9. Shigeru Kayano. 1996. Kayano Shigeru no ainugo jiten [An Ainu dictionary by Kayano
Shigeru]. Tokyo, Sanseido. 10. Suzuko Tamura. 1998. Ainugo Chitose Hogen Jiten: The Ainu-Japanese Dictionary: Saru
Dialect [In Japanese]. Sofukan.
11. Hideo Kirikae. 2003. Ainu shin-yoshu jiten: tekisuto bumpo kaisetsu tsuki (Lexicon to Yukie Chiri’s Ainu Shin-yosyu (Ainu Songs of Gods) with Text and Grammatital Notes) [In Japanese]. Sapporo: Hokkaido Daigaku Bungakubu Gengogaku.
12. Mashiho Chiri. 1974. Ainu goho gaisetu (An outline of Ainu grammar). In Chiri Mashiho chosakushuu (Collection of works by Machiho Chiri) [In Japanese], vol. 4, 3-197. Tokyo, Heibonsha. Reprint from 1936.
13. Kyoko Murasaki. 1979. Karafuto ainugo. Bunpo-hen (Sakhalin Ainu. Grammar volume) [In Japanese]. Tokyo, Kokushokan-kokai.
14. Kirsten Refsing. 1986. The Ainu language. The morphology and syntax of the Shizunai dialect. Aarhus, Aarhus University Press.
15. Kyosuke Kindaichi. 1993. Ainu yukara goho tekiyo (An outline grammar of Ainu epic stories) [In Japanese]. In Ainugogaku kogi 2 (Lectures on Ainu studies 2). Kindaichi Kyosuke zenshu. Ainugo I, v. 5, 145-366. Tokyo, Sanseidoo. Reprint from Ainu jojishi yukara no kenkyu (Research on Ainu epic stories) 2, 1-233, Tokyo: Toyo Bunko, 1931.
16. Tomomi Sato. 2008. Ainugo bunpo no kiso (The basics of Ainu grammar) [In Japanese]. Tokyo, Daigakushorin.
17. Anna Bugaeva. 2010. Internet Applications for Endangered Languages: A Talking Dictionary of Ainu. Waseda Institute for Advanced Study Research Bulletin,No.3, pp. 73-81.
18. Hiroshi Echizen-ya, Kenji Araki, Yoshio Momouchi and Koji Tochinai. 2004. Acquisition of Word Translations Using Local Focus-Based Learning in Ainu-Japanese Parallel Corpora. Lecture Notes in Computer Science, Springer-Verlag, Vol. 2945, pp.300-304.
19. Michal Ptaszynski and Yoshio Momouchi, "Part-of-Speech Tagger for Ainu Language Based on Higher Order Hidden Markov Model", Expert Systems With Applications, Vol. 39, Issue 14 (2012), pp. 11576–11582, Elsevier, 2012.
20. Yasunori Azumi and Yoshio Momouchi. 2009a. Development of Analysis Tool for Hierarchical Ainu-Japanese Translation Data [In Japanese]. Bulletin of the Faculty of Engineering at Hokkai-Gakuen University, No.36, pp.175-193.
21. Yasunori Azumi and Yoshio Momouchi. 2009b. Development of Tools for retrieving and analyzing Ainu-Japanese translation data and their applications to Ainu-Japanese machine translation system [In Japanese]. Engineering Research: The Bulletin of Graduate School of Engineering at Hokkai-Gakuen University, No.9, pp.37-58.
22. Yoshio Momouchi, Yasunori Azumi and Yukio Kadoya. 2008. Research Note: Construction and Utilization of Electronic Data for “Ainu Shin-yosyu” [In Japanese]. Bulletin of the Faculty of Engineering at Hokkai-Gakuen University, No. 35, pp. 159-171.
23. Yoshio Momouchi and Ryosuke Kobayashi. 2010. Dictionaries and Analysis Tools for the Componential Analysis of Ainu Place Names [In Japanese]. Engineering Research: The Bulletin of Graduate School of Engineering at Hokkai-Gakuen University, No.10, pp.39-49.
24. Huang, C., Simon, P., Hsieh, S., & Prevot, L. (2007). Rethinking Chinese Word Segmentation: Tokenization, Character Classification, or Word break Identification, In Proceedings of the ACL 2007 Demo and Poster Sessions, pages 69-72, 2007
25. John Batchelor. 1905. An Ainu-English-Japanese dictionary (including a grammar of the Ainu language). Tokyo Methodist Pub.House.
危機言語のためのNLP:アイヌ語の形態素解析,翻訳支援及び浅い構文解析ツール
DICTIONARY • base dictionary for POST-AL: Ainu shin-yoshu jiten (Lexicon to Yukie Chiri’s Ainu Shin-yosyu (Ainu Songs of Gods)) by Kirikae (2003) • transform dictionary information
to XML database: 1. token (word, morpheme, etc.) 2. part of speech 3. meaning (in Japanese) 4. usage examples (not for all cases) 5. reference to the story it appears in (not for all cases)
SYSTEM DESCRIPTION Tokenization
DL-LSM: (Dictionary Lookup based on Longest Match Principle)
POS Tagging CON-POST: (Contextual Part of Speech Tagging) based on higher order HMM trained on dictionary examples (* higher order HMM = bigrams, trigrams or longer)
Token Translation CON-ToT: (Contextual Token Translation) translation selected specifically for the word selected in CON-POST
POST-AL
Shallow Parser
• Ainu language is a language of Ainu people, mostly living in northern Japan.
• Population of Ainu = about 23 thousand people.
• Number of native speakers = less than hundred (Hohmann, 2008).
• Ainu language is critically endangered (Moseley, 2010).
Purpose of this research:
↓ Create language analysis toolkit including POS tagger,
translation support tool and shallow parser.
↓ Help in linguistic and language anthropology research
and support translators of Ainu texts.
Contribute to the process of reviving Ainu language.
Introduction
input
POS tagging
[CON-POST]
output
1. POS standard selection • -n Nakagawa (1995) compact
• -t Tamura (1998) sophisticated
• [default]Kirikae(2003) balanced
• -s Simple (Japanese) no POS description
• -en English English POS tag names
• -e English (Abbre-
viated) abbreviated English POS tag names (useful for
linguists)
2. Morphological Analysis
-m analysis of morphemes constituting each word
Image source: http://en.wikipedia.org/wiki/Ainu_people
Image s
ourc
e: h
ttp://w
ww
.am
azon
.co.jp
Ima
ge s
ou
rce
: http
://ww
w.a
ma
zo
n.c
o.jp
tokenization
[DL-LSM]
token translation
[CON-ToT]
• -h Horizontal (useful for language anthropologists)
Additional options:
• -ts Translation support
ORIGINAL TRANSLATION
(私はそれを見て、安心をし流れに沿うて帰つて来た.)
Conclusions and Future Work
Applied the POS information from
POS tagger to create a simple
shallow parser, or chunker, which:
1. Divides the sentence into
clauses containing the longest
possible string of consecutive
nouns and verbs and closely
related morphemes (particles,
prefixes, suffixes).
2. Divides the clauses into noun
phrases (NP) and verb phrases
(VP).
Difficult to create
a fully functional
deep parser.
Ainu language grammar is sophisticated:
• polysynthetic language (more morphemes than word stems)
• words relate to each other and change meanings in context
• incomplete model of grammar.
However, the sentence structure of Ainu language
is generally similar to Japanese (SOV).
General rules for
sentence
chunking should
work.
TAMURA STANDARD WITH MORPHOLOGICAL ANALYSIS