![Page 1: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/1.jpg)
Automated Building of Classic Chinese-English
Dictionary and Chinese-Hungarian Dictionary
MQP Final Presentation
Advisors: Gabor Sarkozy, WPI
Andras Kornai, MTA-Sztaki
April 28, 2015
Xiaosong Wen [email protected]
Hongbo Fang [email protected]
![Page 2: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/2.jpg)
Outline
● Introduction & Background
● Methodology
● Analysis
● Conclusion & Future Work
![Page 3: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/3.jpg)
Introduction: Bilingual Dictionary: ● Definition: a specialized dictionary used to
translate words or phrases from one
language to another
● Other usage: Cross-Language Information
Retrieval and Cross-Language Plagiarism
Detection
![Page 4: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/4.jpg)
Classic Chinese and Modern Chinese: ● Ancient Chinese: articles and poems
pre Qin and Han (121 AC)
● Modern Chinese:
after Republic of China (after 1912)
![Page 5: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/5.jpg)
Differences between Classic Chinese and Modern Chinese:
1. Ancient Chinese didn’t have
punctuation marks
![Page 6: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/6.jpg)
Differences between Classic Chinese and Modern Chinese: 2. Different meaning of words:
modern Chinese: “咸” → salty
classic Chinese: “咸” → all
modern Chinese: ‘’
![Page 7: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/7.jpg)
Differences between Classic Chinese and Modern Chinese:
3. Different ways to format words:
In classic Chinese: Every content character is a word
example:
modern Chinese: ‘妻子’ ---> wife
classic Chinese: ‘妻’ ----> wife;
‘子’----> son
two distinct words
![Page 8: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/8.jpg)
Parallel Corpus:
Hunglish;
UM-Corpus;
Chinese Text Project
![Page 9: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/9.jpg)
Sparse Matrix: • A matrix in which most of the elements are zero
• Widely used in the numerical linear algebra computations
![Page 10: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/10.jpg)
Pointwise mutual information (PMI):
Fano 1961: mutual information between particular events X and Y
𝑃𝑀𝐼 𝑋, 𝑌 = log𝑝 𝑋, 𝑌
𝑝(𝑋)𝑝(𝑌)= log
𝑝 𝑋 𝑌
𝑝(𝑋)= log
𝑝 𝑌 𝑋
𝑝(𝑌)
![Page 11: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/11.jpg)
Methodology: Classic Chinese
Download and extract parallel corpus from
ctext.org
• wget, Java, python: openCV
• HTML parsing
• 34140 zh-en sentence pair
• hundict
![Page 12: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/12.jpg)
Evaluate: clzh-en dic
● join with 100 basic set:
o overall precision is 82%
o 92% for the words with confidence above 0.2
![Page 13: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/13.jpg)
Evaluate: clzh-en dic Improve the quality:
• remove punctuation marks:
• stem English corpus:
stemming algorithm: porter2
![Page 14: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/14.jpg)
Results: improved clzh-en dic
error distribution the precision for confidence
above 0.25 is 94%,
for confidence above 0.3 is 96%
598 entries for confidence 0.25
and above
508 entries for confidence 0.3
and above
![Page 15: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/15.jpg)
Results: improved clzh-en dic
诸侯 @ feudal → feudal lord 寐 @ dawn → sleep 之 @ the → of
![Page 16: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/16.jpg)
Results: improved clzh-en dic
Recall against a selected 100 words dictionary
is 42%
manually composed the remaining translation in the
100 word basic dictionary, except for the words:
Wednesday, bread, game
![Page 17: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/17.jpg)
Methodology: Modern Chinese (simplified)
● Purchased Chinese to English Dictionary
● Professor Andras provided English to Hungarian Dictionary
● used linux “join” command to get a “raw” dictionary
![Page 18: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/18.jpg)
![Page 19: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/19.jpg)
joined dictionary
![Page 20: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/20.jpg)
Sparse Matrix:
sentence1 sentence2 sentence3 sentence4 ...
both(hu1, zh1) 0 2 0 3
only hu1 0 0 1 0
only zh1 0 0 0 1
![Page 21: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/21.jpg)
PMI:
𝑃𝑀𝐼 𝑧ℎ, ℎ𝑢 = log𝑝 𝑧ℎ, ℎ𝑢
𝑝(𝑧ℎ)𝑝(ℎ𝑢)= log
𝑛𝑏𝑁
𝑛𝑧 + 𝑛𝑏𝑁
∗𝑛ℎ + 𝑛𝑧
𝑁
= log𝑛𝑏
(𝑛𝑧+𝑛𝑏) ∗ (𝑛ℎ + 𝑛𝑏)∗ 𝑁
hu1 not hu1
zh1 n_b n_z
not zh1 n_h
![Page 22: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/22.jpg)
Example:
PMI(非洲人,afrikai)=log〖(3/(6*45)*10,365,171)〗=14.0012952777
非洲人 非洲人
afrikai 3 42
afrikai 3
![Page 23: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/23.jpg)
Evaluate: spcl-hu dic
Pairs with PMI>6, considered having high precision
0
500
1000
1500
2000
2500
3000
3500
4000
4500
Number of Pairs
Number of Pairs
![Page 24: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/24.jpg)
Errors:
Error I:
gift @ 天分 @ ad
天分: talent, gift
ad: give
silent @ 密 @ csendes.
密: secret or dense
csendes: quiet
![Page 25: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/25.jpg)
Error II: Inconsistent in part of speech
发怒, 愤, 怒, 触怒, 激怒 and 生气 are verbs;
忿, 怒气, 怒火 are nouns;
气愤, 愤怒 are adjectives.
![Page 26: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/26.jpg)
Error III: Slang expression
stupid @ 二 @ buta
bird @ 女人 @ madár
![Page 27: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/27.jpg)
Error IV: Indirect translation
life @ 春 @ energia
bread @食物@ élelmiszer
![Page 28: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/28.jpg)
Future work:
![Page 29: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/29.jpg)
Acknowledgement
● András Kornai, MTA-SZTAKI
● Gábor Sárközy, Worcester Polytechnic Institute
● Huba Bartos, MTA-SZTAKI
![Page 30: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/30.jpg)
Reference: • Fano, Robert M. 1961. Transmission of information; a statistical theory of communications. New York: MIT Press
• Liang Tian, Derek F. Wong, Lidia S.Chao, Paulo Quaresma, Franciso Oliveira, Yi Lu, Shuo Li, Yiming Wang,
Longyue Wang. “A Large English-Chinese Parallel Corpus for Statistical Machine Translation." 2014
• Grigg, Hugh. Past events in Mandarin Chinese grammar. 14 April 2013. 22 4 2015.
• Attila Balogh, Zsolt Both, András Farkas, Péter Halácsy. http://www.hunglish.hu/.
• Chinese Text Project http://ctext.org/
• Parallel Text, http://en.wikipedia.org/wiki/Parallel_text.
![Page 31: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/31.jpg)
Question?
![Page 32: MQP Final Presentation - Worcester Polytechnic Institute · MQP Final Presentation Advisors: Gabor Sarkozy, WPI Andras Kornai, MTA-Sztaki April 28, 2015 ... improved clzh-en dic](https://reader034.vdocuments.net/reader034/viewer/2022042101/5e7e51c8f21af966c92eee61/html5/thumbnails/32.jpg)
Thank you!