CAS Open APIChloris KuoY! Search Asia2009/10/17

CAS?

Content Analysis Service

成語新解

• 斷章取義 :

對一篇文 進行 詞，然後 出符合本文主要意 的關鍵字

斷章 取義

CAS Open API

• Functionality-Word Segmentation (WS)

- POS tagging - Language detection

-Keyword Extraction (KE)

Word Segmentation - Concept

• Word-A minimal-form meaning pair

• Word Segmentation-王建民首場熱身賽預定下星期二在客場出戰太空人隊

• Part-of-Speech (POS)-Noun, Verb, Adjective, etc.

王建民∕首場∕熱身賽∕預定∕下∕星期二∕在∕客場∕出戰∕太空人隊

Challenges

• Ambiguity-下雨天留客天天留我不留-不要把大便當飯吃

• Unknown Word-named entity (ex. 丁丁 )-acronym (ex. 貓纜 )-derivation (ex. 數位化 )-compound (ex. 台北火車站 )

Methodology

• Lexical rule-based-Maximum Matching

• Grammar-based

• Learning-based-HMM

Keyword Extraction

• Extract keywords representing the concept of a given context.

• Dictionary-based and statistical method

tf-idf

• A measurement widely used in text mining and information retrieval

• tf (term frequency)-The frequency of a specific term appears in a given context

• idf (inverse document frequency)-The general importance of a specific term

tf-idf (cont.)

tf-idf (cont.)

• 飛呀！飛呀！小飛俠！在那天空邊緣拚命的飛翔看看他多麼勇敢多麼堅強

為了正義他要消滅敵人為了公理他要奮鬥到底

飛呀！飛呀！飛呀！小飛俠！衝呀！衝呀！衝呀！小飛俠！

我愛科學小飛俠我愛科學小飛俠多勇敢呀！小飛俠！

tf-idf (cont.)

• 飛∕呀∕飛∕呀∕小飛俠在∕那∕天空∕邊緣∕拚命∕的∕飛翔看看∕他∕多麼∕勇敢∕多麼∕堅強

為了∕正義∕他∕要∕消滅∕敵人為了∕公理∕他∕要∕奮鬥∕到底

飛∕呀∕飛∕呀∕飛∕呀∕小飛俠衝∕呀∕衝∕呀∕衝∕呀∕小飛俠

我∕愛∕科學∕小飛俠我∕愛∕科學∕小飛俠多∕勇敢∕呀∕小飛俠

tf idf tf*idf (10-6)

呀 9 1/8394663 1.07

公理 1 1/110369 9.06

小飛俠 6 1/151774 39.53

Applications: Dynamic Content Match

Applications: Yahoo! Knowledge Plus

Applications: Yahoo! Sports

Applications: Related Search for Yahoo! News

Applications: Atlaspost

Resources

• CAS API Doc-http://tw.developer.yahoo.com/cas/

• Sample Code-PHP

- http://program.faryne.tw/2009/02/yahoo-apiphp.html-Ruby

- http://blog.xdite.net/?p=971-Java

- http://blog.cheyingwu.tw/index.php/2009/02/27/ckip-client-03/