RULES FREQUENCY ORDER STEMMER FOR MALAY LANGUAGE

A review for Information Retrieval Subject :

GROUP MEMBERS

AHMAD KAMAL HARIDAN JAJULI P61037

NADIA BINTI KAMARUDIN P61026

ZURINA BINTI ZOLKAFFLY P61066

INTRODUCTION (WHAT IS STEMMING ALGORITHM?)

Stemming algorithm : computational procedure that will reduce all the inflectional derivational variants of words to a common form called the stem

Removing all or some of the affixes attached to the word.

Example : group,groups,grouped group

INTRODUCTION ( WHAT IS RFO? )

developed based on Rules Application Order (RAO) approach. adding a few appropriate affixes into the list

of rules, modifications of the spelling variations rules adding a few missing words into the

dictionary of root sorting in decreasing order according to the

frequency of rule’s usage in previous stemming.

MALAY AFFIXES

Infix

El Em Er in

Prefix - suffix

ber…an ber…kan di…i Ke…

anMen…i

Men…kan

Memper…i

Pen…an Per…an Se…

nya

SuffixI An Kan lah nya

Prefix

di Ke Se Ver Men Pen per ter

RULES FORMATS

PREFIX + + SUFFIX PREFIX + SUFFIX +INFIX+

DISCUSSION

Experiment Evaluate Summary

Source of translation : Quranic Collection

TOOLS

RFOList of

Affixes rules

Spelling Variation

rules -Ahmad

Root word dictionary SISDOM98

Stop words-Ahmad

EXPERIMENT ( RAO VS RAO2 VS NRAO VS RFO )

Test 1 = pr – ps – su – in Test 2 = pr – su – ps – in Test 3 = ps - pr – su – in Test 4 = ps – su – pr – in Test 5 = su – pr – ps – in Test 6 = su – ps – pr – in Test 7 = alphabetical

Legend :

pr = Prefix ps = Prefix – Suffix su = Suffix in = Infix alphabetical = the

alphabetical order of all rules

ROADMAP FOR NEW MALAY STEMMER

RAO•Ahmad Algorithm•Dict.Root word•Spelling Variation•List of Affixes

RAO2•Ahmad Algorithm•Modified Dict. Root word•Modified Speeling Variation •Modified list of rules

NRAO•Modified Algorithm•RAO2 Dict root word•RAO2 spelling variation•RAO2 list of rules

RFO •NRAO2 funtionality•Sort d creasing order of frequent rules.

COMPARISON BETWEEN THE STEMMER

ERROR FOUND IN TEST 7 FOR RFO

UNIQUE ERROR USING RFO STEMMER

GENERAL TYPES OF CONSTRAINT

Quantitative

•Minimum stem length after the removal of affix•Prefix & suffix min. Length =2•Prefix – Suffix & Infix min. Length = 3

ReCoding

•Spelling rules – spelling exception & variation•Handle by program because of complexity•Apply for Prefix , Prefix- Suffix & Suffix•First letters of root words need to be dropped when combined with these Affixes

SPELLING EXCEPTION ( RECODING )

Prefixes

Suffix

* Sample notation rules : Men + c, d, sy, t, z

RFO ALGORITHM FLOWCHART

Step 1 • Get the next word until the last word

Step 2 •Check the word against the dictionary; if it appears in the dictionary, the word is the root word and goto Step-1;

Step 3 •Step-3: Get the next rule; if no more rules available, the word is considered as a root word and goto Step-1;

CONT…

Step 4 •Step-4: Apply the rule on the word to get a stem;

Step 5 •Perform recoding for prefix spelling exceptions and check the dictionary;

Step 6 •If the stem appears in the dictionary, the stem is the root of the word and goto Step-1; else goto Step-7;

CONT…

Step 7 •Check the stem from Step-4 for spelling variations and check the dictionary;

Step 8 •If the stem appears in the dictionary, the stem is the root of the word and goto Step-1; else goto Step-9;

Step 9 •Perform recoding for suffix spelling exceptions and check the dictionary;

Step 10 •If the stem appears in the dictionary, the stem is the root of the word and goto Step-1; else goto Step-3;

RFO EVALUATION

Compression Achived

Reduce Error

Stemmer

Distinct Word

Compression

RAO 2667 61.4%

RFO 2602 62.3%

• RFO is an improvement because it returns less distinct words and higher compression percentage

Stemmer

Number of errors

Percentage of errors

RAO 93 4.4%

RFO 30 1.4%

• RFO also recorded the least amount of errors

SUMMARY

From the experiments performed, it is found that :

- The order of rules to use is not necessary to follow any order of affixes types.

-Let the rules sorted in alphabetical order for the first pass, and for the second pass, sort the rules according to usage frequency of each rule.

- Experiments showed that the new approaches in stemming are better than other Malay stemmer as RAO by Ahmad.