A Genetic Algorithm using Semantic

Relations for Word Sense Disambiguation

Master’s Project

Michael Hausman

Overview

• Introduction

• Semantic Relations

• Genetic Algorithm

– Cost Function

– Mating

– Mutation

• Results

2

Introduction • Word Sense Disambiguation

– Another way of saying, “which dictionary definition is correct in context”

– Main problem of the project

• Example: “Time flies like an arrow.”

– Time (noun #1) -- an instance or single occasion for some event

– Time (noun #5) -- the continuum of experience in which events pass from the future through the present to the past

– Fly (noun #1) -- two-winged insects characterized by active flight

– Fly (verb #1) -- travel through the air; be airborne;

– Fly (verb #2) -- move quickly or suddenly

– Etc.

• Objective

– To demonstrate that a simple algorithm can achieve comparable results to the best algorithms for Word Sense Disambiguation

3

How to Disambiguate • Example: The boy walked down the hillside to the river

bank and went for a swim.

– Is bank “sloping land” or a “financial institution?”

• Humans look at the surrounding words to find the context

• Many words are related

– river bank is a compound word

– hillside and bank (sloping land) are both geological features

– river is a body of water and swim is a water sport

– Money is difficult to swim in, except for Scrooge McDuck

• A machine can use some of these relations 4

Related Work • Zhang et al. (2008)

– Use three semantic relations as the cost function in a very simple implementation of a genetic algorithm for nouns

– Took the idea of semantic relations in a genetic algorithm

• Basile et al. (2007)

– JIGSAW: Use a different algorithm and/or weights for every part of speech. Nouns and verbs use modified hypernym measures. Adjectives and Adverbs use Adapted Lesk.

– Took the idea of using different measures based on part of speech

• Patwardhan et al. (2007)

– UMND1: Only use semantic relations from WordNet for the Lexical Sample task in the SemEval 2007 competition*

– Proof that others are trying semantic relations as an unsupervised approach

• Decadt et al. (2004)

– GAMBL: Uses a genetic algorithm and several corpuses to optimize two TIMBL(memory based learning) classifiers

– The extra complication seems excessive

*Lexical sample task is word sense disambiguation for a single word in a sentence while this project focuses on every word 5

Semantic Relations

• Semantic Relations use some aspect of a word to define how closely two objects are related

• Early papers focus on one semantic relation

– Somewhat good results

• Later papers focus on two or three relations

– Better results over early papers

• Therefore, this project uses several semantic relations

– Frequency, Hypernyms, Coordinate Sisters, Domain, Synonyms, Antonyms

6

Tools/Resources

• Part of Speech Tagger

– From Tokyo University

– 97.10% accuracy on WSJ corpus

• WordNet

– Lexical database

– Machine readable dictionary

– Contains all the semantic relations in this project

• SemCor

– 20,000+ tagged words across 352 files

– Tagged with WordNet senses 7

Semantic Relations: Frequency • WordNet orders senses (definitions) by how often it appears in the corpus

used to make WordNet

• More common senses are more likely to be correct

𝐹𝑟𝑒𝑞 𝑤 =

𝑆𝑒𝑛𝑠𝑒𝐶𝑛𝑡(𝑤)

𝑇𝑜𝑡𝑎𝑙𝐶𝑛𝑡(𝑤), 𝑇𝑜𝑡𝑎𝑙𝐶𝑛𝑡 𝑤 > 0

𝑆𝑒𝑛𝑠𝑒𝑇𝑜𝑡𝑎𝑙 𝑤 − 𝑆𝑒𝑛𝑠𝑒 𝑤 − 1

𝑆𝑒𝑛𝑠𝑒𝑇𝑜𝑡𝑎𝑙(𝑤), 𝑂𝑡𝑕𝑒𝑟𝑤𝑖𝑠𝑒

SenseCnt(w): The number of times this was referenced in WordNet corpus

TotalCnt(w): The total number of references for this word in WordNet corpus

SenseTotal(w): The total number of senses for this word

Sense(w): The sense number of the word currently in use

Frequency WordNet senses of the noun bank 883/1130 = 0.78 (883) depository financial institution, bank, banking concern, banking

company -- (a financial institution that accepts deposits and channels the money into lending activities)

76/1130 = 0.07 (99) bank -- (sloping land (especially the slope beside a body of water)) -/1130 = <0.08 …

Semantic Relations: Hypernyms

• A more generic way of saying a word

• Many hypernyms in a row creates a tree

– More specific down the tree

– Length of path between words is the similarity of the subjects

• Every definition has a different hypernym tree

– Bank: financial institution vs. bank: ground next to a river

𝐻𝑦𝑝(𝑤1, 𝑤2) =2 × 𝐷

A + 𝐵 + 2 × 𝐷

LSO: most specific common hypernym

D: length from the root to LSO

A: length from w1 to LSO

B: length from w2 to LSO

Hyp(boy,girl)≈0.714

entity

3 hypernyms…

organism

person

male female

boy girl

LSO

A B

D

animal

9

Semantic Relations: Others

• All four either are in the same group (returns 1), or are not (returns 0)

1. Coordinate Sisters

– Two words that have the same hypernym are coordinate sisters

2. Domain

– The word or collection a word belongs to

– Example: words in the domain computer science

• Buffer, drive, cache, program, software

3. Synonym

– Two words that can be interchanged in a given context are synonyms

4. Antonym

– Two words that are opposites of each other in a given context are antonyms

10

Genetic Algorithms: Overview

• Based off of Darwin’s theory of Evolution

• Use a subset of total number of solutions to evolve a “good” answer over time

1. Start with a set of solutions (1st Generation)

2. Take original “parent” solutions and combine them with each other to create a new set of “child” solutions (Mating)

3. Introduce some random changes in case solutions are “stuck” or are all the same (Mutation)

4. Somehow measure the solutions to evaluate the best solution (Cost Function)

5. Repeat starting with Step 2 11

Cost Function

• Determines which solution is “better” when comparing solutions

• Most important part of genetic algorithm

• Needs to be able to determine the correct answer

– Otherwise the genetic algorithm converges on the wrong solution

• Higher value means a better solution

• This project has three main calculations

– Semantic Relation Ratio

– Sense Distribution

– Semantic Relation Distribution 12

Sense Combinations from SemCor

Correct Area

Least Frequent

Senses

Most Frequent

Senses

Separate the

Parts of

Speech

Set of Sense

Combinations

Average

Frequency Value

for All Words

Percentage of

Words Correct

Cost Function: Semantic Relation Ratio

1. Take several sense combinations for one SemCor file

2. Record the min, max, and ratio of the correct answer for each Semantic Relation

3. Find the average and standard deviation of the correct solution ratio from multiple SemCor files

4. Use four lines to transform the result for a Semantic Relation

(Avg, stdDev2)

(Avg – 2*stdDev, 0.9*stdDev2)

(Min, 0) (Max, 0)

(Avg + 2*stdDev, 0.9*stdDev2)

14

Cost Function: Sense Distribution

1. Take several SemCor files and find the average percentage of each sense

2. Find the error using the following equation. It basically adds the error for each sense and subtracts it from one.

𝑊𝑒𝑖𝑔𝑕𝑡 𝑠 = 1 − 𝐴𝑏𝑠 𝑇𝑜𝑡𝑎𝑙𝐶𝑛𝑡 𝑠𝑒𝑛𝑠𝑒 − 𝑆𝑒𝑛𝑠𝑒𝐶𝑛𝑡(𝑠𝑒𝑛𝑠𝑒)

𝑇𝑜𝑡𝑎𝑙𝐶𝑛𝑡(𝑠𝑒𝑛𝑠𝑒)𝑆𝑒𝑛𝑠𝑒

S: The current solution

Sense: The sense number

SenseCnt: Find the current number of the given sense

TotalCnt: The expected number of the given sense

Abs: Absolute value

15

Cost Function: Semantic Relation Distribution

1. Take several SemCor files and find the average semantic relation value for each sense

2. Find the error using the following equation. It basically adds the error for each sense and subtracts it from one.

𝑊𝑒𝑖𝑔𝑕𝑡 𝑆𝑒𝑚𝑅𝑒𝑙

= 1 − 𝐴𝑏𝑠 𝑆𝑒𝑚𝑂𝑝𝑡 𝑠𝑒𝑛𝑠𝑒 − 𝑆𝑒𝑚𝑉𝑎𝑙(𝑠𝑒𝑛𝑠𝑒)

𝑆𝑒𝑚𝑂𝑝𝑡(𝑠𝑒𝑛𝑠𝑒)𝑆𝑒𝑛𝑠𝑒

SemRel: The current semantic relation

Sense: The sense number

SemVal: The average semantic relation value for this sense

SemOpt: The optimal semantic relation value for this sense

Abs: Absolute value

16

Cost Function: Final Equation This equation

1. Adds the Semantic Relation contributions for the word pairs in a sliding window

2. Transforms the average with the Semantic Relation Ratio 3. Weights it with the Semantic Relation Distribution 4. Weights the total semantic relation sum with the Sense Distribution

𝐶𝑜𝑠𝑡 𝑠 = 𝑊𝑒𝑖𝑔𝑕𝑡 𝑆𝑒𝑛𝑠𝑒 ∗ 𝑊𝑒𝑖𝑔𝑕𝑡 𝑆𝑒𝑚𝑅𝑒𝑙

𝑆𝑒𝑚𝑅𝑒𝑙

∗ 𝑇𝑟𝑎𝑛𝑠𝑓𝑜𝑟𝑚 𝑆𝑒𝑚𝑅𝑒𝑙𝐸𝑞𝑛 𝑆𝑒𝑚𝑅𝑒𝑙,𝑊𝑜𝑟𝑑1,𝑊𝑜𝑟𝑑2𝑊𝑜𝑟𝑑2𝑊𝑜𝑟𝑑1

S: The current solution

Word1: Every word in the system

Word2: The surrounding 10 words before and after word1

SemRel: Every semantic relation

SemRelEqn: Use the words in the correct semantic relation equation

Transform: Find the average value then transform with the proportional equation 17

Dominant Gene Genetic Algorithm

• Apply the cost function at a gene level

– Apply to every word

– Words with high cost are most likely to be correct

• Mating functions focus on high cost words (dominant genes)

– Two variations in this project

• Mutation functions focus on low cost words (recessive genes)

– Four variations in this project 18

Mating Function: Mate Top Third 1. Find Dominant Genes

Parent 1 Words: a1 a2 a3 a4 a5 a6 …

Gene Cost: 11 12 3 24 15 4 …

Select the upper 1/3

Dominant Words: a4, a5, etc

Parent 2 Words: b1 b2 b3 b4 b5 b6 …

Gene Cost: 25 15 5 10 8 14 …

Select the upper 2/3

Dom. Genes: b1, b2, b4, b6, etc

2. Place Dominant Genes Based on First Parent

Child: * * * a4 a5 * …

3. Fill in Blanks from Second Parent

Child: b1 b2 * a4 a5 b6 …

4. Fill in any Remaining Blanks from First Parent

Child: b1 b2 a3 a4 a5 b6 … 19

Mating Function: Mate Middle Third 1. Find Dominant Genes

Parent 1 Words: a1 a2 a3 a4 a5 a6 …

Gene Cost: 11 12 3 24 15 4 …

Select the middle 1/3

Dominant Words: a1, a2, etc

Parent 2 Words: b1 b2 b3 b4 b5 b6 …

Gene Cost: 25 15 5 10 8 14 …

Select the upper 2/3

Dom. Genes: b1, b2, b4, b6, etc

2. Place Dominant Genes Based on First Parent

Child: a1 a2 * * * * …

3. Fill in Blanks from Second Parent

Child: a1 a2 * b4 * b6 …

4. Fill in any Remaining Blanks from First Parent

Child: a1 a2 a3 b4 a5 b6 … 20

Mutation Function: Random Mutation

1. Randomly pick a percentage between 0%

and 20%

2. Randomly pick that percentage of words

from the solution

3. For each of those words, randomly pick one

of the available senses

21

Mutation Function: Semantic Relation

Score Mutation 1. Start with the given semantic relation average

2. Find the percentage this average is off from the optimal semantic relation score

3. Randomly pick the comparison percentage of words

4. If semantic relation cost needs to be increased

a. If the semantic relation is frequency, then randomly pick a sense lower than the current sense. Otherwise use step b.

b. Look at each sense starting at the first sense. Stop and select that sense when the sense increases the semantic relation cost.

5. If the semantic relation cost needs to be decreased

a. If the semantic relation is frequency, then randomly pick a sense higher than the current sense. Otherwise use step b.

b. Look at each sense starting at the first sense. Stop and select that sense when the sense decreases the semantic relation cost. 22

Mutation Function: Sense Distribution

1. Find the number of words extra or missing for

each sense compared to the optimal sense

distributions

2. Find the genes that have the lowest gene cost

3. Look at each sense distribution

a. If the current sense distribution has extra

words, move the lowest cost genes to a sense

distribution needing words

23

Mutation Function: Semantic

Relation Distribution 1. Randomly pick a semantic relation

2. Compare each average semantic relation value for each sense to the optimal semantic relation value for that sense. This should result in the number of words that need to change for each sense.

3. Find the genes that have the lowest gene cost

4. Look at each sense distribution

a. If the current sense distribution has extra words, move the lowest cost genes to a sense distribution needing words 24

Measuring the Results

• Typically three different measurements to measure “accuracy” for word sense disambiguation

𝐶𝑜𝑣𝑒𝑟𝑎𝑔𝑒 = 𝑊𝑜𝑟𝑑𝑠𝐴𝑛𝑠𝑤𝑒𝑟𝑒𝑑

𝑇𝑜𝑡𝑎𝑙𝑊𝑜𝑟𝑑𝑠

𝑅𝑒𝑐𝑎𝑙𝑙 = 𝑊𝑜𝑟𝑑𝑠𝐶𝑜𝑟𝑟𝑒𝑐𝑡

𝑇𝑜𝑡𝑎𝑙𝑊𝑜𝑟𝑑𝑠

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =𝑊𝑜𝑟𝑑𝑠𝐶𝑜𝑟𝑟𝑒𝑐𝑡

𝑊𝑜𝑟𝑑𝑠𝐴𝑛𝑠𝑤𝑒𝑟𝑒𝑑

25

Comparison to Michael Billot • Colleague at University of Colorado at Colorado Springs

– Uses a Page Rank algorithm for verbs only

– Tested using a walkthrough for the game Zelda: Ocarina of Time

– Evaluated by hand

• Don’t have the walkthrough or senses used

– “Estimate” by comparing verb SemCor results to Billot’s estimated results

– Compare to the page rank paper Billot was referencing (2004)

Comparison Results Hausman

Cov. Pre. Cov. Pre.

Billot (Verbs only) 100% 46.4% 99.3% 49.6%

Page Rank (SemCor) 100% 45.1% 93.9% 61.9%

Page Rank (SenseEval 2) 100% 50.8% 95.1% 52.2%

Page Rank with Frequency (SemCor) 100% 70.1% 93.9% 61.9%

Page Rank with Frequency (SenseEval 2) 100% 63.3% 95.1% 52.2%

Comparison to Another Genetic Algorithm

• Zhang et al. (2008) also use Semantic Relations in a Genetic Algorithm

– Only focus on Nouns

– Use SemCor Results

• This project:

– Solves for every part of speech

– Is faster (sliding window vs. paragraph)

Measurement Zhang Hausman (Nouns)

Avg. Coverage 100% 89.25%

Avg. Recall 71.96% 55.45%

Avg. Precision 71.96% 62.13%

# Files >70% 51 9 27

Comparison to SemEval

• Well known Language Processing competitions

• Occurs approximately every 3 years

• Every competition has a Word Sense Disambiguation task

• Results in the Similar System column

– Primarily rely on Semantic Relations from WordNet

– Use SemCor statistics for any weighting

Competition Top Place Similar System Hausman

Cov. Pre. Cov. Pre. Rank Cov. Pre. Rank

SemEval 2 100% 69.8% 100% 37% 11/22 95.1% 52.2% 6/23

SemEval 3 100% 65.1% 97.2% 51% 16/26 96.9% 53.7% 15/27

SemEval 2007 100% 82.5% 100% 70.2% 7/14 100% 74.5% 7/15

Top SemEval Competitors • Typically use:

– Keep track of information on the sense level

– Typically use semantic relations, grammatical information, and anything else they can think of

– Use multiple training texts

– Use complicated algorithms

• SenseEval 3: GAMBL uses a genetic algorithm and several corpuses to optimize two TIMBL classifiers (memory based learning)

• SenseEval 2007: NUS-PT uses SVMs on several resources

• SenseEval 2007 (2nd Place): NUS-ML uses a three-level hierarchical bayesian model

• This project is MUCH simpler

– Only uses semantic relations between words

– Single training text

• Wanted to demonstrate that a simple algorithm can achieve comparable results to the best algorithms for word sense disambiguation 29

Questions?

30

References 1. Torsten Zesch and Iryna Gurevich, "Wisdom of Crowds Versus Wisdom of Linguists: Measuring the

Semantic Relatedness of Words," Natural Language Engineering, vol. 16, no. 01, pp. 1351-3249, 2010.

2. R. Rada, H. Mili, E. Bicknell, and M. Blettner, "Development and Application of a Metric on Semantic Nets," Systems, Man and Cybernetics, IEEE Transactions, vol. 19, no. 1, pp. 17-30, Jan/Feb 1989.

3. Claudia Leacock and Martin Chodorow, "Combining Local Context and WordNet Similarity for Word Sense Identification," in WordNet: An Electronic Lexical Database, C. Fellbaum, Ed.: MIT Press, 1998, pp. 305-332.

4. Philip Resnik, "Using Information Content to Evaluate Semantic Similarity in a Taxonomy," in Proceedings of the 14th International Joint Conference on Artificial Intelligence, 1995, pp. 448-453.

5. Zhibiao Wu and Martha Stone Palmer, "Verb Semantics and Lexical Selection," in Proceedings of the 32th Annual Meeting on Association for Computational Linguistics, Las Cruces, New Mexico, 1994, pp. 133-138.

6. Michael Lesk, "Automatic Sense Disambiguation using Machine Readable Dictionaries: How to Tell a Pine Cone from an Ice Cream Cone," in Proceedings of the 5th Annual International Conference on Systems Documentation, Toronto, Ontario, Canada, 1986, pp. 24-26.

7. Siddharth Patwardhan, Satanjeev Banerjee, and Ted Pedersen, "An Adapted Lesk Algorithm for Word Sense Disambiguation Using WordNet," in Proceedings of the Third International Conference on Computational Linguistics and Intelligent Text Processing, London, UK, 2002, pp. 136-145.

8. Siddharth Patwardhan, "Using WordNet-Based Context Vectors to Estimate the Semantic Relatedness of Concepts," in Proceedings of the EACL, 2006, pp. 1-8.

9. Siddharth Patwardhan, Satanjeev Banerjee, and Ted Pedersen, "Using Measures of Semantic Relatedness for Word Sense Disambiguation," in Proceedings of the Fourth International Conference on Intelligent Text Processing and Computational Linguistics, Mexico City, Mexico, 2003, pp. 241-257.

31

References 10. P. Basile, M. Degemmis, A. L. Gentile, P. Lops, and G. Semeraro, "The JIGSAW Algorithm for Word Sense

Disambiguation and Semantic Indexing of Documents," in Proceedings of the 10th Congress of the Italian Association for Artificial Intelligence on AI*IA 2007: Artificial Intelligence and Human-Oriented Computing, Rome, Italy, 2007, pp. 314-325.

11. Chunhui Zhang, Yiming Zhou, and Trevor Martin, "Genetic Word Sense Disambiguation Algorithm," in Intelligent Information Technology Applications, 2008, pp. 123-127.

12. George A. Miller, "WordNet: A Lexical Database for English," Communications of the ACM, vol. 38, no. 11, pp. 39-41.

13. Princeton University. (2010) About WordNet. [Online]. http://wordnet.princeton.edu

14. Matt Gerber. (2010, October) NLP Software. [Online]. http://links.cse.msu.edu:8000/members/matt_gerber/index.php/NLP_software#WordNet_API

15. Yoshimasa Tsuruoka and Jun'ichi Tsujii, "Bidirectional Inference with the Easiest-First Strategy for Tagging Sequence Data," in HLT/EMNLP, 2005, pp. 467-474.

16. Princeton University. (2011, March) The SemCor Corpus. [Online]. http://multisemcor.fbk.eu/semcor.php

17. ACL-SIGLEX. (2011, March) SENS EVAL. [Online]. http://www.senseval.org

18. Raytheon BBN Technologies, the University of Colorado, the University of Pennsylvania, and the University of Southern California's Information Sciences Institute. (2011, March) OntoNotes. [Online]. http://www.bbn.com/ontonotes/

19. Zhi Zhong, Hwee Tou Ng, and Yee Seng Chan, "Word sense disambiguation using OntoNotes: an empirical study," in Proceedings of the Conference on Empirical Methods in Natural Language Processing, Honolulu, Hawaii, 2008, pp. 1002--1010.

32

References

20. Siddharth Patwardhan, Satanjeev Banerjee, and Ted Pedersen, "UMND1: Unsupervised Word Sense

Disambiguation Using Contextual Semantic Relatedness," in Proceedings of the 4th International Workshop

on Semantic Evaluations, Prague, Czech Republic, 2007, pp. 390-393.

21. Bart Decadt Bosch, Véronique Hoste, Walter Daelemans, and Walter Van Den, "GAMBL, Genetic Algorithm

Optimization of Memory-Based WSD," in Proceedings of ACL/SIGLEX Senseval-3, 2004, pp. 108-112.

22. Rada Mihalcea and Andras Csomai, "SenseLearner: Word Sense Disambiguation for All Words in

Unrestricted Text," in Proceedings of the ACL 2005 on Interactive Poster and Demonstration Sessions,

Stroudsburg, PA, 2005, pp. 53-56.

23. Michael Hausman, "A Dominant Gene Genetic Algorithm for a Transposition Cipher in Cryptography," in CS-

571 at the University of Colorado, Colorado Springs, 2009.

24. Michael Hausman, "A Dominant Gene Genetic Algorithm for a Substitution Cipher in Cryptography," in

CS591 at the University of Colorado, Colorado Springs, 2009.

25. Michael Billot, "Combing Lexical Resources for Text Analysis of Game Walkthroughs," in Summer 2010

National Science Foundation REU, Colorado Springs, 2010.

26. B. Snyder and M Palmer, "The English all-words task," in Proceedings of the 3rd International Workshop on

the Evaluation of Systems for the Semantic Analysis of Text, Barcelona, Spain, 2004, pp. 41-43.

27. Roberto Navigli, "SemEval-2007 Task 07: Coarse-Grained English All-Words Task," in SemEval 2007, 2007.

33