Fuzzy Genetics-based Fuzzy Genetics-based Machine Learning Machine Learning

AlgorithmsAlgorithms

Presented by Vahid JazayeriPresented by Vahid Jazayeri

outlineoutline

Introduction to fuzzy sets and fuzzy Introduction to fuzzy sets and fuzzy rulesrules

Introduction to GBML algorithmsIntroduction to GBML algorithms GBML AlgorithmsGBML Algorithms

• MichiganMichigan• PittsburghPittsburgh• HybridHybrid

Fuzzy SetsFuzzy Sets

A Fuzzy set includes a small number of A Fuzzy set includes a small number of linguistic valueslinguistic values• {Small, Large, Medium}{Small, Large, Medium}• {Small, Medium-small, Medium, Large, {Small, Medium-small, Medium, Large,

Medium-large}Medium-large} Fuzzy values are assigned to variables Fuzzy values are assigned to variables

instead of exact numerical valuesinstead of exact numerical values Numerical values should be normalized Numerical values should be normalized

into [0,1]into [0,1]

Cont.Cont. Each number in [0,1] can be each one of fuzzy Each number in [0,1] can be each one of fuzzy

values but with a degree of membershipvalues but with a degree of membership

ExampleExample• 0 is 100% small but 0% large.0 is 100% small but 0% large.• 0.125 is 50% small, 50% medium-small, but 0% other 0.125 is 50% small, 50% medium-small, but 0% other

valuesvalues

Fuzzy RulesFuzzy Rules Answers to classification problemsAnswers to classification problems

• m classes existm classes exist• n variable existsn variable exists• A fuzzy rule determines a class according to n variable A fuzzy rule determines a class according to n variable

and with a certainty gradeand with a certainty grade

• RRjj is the is the jj--th ruleth rule

• XXii is is i-i-th variableth variable

• AAjiji is the fuzzy value of is the fuzzy value of XXii in the in the j-j-th Ruleth Rule• CCjj is the class of the rule is the class of the rule RRjj

• CFCFjj is the certainty factor of the rule is the certainty factor of the rule RjRj

Cont.Cont. Don’t-care is also a fuzzy valueDon’t-care is also a fuzzy value Every number is 100% Don’t-careEvery number is 100% Don’t-care

For shortening the size of rules.For shortening the size of rules. If in a rule If in a rule XiXi is Don’t-care it means we can omit is Don’t-care it means we can omit

that checking from the rule, so the size of rule is that checking from the rule, so the size of rule is shortened.shortened.

GBML AlgorithmsGBML Algorithms Genetics-Based Machine Learning Genetics-Based Machine Learning

algorithms are a simulation of natural algorithms are a simulation of natural genetic phenomena.genetic phenomena.

Each individual is a string of some Each individual is a string of some meaningful units (characters, bits, …).meaningful units (characters, bits, …).

A population of individuals generates new A population of individuals generates new populations by the operators:populations by the operators:• CrossoverCrossover

The mixing of two strings and generating two other The mixing of two strings and generating two other onesones

• MutationMutation A lowly probable change in some characters of a A lowly probable change in some characters of a

stringstring Old, weak individuals are replaced by new, Old, weak individuals are replaced by new,

stronger individualsstronger individuals

Fuzzy GBML AlgorithmsFuzzy GBML Algorithms Individuals are fuzzy rulesIndividuals are fuzzy rules Fuzzy rules are coded as strings of the Fuzzy rules are coded as strings of the

following characters: following characters: • 1: small1: small• 2: medium-small2: medium-small• 3: medium3: medium• 4: medium-large4: medium-large• 5: large5: large• #: don’t-care#: don’t-care

Cont.Cont.

Example:Example:

Says that if x1 is small, x2 is large, x3 is don’t-Says that if x1 is small, x2 is large, x3 is don’t-care and x4 is medium-smallcare and x4 is medium-small

There is no need to encoding the class or certainty There is no need to encoding the class or certainty gradegrade• They can be calculated dynamicallyThey can be calculated dynamically

Each encoded rule has a fitness value determined Each encoded rule has a fitness value determined by a fitness function for discriminating good rulesby a fitness function for discriminating good rules

11 55 ## 22

Heuristic Rule Generation Heuristic Rule Generation ProcedureProcedure

An encoded rule is prepared.An encoded rule is prepared. It is wanted to figure out the class and It is wanted to figure out the class and

certainty grade (CF) of the rule Rj certainty grade (CF) of the rule Rj according to m experienced predefined according to m experienced predefined training patternstraining patterns

1) Calculate the compatibility of each 1) Calculate the compatibility of each training pattern Xtraining pattern Xpp with R with Rjj::

• MMAAjiji if the membership function of if the membership function of AAjiji

Cont.Cont. 2) Calculate the sum of compatibility grades for each class2) Calculate the sum of compatibility grades for each class

Among c classes of the problem, find the class Cj for Rule Rj Among c classes of the problem, find the class Cj for Rule Rj that has the maximum value of Bthat has the maximum value of BClass hClass h (Rj)(Rj)

The certainty grade of Rj can be calculated as follows:The certainty grade of Rj can be calculated as follows:

Fuzzy ReasoningFuzzy Reasoning A set of Fuzzy rules are availableA set of Fuzzy rules are available A set of normalized values for n variables A set of normalized values for n variables

of a pattern is in handof a pattern is in hand It is wanted to classify the patternIt is wanted to classify the pattern The winner rule of the rule-set is the rule The winner rule of the rule-set is the rule

Rj for which:Rj for which:

If Rj is the winner rule then it says that the If Rj is the winner rule then it says that the pattern is in class Cpattern is in class Cjj with the probability of with the probability of CFCFjj

Michigan AlgorithmMichigan Algorithm1.1. Generate a set of N fuzzy rules as the initial population of Generate a set of N fuzzy rules as the initial population of

individualsindividuals2.2. Evaluate each rule with a fitness function.Evaluate each rule with a fitness function.3.3. Select N/2 pair of individualsSelect N/2 pair of individuals4.4. Each pair generate two new individual as the result of Each pair generate two new individual as the result of

crossovercrossover5.5. Each newly-born individual may change with a small Each newly-born individual may change with a small

probability by mutation.probability by mutation.Now N new individuals are preparedNow N new individuals are prepared

6.6. Evaluate the new individualsEvaluate the new individuals7.7. Replace the best new individuals (with the highest fitness Replace the best new individuals (with the highest fitness

values) with the old bad individuals of the previous values) with the old bad individuals of the previous population (those with the lowest fitness values) population (those with the lowest fitness values) the the new population is now readynew population is now ready

8.8. If stopping condition is not satisfied return to step 2If stopping condition is not satisfied return to step 2

ExampleExample

Evaluation (fitness function)Evaluation (fitness function)

1.1. Set the fitness of each rule to zeroSet the fitness of each rule to zero

2.2. For each training pattern find the For each training pattern find the winner rule for that pattern.winner rule for that pattern.

If the class of the winner rule is the If the class of the winner rule is the same as the class of the pattern, then same as the class of the pattern, then add one score to the fitness of that add one score to the fitness of that rulerule

Else give a negative score to its fitness Else give a negative score to its fitness or do nothing.or do nothing.

Pittsburgh AlgorithmPittsburgh Algorithm

The procedure of Pittsburgh The procedure of Pittsburgh algorithm is the same as that of algorithm is the same as that of Michigan but with partial differencesMichigan but with partial differences

Each individual in Michigan is a rule Each individual in Michigan is a rule and the population is a collection of and the population is a collection of rulesrules

Each individual in Pittsburgh is a rule-Each individual in Pittsburgh is a rule-set and the population is a collection set and the population is a collection of rule-setsof rule-sets

Cont.Cont. The encoding of each rule-set is the The encoding of each rule-set is the

concatenation of its encoded rulesconcatenation of its encoded rules Below is an encoded set of 3 rules. Each rule of 3 Below is an encoded set of 3 rules. Each rule of 3

variablesvariables

The fitness of each rule-set is determined by the The fitness of each rule-set is determined by the sum of the fitness of its rulessum of the fitness of its rules

Final answer in Michigan is the whole population Final answer in Michigan is the whole population but here is the best rule-set with the highest but here is the best rule-set with the highest fitnessfitness

Comparing two methodsComparing two methods

HybridHybrid The basis of the algorithm is PittsburghThe basis of the algorithm is Pittsburgh A mutation in Pittsburgh means a probable A mutation in Pittsburgh means a probable

change in an individual, that is a change in a rule-change in an individual, that is a change in a rule-set. A change in a rule-set can be done by set. A change in a rule-set can be done by executing Michigan on that rule-set, So using executing Michigan on that rule-set, So using Michigan as mutation operator in Pittsburgh is Michigan as mutation operator in Pittsburgh is called called Hybrid AlgorithmHybrid Algorithm

Michigan yields good rules but not necessarily Michigan yields good rules but not necessarily good rule-sets.good rule-sets.

Pittsburgh yields good rule-sets but not Pittsburgh yields good rule-sets but not necessarily good rules.necessarily good rules.

But hybrid algorithm enjoys goods rule-sets But hybrid algorithm enjoys goods rule-sets with good ruleswith good rules

resourcesresources

H. Ishibuchi, T. Nakashima, H. Ishibuchi, T. Nakashima, Data Mining Data Mining techniques, chapter 40, techniques and techniques, chapter 40, techniques and applications of genetic algorithm-based applications of genetic algorithm-based methods for desiring compact fuzzy methods for desiring compact fuzzy classification systems, classification systems, 1999.1999.

H. Ishibuchi, T. Nakashima, and T. Kuroda, H. Ishibuchi, T. Nakashima, and T. Kuroda, A Hybrid Fuzzy Genetics-based Machine A Hybrid Fuzzy Genetics-based Machine Learning Algorithm: Hybridization of Learning Algorithm: Hybridization of Michigan approach and Pittsburgh Michigan approach and Pittsburgh approach, approach, 2000.2000.

ThanksThanks

Any question (?)Any question (?)