symbolic rules extraction from trained neural networks

26/5/201226/5/2012 Koushal KumarKoushal Kumar 11

Symbolic Rules Symbolic Rules Extraction From Extraction From Trained Neural Trained Neural

Networks Networks

Koushal Kumar

M .Tech CSE

Mob: +918968939621


What are Artificial Neural Networks?What are Artificial Neural Networks?

• Artificial Neural Networks are powerful computational systems consisting of many simple processing elements connected together to perform tasks analogously to biological brains.

• They are massively parallel, which makes them efficient, robust, fault tolerant and noise tolerant

• They can learn from training data and generalize to new situations..


The limitation of Neural NetworkThe limitation of Neural Network

The major criticism against Neural Network is that The major criticism against Neural Network is that

decision given by neural networks is difficult to decision given by neural networks is difficult to

understand by a human being. Reasons for this areunderstand by a human being. Reasons for this are

Knowledge in Neural Networks are stored as real Knowledge in Neural Networks are stored as real

valuesvalues

parameters (weights and bias) of networksparameters (weights and bias) of networks

Neural Networks are unable to explain its internal Neural Networks are unable to explain its internal

processing how they come to particular decisionprocessing how they come to particular decision

This behavior makes Neural Networks Black Box in This behavior makes Neural Networks Black Box in

NatureNature


Continue..Continue..


Rules extraction From Rules extraction From Neural Networks.Neural Networks.

So to overcome the Black Box nature of Neural So to overcome the Black Box nature of Neural

Networks we need to extract rules from Neural Networks we need to extract rules from Neural

Networks so that the user can gain a better Networks so that the user can gain a better

understanding of the decision process. understanding of the decision process.

following types of rules can be extracted from following types of rules can be extracted from

neural networksneural networks

I) M OF N types rulesI) M OF N types rules

II) Fuzzy rulesII) Fuzzy rules


continue..continue..

III) III) IF THEN RULESIF THEN RULES

IV) Decision RulesIV) Decision Rules

V) First order logic rulesV) First order logic rules

From all above types of rules IF THEN RULES From all above types of rules IF THEN RULES

and Decision rules are easy to understand and Decision rules are easy to understand

then others kind of rules.then others kind of rules.


J48 Algorithm for extracting J48 Algorithm for extracting decision treesdecision trees

• J48 is an algorithm used to generate a decision tree.J48 is an algorithm used to generate a decision tree.• Developed by quinlan and most widely used decision Developed by quinlan and most widely used decision

tree induction algorithm.tree induction algorithm.• It is based upon greedy search approach i.e select the It is based upon greedy search approach i.e select the

best attribute and never looks back to reconsider early best attribute and never looks back to reconsider early choices.choices.

• It select the best attribute according to its entropy It select the best attribute according to its entropy value.value.

• More preference will be given to that attribute which More preference will be given to that attribute which has more value of entropy.has more value of entropy.


Before normalizationBefore normalization


After normalizationAfter normalization


MATLAB SimulatorMATLAB Simulator

• Matlab stands for matrix laboratory.• It integrate computation, visualization, and

programming in an easy-to-use environment.• MATLAB is a package that has been purpose-

designed to make computations easy, fast and reliable.

• Matlab can be used in math and computation, algorithm development, simulation purposes.

• MATLAB is a powerful system that can plot graphs and perform a large variety of calculations with numbers.


Weka simulatorWeka simulator

• WEKA is abbreviation of Waikato Environment for Knowledge Analysis.

• Weka is open source simulator with machine learning algorithms.

• The Weka workbench contains a collection of visualization tools and algorithms for data analysis and predictive modelling..


Training data set and its HistogramTraining data set and its Histogram


Histogram of input target valuesHistogram of input target values


Histogram of testing set as InputsHistogram of testing set as Inputs


Histogram of testing set as Target valuesHistogram of testing set as Target values


Neural networks Training with BPANeural networks Training with BPA


Best Training Performance curveBest Training Performance curve


Gradient Performance Curve After Training & Gradient Performance Curve After Training & TestingTesting


Data import window in wekaData import window in weka


J48 algorithm Output (pruned)J48 algorithm Output (pruned)


Results from pruned TreeResults from pruned Tree


J48 algorithm Output (Unpruned)J48 algorithm Output (Unpruned)


Unpruned Tree ResultsUnpruned Tree Results


Results of classification from Decision TreeResults of classification from Decision Tree


Decision Tree Extracted From Exp no 7Decision Tree Extracted From Exp no 7


The Following Rule Set Is Obtained From the above Decision The Following Rule Set Is Obtained From the above Decision

TreeTree I. Applying Remove Redundancy ConditionsI. Applying Remove Redundancy Conditions IF Children ≥ 1 AND Children >2AND Children >3 THEN Marital status =YES

Children ≥ 1 is more specific than Children >3 and Children > 2. So we remove all such conditions

Rule 1:

a) IF Current_act = NO AND Age ≤ 48.0 AND Sex = FEMALE AND Children ≤ 0 THEN Region Town

b) IF AGE > 48.0 AND Region Suburban AND Current_act = NO then Pep = NO

II. For every pair decision trees Remove redundancy rules. For example

Rule 1: IF Age ≤60 AND Salary ≤ 3500 AND Pep = NO THEN Mortage = YES

Rule 2: IF Age ≤ 50 AND Salary ≤ 3500 AND Pep = NO THEN Mortage = YES

New Rule: IF Age ≤ 50 AND Salary ≤ 3500 AND Pep = NO THEN Mortage =YES

III. Remove more specific rules. The rules with a condition set which is a superset of

another rule should be removed. For example


Continue….Continue….

• Rule 1: IF Age ≤ 60 AND Region = Rural AND Saving_ act = YES THEN Pep = NO

• Rule 2: IF Age <= 60 AND Children <= 1 AND Region =Rural AND saving act = YES THEN Pep= NO

• Rule 3: IF Region = Rural AND saving _ act =YES THEN Pep = NO

• New Rule: IF Region = Rural AND saving _ act =YES THEN Pep = NO


Comparison of J48 algorithm with Others ClassifiersComparison of J48 algorithm with Others Classifiers


Graphical comparisons of algorithmsGraphical comparisons of algorithms


Data Flow diagram of J48 algorithmData Flow diagram of J48 algorithm


Paper PublishedPaper Published

• ““Extracting Explanation From Artificial Neural Networks” is Extracting Explanation From Artificial Neural Networks” is published in International Journal of Computer Science and published in International Journal of Computer Science and Information Technologies.Information Technologies.

• ““Advanced Applications of Neural Networks and Artificial Advanced Applications of Neural Networks and Artificial Intelligence: A Review” has been selected in International journal of Intelligence: A Review” has been selected in International journal of information technologies and computer science and it will published information technologies and computer science and it will published on May June volume of journal. on May June volume of journal.

• Seminar: Published Research paper on “Symbolic Rules Extraction Seminar: Published Research paper on “Symbolic Rules Extraction From Trained Neural Networks” in two days UGC Sponsored From Trained Neural Networks” in two days UGC Sponsored National Seminar on Social Implications of Artificial Intelligence National Seminar on Social Implications of Artificial Intelligence organized in KMV College in Jalandhar. organized in KMV College in Jalandhar.


ReferencesReferences

• Olcay Boz AT & T Labs”Converting a trained neural network to Olcay Boz AT & T Labs”Converting a trained neural network to a decision tree DecText-Decision Tree Extractor.a decision tree DecText-Decision Tree Extractor.

• J.T Yao Dept of computer sci university of Regina “Knowledge J.T Yao Dept of computer sci university of Regina “Knowledge extracted from Trained neural networks whats next?extracted from Trained neural networks whats next?

• Simon Haykin ”Neural networks a Comprehensive foundation” Simon Haykin ”Neural networks a Comprehensive foundation” Pearson education(second edition).Pearson education(second edition).

• R. Davis, B.G. Buchanan, and E. Shortcliff, “Production Rules R. Davis, B.G. Buchanan, and E. Shortcliff, “Production Rules as aas a

• Representation for a Knowledge Based Consultation Progra”,Representation for a Knowledge Based Consultation Progra”,

• Artificial Intelligence, 1977, vol. 8(1), pp.15-45.Artificial Intelligence, 1977, vol. 8(1), pp.15-45.

• Knowledge Extraction from the Neural ‘Black Box’ in Knowledge Extraction from the Neural ‘Black Box’ in Ecological Monitoring Journal of Industrial and Systems Ecological Monitoring Journal of Industrial and Systems Engineering Vol. 3, No. 1, pp 38-55 Spring 2009Engineering Vol. 3, No. 1, pp 38-55 Spring 2009

symbolic rules extraction from trained neural networks

Education