decision-tree induction & decision-rule induction

Decision-Tree Induction &Decision-Tree Induction &Decision-Rule InductionDecision-Rule Induction

Evgueni Smirnov

OverviewOverview

1. Instances, Classes, Languages, Hypothesis Spaces

2. Decision Trees

3. Decision Rules

4. Evaluation Techniques

5. Intro to Weka

Instances and ClassesInstances and Classes

friendly robots

A class is a set of objects in a world that are unified by a reason. A reason may be a similar appearance, structure or function.

Example. The set: {children, photos, cat, diplomas} can be viewed as a class “Most important things to take out of your apartment when it catches fire”.

head = squarebody = roundsmiling = yesholding = flagcolor = yellow

Instances, Classes, LanguagesInstances, Classes, Languages

friendly robots

head = squarebody = roundsmiling = yesholding = flagcolor = yellow

Instances, Classes, Hypothesis SpacesInstances, Classes, Hypothesis Spaces

friendly robots

smiling = yes friendly robots

The Classification TaskThe Classification Task

Decision Trees for ClassificationDecision Trees for Classification

• Decision trees

• Appropriate problems for decision trees

• Entropy and Information Gain

• The ID3 algorithm

• Avoiding Overfitting via Pruning

• Handling Continuous-Valued Attributes

• Handling Missing Attribute Values

Decision TreesDecision TreesDefinition: A decision tree is a tree s.t.:

• Each internal node tests an attribute

• Each branch corresponds to attribute value

• Each leaf node assigns a classification

Outlook

Sunny Overcast Rainy

Humidity Windy

High Normal

False True

yes yes no

Data Set for Playing TennisData Set for Playing Tennis

Outlook Temp. Humidity Windy Play

Sunny Hot High False no

Sunny Hot High True no

Overcast Hot High False yes

Rainy Mild High False yes

Rainy Cool Normal False yes

Rainy Cool Normal True no

Overcast Cool Normal True yes

Outlook Temp. Humidity Windy play

Sunny Mild High False no

Sunny Cool Normal False yes

Rainy Mild Normal False yes

Sunny Mild Normal True yes

Overcast Mild High True yes

Overcast Hot Normal False yes

Rainy Mild High True no

Decision Tree For Playing TennisDecision Tree For Playing Tennis

Outlook

Humidity Windy

High Normal

False True

yes yes no

When to Consider Decision TreesWhen to Consider Decision Trees

• Each instance consists of an attribute with discrete values (e.g. outlook/sunny, etc..)

• The classification is over discrete values (e.g. yes/no )

• It is okay to have disjunctive descriptions – each path in the tree represents a disjunction of attribute combinations. Any Boolean function can be represented!

• It is okay for the training data to contain errors – decision trees are robust to classification errors in the training data.

• It is okay for the training data to contain missing values – decision trees can be used even if instances have missing attributes.

Rules in Decision TreesRules in Decision Trees

If Outlook = Sunny & Humidity = High then Play = no

If Outlook = Sunny & Humidity = Normal then Play = yes

If Outlook = Overcast then Play = yes

If Outlook = Rainy & Windy = False then Play = yes

If Outlook = Rainy & Windy = True then Play = no

Outlook

Humidity Windy

High Normal

False True

yes yes no

Decision Tree InductionDecision Tree Induction

Basic Algorithm:

1. A the “best" decision attribute for a node N.

2. Assign A as decision attribute for the node N.

3. For each value of A, create new descendant of the node N.

4. Sort training examples to leaf nodes.

5. IF training examples perfectly classified, THEN STOP.

ELSE iterate over new leaf nodes

Decision Tree InductionDecision Tree Induction

_____________________________________Outlook Temp Hum Wind Play ---------------------------------------------------------Rain Mild High Weak yesRain Cool Normal Weak yesRain Cool Normal Strong noRain Mild Normal Weak yesRain Mild High Strong no

Outlook

____________________________________Outlook Temp Hum Wind Play -------------------------------------------------------Sunny Hot High Weak noSunny Hot High Strong noSunny Mild High Weak noSunny Cool Normal Weak yesSunny Mild Normal Strong yes

_____________________________________Outlook Temp Hum Wind Play ---------------------------------------------------------Overcast Hot High Weak yesOvercast Cool Normal Strong yes

SunnyOvercast

EntropyEntropyLet S be a sample of training examples, and

p+ is the proportion of positive examples in S and

p- is the proportion of negative examples in S.

Then: entropy measures the impurity of S:

E( S) = - p+ log2 p+ – p- log2 p

Entropy Example from the DatasetEntropy Example from the Dataset

9 9log 0.4114 14

5 5log 0.5314 14

( ) 0.94

yes no

E S p p

Outlook Temp. Humidity Windy Play

Sunny Hot High False no

Sunny Hot High True no

Overcast Hot High False yes

Rainy Mild High False yes

Rainy Cool Normal False yes

Rainy Cool Normal True no

Overcast Cool Normal True yes

Outlook Temp. Humidity Windy play

Sunny Mild High False no

Sunny Cool Normal False yes

Rainy Mild Normal False yes

Sunny Mild Normal True yes

Overcast Mild High True yes

Overcast Hot Normal False yes

Rainy Mild High True no

Information GainInformation Gain

Information Gain is the expected reduction in entropy caused by partitioning the instances according to a given attribute.

Gain(S, A) = E(S) -

where Sv = { s S | A(s) = v}

)(||||

AValuesv

v SESS

Sv1 = { s S | A(s) = v1} Sv12 = { s S | A(s) = v2}

ExampleExample

_____________________________________Outlook Temp Hum Windy Play ---------------------------------------------------------Rain Mild High False YesRain Cool Normal False YesRain Cool Normal True NoRain Mild Normal False YesRain Mild High True No

Outlook

____________________________________Outlook Temp Hum Wind Play -------------------------------------------------------Sunny Hot High False NoSunny Hot High True NoSunny Mild High False NoSunny Cool Normal False YesSunny Mild Normal True Yes

_____________________________________Outlook Temp Hum Wind Play ---------------------------------------------------------Overcast Hot High Weak YesOvercast Cool Normal Strong Yes

SunnyOvercast

Which attribute should be tested here?

Gain (Ssunny , Humidity) = = .970 - (3/5) 0.0 - (2/5) 0.0 = .970

Gain (Ssunny , Temperature) = .970 - (2/5) 0.0 - (2/5) 1.0 - (1/5) 0.0 = .570

Gain (Ssunny , Wind) = .970 - (2/5) 1.0 - (3/5) .918 = .019

ID3 AlgorithmID3 Algorithm

Informally:– Determine the attribute with the highest

information gain on the training set.– Use this attribute as the root, create a branch for

each of the values the attribute can have.– For each branch, repeat the process with subset

of the training set that is classified by that branch.

Hypothesis Space Search in ID3Hypothesis Space Search in ID3

• The hypothesis space is the set of all decision trees defined over the given set of attributes.

• ID3’s hypothesis space is a compete space; i.e., the target description is there!

• ID3 performs a simple-to-complex, hill climbing search through this space.

Hypothesis Space Search in ID3Hypothesis Space Search in ID3

• The evaluation function is the information gain.

• ID3 maintains only a single current decision tree.

• ID3 performs no backtracking in its search.

• ID3 uses all training instances at each step of the search.

Posterior Class ProbabilitiesPosterior Class ProbabilitiesOutlook

no: 2 pos and 3 negPpos = 0.4, Pneg = 0.6

False True

OverfittingOverfitting Definition: Given a hypothesis space H, a hypothesis h

H is said to overfit the training data if there exists some hypothesis h’ H, such that h has smaller error that h’ over the training instances, but h’ has a smaller error that h over the entire distribution of instances.

Reasons for OverfittingReasons for Overfitting

• Noisy training instances. Consider an noisy training example: Outlook = Sunny; Temp = Hot; Humidity = Normal; Wind = True; PlayTennis = No

This instance affects the training instances: Outlook = Sunny; Temp = Cool; Humidity = Normal; Wind = False; PlayTennis = Yes Outlook = Sunny; Temp = Mild; Humidity = Normal; Wind = True; PlayTennis = Yes

Outlook

sunny overcast rainy

Humidity Windy

high normal

false true

yes yes no

Reasons for OverfittingReasons for OverfittingOutlook

Humidity Windy

high normal

false true

yes noWindy

yes no

mild cool

Outlook = Sunny; Temp = Hot; Humidity = Normal; Wind = True; PlayTennis = NoOutlook = Sunny; Temp = Cool; Humidity = Normal; Wind = False; PlayTennis = Yes Outlook = Sunny; Temp = Mild; Humidity = Normal; Wind = True; PlayTennis = Yes

area with probablywrong predictions

Reasons for OverfittingReasons for Overfitting• Small number of instances are associated with leaf nodes. In this case it is possible that for coincidental regularities to occur that are unrelated to the actual target concept.

Approaches to Avoiding OverfittingApproaches to Avoiding Overfitting

• Pre-pruning: stop growing the tree earlier, before it reaches the point where it perfectly classifies the training data

• Post-pruning: Allow the tree to overfit the data, and then post-prune the tree.

Pre-pruning

Outlook

Humidity Windy

High Normal

False True

yes yes no

Outlook

noyes?

• It is difficult to decide when to stop growing the tree.

• A possible scenario is to stop when the leaf nodes gets less than m training instances. Here is an example for m = 5.

Validation SetValidation Set

• Validation set is a set of instances used to evaluate the utility of nodes in decision trees. The validation set has to be chosen so that it is unlikely to suffer from same errors or fluctuations as the training set.

• Usually before pruning the training data is split randomly into a growing set and a validation set.

Reduced-ErrorReduced-Error PruningPruningSplit data into growing and

validation sets.

Pruning a decision node d consists of:1. removing the subtree rooted at d.2. making d a leaf node. 3. assigning d the most common

classification of the training instances associated with d.

Outlook

Humidity Windy

high normal

false true

yes yes no

3 instances 2 instances

Accuracy of the tree on the validation set is 90%.

Reduced-Error PruningReduced-Error PruningSplit data into growing and

validation sets.

Outlook

Windyno

false true

yes no

Accuracy of the tree on the validation set is 92.4%.

Reduced-Error PruningReduced-Error PruningSplit data into growing and validation

Do until further pruning is harmful:1. Evaluate impact on validation set of

pruning each possible node (plus those below it).

2. Greedily remove the one that most improves validation set accuracy.

Outlook

Windyno

false true

yes no

Accuracy of the tree on the validation set is 92.4%.

Reduced Error Pruning ExampleReduced Error Pruning Example

Rule Post-PruningRule Post-Pruning

IF (Outlook = Sunny) & (Humidity = High)THEN PlayTennis = NoIF (Outlook = Sunny) & (Humidity = Normal)THEN PlayTennis = Yes……….

1. Convert tree to equivalent set of rules.2. Prune each rule independently of others.3. Sort final rules by their estimated accuracy, and consider them

in this sequence when classifying subsequent instances.

Outlook

Humidity Windy

normal

false true

yes yes no

Continuous Valued AttributesContinuous Valued Attributes

1. Create a set of discrete attributes to test continuous.

2. Apply Information Gain in order to choose the best attribute.

Temperature: 40 48 60 72 80 90

PlayTennis: No No Yes Yes Yes No

Temp>54 Tem>85

Missing Attribute ValuesMissing Attribute Values

Strategies:

1. Assign most common value of A among other instances belonging to the same concept.

2. If node n tests the attribute A, assign most common value of A among other instances sorted to node n.

3. If node n tests the attribute A, assign a probability to each of possible values of A. These probabilities are estimated based on the observed frequencies of the values of A. These probabilities are used in the information gain measure (via info gain) ( ).)(

AValuesv

v SESS

Summary PointsSummary Points

1. Decision tree learning provides a practical method for concept learning.

2. ID3-like algorithms search complete hypothesis space.3. The inductive bias of decision trees is preference (search)

bias.4. Overfitting the training data is an important issue in

decision tree learning.5. A large number of extensions of the ID3 algorithm have

been proposed for overfitting avoidance, handling missing attributes, handling numerical attributes, etc.

Learning Decision RulesLearning Decision Rules

• Decision Rules• Basic Sequential Covering Algorithm• Learn-One-Rule Procedure• Pruning

Definition of Decision RulesDefinition of Decision Rules

Example: If you run the Prism algorithm from Weka on the weather data you will get the following set of decision rules:

if outlook = overcast then PlayTennis = yes

if humidity = normal and windy = FALSE then PlayTennis = yes

if temperature = mild and humidity = normal then PlayTennis = yes

if outlook = rainy and windy = FALSE then PlayTennis = yes

if outlook = sunny and humidity = high then PlayTennis = no

if outlook = rainy and windy = TRUE then PlayTennis = no

Definition: Decision rules are rules with the following form:

if <conditions> then class C.

Why Decision Rules?Why Decision Rules?• Decision rules are more compact.• Decision rules are more understandable.

Example: Let X {0,1}, Y {0,1}, Z {0,1}, W {0,1}. The rules are:

if X=1 and Y=1 then 1

if Z=1 and W=1 then 1

Otherwise 0;

Why Decision Rules?Why Decision Rules?

- -- -

Decision boundaries of decision trees

- -- -

Decision boundaries of decision rules

How to Learn Decision Rules?How to Learn Decision Rules?

1. We can convert trees to rules2. We can use specific rule-learning methods

Sequential Covering AlgorithmsSequential Covering Algorithmsfunction LearnRuleSet(Target, Attrs, Examples, Threshold):

LearnedRules :=

Rule := LearnOneRule(Target, Attrs, Examples)

while performance(Rule,Examples) > Threshold, do

LearnedRules := LearnedRules {Rule}

Examples := Examples \ {examples covered by Rule}

Rule := LearnOneRule(Target, Attrs, Examples)

sort LearnedRules according to performance

return LearnedRules

IF true THEN pos

IllustrationIllustration

IF true THEN posIF A THEN pos

IF true THEN posIF A THEN pos IF A & B THEN pos

IF true THEN pos

IF A & B THEN pos

IF true THEN posIF C THEN pos

IF A & B THEN pos

IF true THEN posIF C THEN posIF C & D THEN pos

IF A & B THEN pos

Learning One RuleLearning One Rule

To learn one rule we use one of the strategies below:• Top-down:

– Start with maximally general rule– Add literals one by one

• Bottom-up:– Start with maximally specific rule– Remove literals one by one

• Combination of top-down and bottom-up: – Candidate-elimination algorithm.

Bottom-up vs. Top-downBottom-up vs. Top-down

Top-down: typically more general rules

Bottom-up: typically more specific rules

Learning One RuleLearning One Rule

Bottom-up:• Example-driven (AQ family).

Top-down:• Generate-then-Test (CN-2).

Example of Learning One RuleExample of Learning One Rule

Heuristics for Learning One RuleHeuristics for Learning One Rule

– When is a rule “good”?• High accuracy;• Less important: high coverage.

– Possible evaluation functions:• Relative frequency: nc/n, where nc is the number of correctly

classified instances, and n is the number of instances covered by the rule;

• m-estimate of accuracy: (nc+ mp)/(n+m), where nc is the number of correctly classified instances, n is the number of instances covered by the rule, p is the prior probablity of the class predicted by the rule, and m is the weight of p.

• Entropy.

How to Arrange the RulesHow to Arrange the Rules 1. The rules are ordered according to the order they have been

learned. This order is used for instance classification.

2. The rules are ordered according to their accuracy. This order is used for instance classification.

3. The rules are not ordered but there exists a strategy how to apply the rules (e.g., an instance covered by conflicting rules gets the classification of the rule that classifies correctly more training instances; if an instance is not covered by any rule, then it gets the classification of the majority class represented in the training data).

Approaches to Avoiding OverfittingApproaches to Avoiding Overfitting

• Pre-pruning: stop learning the decision rules before they reach the point where they perfectly classify the training data

• Post-pruning: allow the decision rules to overfit the training data, and then post-prune the rules.

Post-PruningPost-Pruning

1. Split instances into Growing Set and Pruning Set;

2. Learn set SR of rules using Growing Set;

3. Find the best simplification BSR of SR.

4. while (Accuracy(BSR, Pruning Set) >

Accuracy(SR, Pruning Set) ) do

4.1 SR = BSR;

4.2 Find the best simplification BSR of SR.

5. return BSR;

Incremental Reduced Error PruningIncremental Reduced Error Pruning

D1 D21

Post-pruning

Incremental Reduced Error PruningIncremental Reduced Error Pruning

1. Split Training Set into Growing Set and Validation Set;

2. Learn rule R using Growing Set;

3. Prune the rule R using Validation Set;

4. if performance(R, Training Set) > Threshold

4.1 Add R to Set of Learned Rules

4.2 Remove in Training Set the instances covered by R;

4.2 go to 1;

5. else return Set of Learned Rules

Summary PointsSummary Points

1. Decision rules are easier for human comprehension than decision trees.

2. Decision rules have simpler decision boundaries than decision trees.

3. Decision rules are learned by sequential covering of the training instances.

Model Evaluation Techniques

• Evaluation on the training set: too optimistic

Training set

Classifier

Training set

• Hold-out Method: depends on the make-up of the test set.

Training set

Classifier

Test set

• To improve the precision of the hold-out method: it is repeated many times.

• k-fold Cross Validation

Classifier

train train test

train test train

test train train

Intro to WekaIntro to Weka@relation weather.symbolic

@attribute outlook {sunny, overcast, rainy}@attribute temperature {hot, mild, cool}@attribute humidity {high, normal}@attribute windy {TRUE, FALSE}@attribute play {TRUE, FALSE}

@datasunny,hot,high,FALSE,FALSEsunny,hot,high,TRUE,FALSEovercast,hot,high,FALSE,TRUErainy,mild,high,FALSE,TRUErainy,cool,normal,FALSE,TRUErainy,cool,normal,TRUE,FALSEovercast,cool,normal,TRUE,TRUE………….

decision-tree induction & decision-rule induction

decision treesif outlook

best decision attribute

decision treesentropy

errors decision trees

missing values decision

yesif outlook

decision treeseach instance

noif outlook

Documents

fast effective rule induction (willian w. cohen)

induction and decision trees

forest rule decision

avon home rule decision

learning bias and phonological rule induction

emnlp’02 11/11/2002 ml: classical methods from ai...

decision rule 11 hoisin

induction of decision trees (idt)

design and analysis of rule induction systems

ripper fast effective rule induction

induction on decision trees

decision tree (rule induction). poll: which data mining...

global currency movement and generalized rule induction

home rule decision

a distributed problem-solving approach to rule induction

emnlp’01 19/11/2001 ml: classical methods from ai...

timu a11 classification decision tree induction

scaling up classification rule induction through parallel...

rule induction for sentence reduction -...

induction of decision trees - williams college