CLASSIFICATION: Ensemble Methods

Combines multiple models Construct multiple classifiers from

training set Aggregate their predictions on

testing set

Meta-algorithm

CLASSIFICATION: Ensemble Methods

Improves stability and accuracy

Reduces variance

Helps avoid overfitting

Compensates for poor learning algorithms

Uses more computation

ENSEMBLE METHODS: Examples

Bagging (bootstrap aggregation) Bagging with MetaCost

Random forests

Boosting

Stacked generalization Usually used on different learning

algorithms

Bayesian model combination

ENSEMBLE METHODS: Bagging

Randomly create samples (with replacement) from a data set

Create classifiers (same type) for each sample

Run classifiers on testing sample

Use majority voting to determine classification of testing sample

ENSEMBLE METHODS: Bagging with MetaCost

Used when each model can output probability estimates

Probability estimates used to obtain expected cost of each prediction

Classifies training instances to minimize the expected cost

Learns new classifier

ENSEMBLE METHODS: Random Forests

Modification of applying bagging to tree learners

Uses only random subsets of features at each split

Promotes tree diversity

ENSEMBLE METHODS: Boosting

Seeks models that complement one another

Combines models of same type

New models constructed to better handle those instances incorrectly handled by previous models – focuses on hard to classify examples

Uses weighted averaging often adaptively

ENSEMBLE METHODS: Stacked Generalization

Introduced by David Wolpert, 1992

Other algorithms trained from training set

Stacking (“level-1”) algorithm uses predicitions from base (“level-0”) algorithms as inputs

ENSEMBLE METHODS: Stacked Generalization

Employs j-fold cross validation of training set

Train and test each of the level-0 algorithms using the split training data to create the level-0 models

Test each model on each split to create level-1 data

ENSEMBLE METHODS: Stacked Generalization

ENSEMBLE METHODS: Stacked Generalization

Can be used for both supervised and unsupervised learning

Best performers in Netflix competition were forms of stacked generalization

Can even create multiple levels of stacking(“level-2”, etc.)

ENSEMBLE METHODS: Stacked Generalization

Best performers in Netflix competition were forms of stacked generalization

Can even create multiple layers (“stacked stacking”)

Works best with class probabilities (Tang and Witten, 1999)

ENSEMBLE METHODS: Bayesian Model Combination

Built upon Bayes Model Averaging and Bayes Optimal Classifier

Bayes Optimal Classifier Ensemble (using Bayes’ rule) of all

hypotheses in hypothesis space

On average, it is the ideal ensemble

ENSEMBLE METHODS: Bayesian Model Combination

Bayes Model Averaging Approximates Bayes optimal classifer Samples from hypothesis space

Monte Carlo sampling Tends to promote overfitting Performs worse in practice than simpler

techniques (eg bagging)

ENSEMBLE METHODS: Bayesian Model Combination

Bayes Model Combination Correction to Bayes Model Averaging Uses model weightings to create samples Overcomes drawback of BMA giving weight

to single model Better performance than BMA or bagging