Upload File

cse 473 ensemble learning. © cse ai faculty 2 ensemble learning sometimes each learning technique...

  • Home
  • Documents
  • CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect
28
CSE 473 Ensemble Learning

Upload: allen-floyd

Post on 21-Jan-2016

215 views

Category:

Documents


0 download

Report

Tags:

TRANSCRIPT

Page 1: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

CSE 473

Ensemble Learning

CSE 473

Ensemble Learning

Page 2: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

© CSE AI Faculty 2

Ensemble Learning

• Sometimes each learning technique yields a different hypothesis (or function)

• But no perfect hypothesis…

• Could we combine several imperfect hypotheses to get a better hypothesis?

Page 3: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

© CSE AI Faculty 3

Example

this line is one simple classifier saying that everything to the left is + and everything to the right is -

Combining 3 linear classifiers

More complex classifier

Page 4: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

© CSE AI Faculty 4

• Analogies:  Elections combine voters’ choices to pick a good

candidate (hopefully)  Committees combine experts’ opinions to make

better decisions  Students working together on Othello project

• Intuitions:Individuals make mistakes but the “majority” may be less likely to

  Individuals often have partial knowledge; a committee can pool expertise to make better decisions

Ensemble Learning: Motivation

Page 5: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

© CSE AI Faculty 5

Technique 1: Bagging

• Combine hypotheses via majority voting

Page 6: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

© CSE AI Faculty 6

Bagging: Analysis

Error probability went down from 0.1 to 0.01!

Page 7: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

© CSE AI Faculty 7

Weighted Majority Voting

• In practice, hypotheses rarely independent

• Some hypotheses have less errors than others all votes are not equal!

• Idea: Let’s take a weighted majority

Page 8: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

© CSE AI Faculty 8

Technique 2: Boosting

• Most popular ensemble learning technique  Computes a weighted majority of hypotheses  Can “boost” performance of a “weak learner”

• Operates on a weighted training set  Each training example (instance) has a “weight”  Learning algorithm takes weight of input into account

• Idea: when an input is misclassified by a hypothesis, increase its weight so that the next hypothesis is more likely to classify it correctly

Page 9: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

© CSE AI Faculty 9

Boosting Example with Decision Trees (DTs)

training casecorrectlyclassified

training casehas large weightin this round

this DT has a strong vote.

Output of hfinal is weighted majority of outputs of h1,…,h4

Page 10: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

© CSE AI Faculty 10

AdaBoost Algorithm

Page 11: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

© CSE AI Faculty 11

AdaBoost Example

Original training set D1 : Equal weights to all training inputsGoal: In round t, learn classifier ht that minimizes error with respect to weighted training setht maps input to True (+1) or False (-1)

Taken from “A Tutorial on Boosting” by Yoav Freund and Rob Schapire

Page 12: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

© CSE AI Faculty 12

AdaBoost ExampleROUND 1

Misclassified Increase weights

z1 = 0.42

Page 13: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

© CSE AI Faculty 13

AdaBoost Example

ROUND 2

z2 = 0.65

Page 14: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

© CSE AI Faculty 14

AdaBoost Example

ROUND 3

z3 = 0.92

Page 15: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

© CSE AI Faculty 15

AdaBoost Example

hfinal

sign(x) = +1 if x > 0 and -1 otherwise

Page 16: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

Example 1: Semantic Mapping

Page 17: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

Motivation

• Human-Robot interaction:• User: “Go to the corridor”

RoomRoom

CorridorCorridor

DoorwayDoorway

Page 18: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

Shape

RoomRoom

Page 19: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

Observations

RoomRoom DoorwayDoorway

Page 20: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

Observations

RoomRoomCorridorCorridor DoorwayDoorway

Page 21: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

Simple Features

• Gap = d > θ• f = # Gaps

•Minimum

• f =Area • f =Perimeter • f = d

•d•di

N

1f

•d

• f = d•

•d

•Σ di

Page 22: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

Experiments

Training (top) # examples: 16045

•Test (bottom)•# examples: •18726•classification:

•93.94%

•Building 079 Uni. Freiburg

•Room•Room•Corridor•Corridor •Doorway•Doorway

Page 23: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

Application to a New Environment

Training map

Intel Research Lab in Seattle

Page 24: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

Application to a New Environment

Training map

Intel Research Lab in Seattle

•Room•Room•Corridor•Corridor •Doorway•Doorway

Page 25: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

Example 2: Wearable Multi-Sensor Unit

© CSE AI Faculty

•Battery

•Camera (on ribbon cable)

•GPS receiver

•iMote2 +two sensor boards

•Microphone •Camera

•Light •sensors

•2 GB •SD card

•Indicator LEDs

• Records 4 hours of audio, images (1/sec), GPS, and sensor data (accelerometer, barometric pressure, light intensity, gyroscope, magnetometer)

Page 26: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

Data Stream

© CSE AI Faculty

Courtesy of G. Borriello

Page 27: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

Activity Recognition Model

© CSE AI Faculty

• Boosted cassifiers [Lester-Choudhury-etAl: IJCAI-05]• Virtual evidence boosting [Liao-Choudhury-Fox-Kautz:

IJCAI-07]

•Accuracy: 88% activities, 93% environment

Page 28: CSE 473 Ensemble Learning. © CSE AI Faculty 2 Ensemble Learning Sometimes each learning technique yields a different hypothesis (or function) But no perfect

Boosting• Extremely flexible framework• Handles high-dimensional continuous data• Easy to implement

• Limitation:  Only models local classification problems

28© CSE AI Faculty


L25: Ensemble learning - Texas A&M Universityresearch.cs.tamu.edu/prism/lectures/pr/pr_l25.pdf · CSCE 666 Pattern Analysis | Ricardo Gutierrez-Osuna | CSE@TAMU 2 Introduction •

Ensemble Learning Targeted Maximum Likelihood Estimation

Chapter 18 Decision Trees and Ensemble Learning · 2006-12-01 · Decision Trees and Ensemble Learning ©CSE AI Faculty 2 Recall: Learning Decision Trees ... Using information theory

CSE 546: Machine Learning

LEC 8: Classification trees and ensemble learning€¦ · LEC 8: Classification trees and ensemble learning Dr. Guangliang Chen April 21, 2016. Outline Classification trees Ensemble

Ensemble Learning - textmining.biz: Text-Mining in

Aprendizado de Máquina Ensemble Learning

L25: Ensemble learning

Enhanced Ensemble Learning Technique - Computational Vision

Ensemble Learning Model for MS/MS

Reinforcement Learning CSE 446 – Winter 2012

Ensemble Deep Learning Models for Forecasting

CSE 446: Machine Learning Assignment 2

Varnek A., Tutorial on Ensemble Learning - Unistrainfochim.u-strasbg.fr/new/CS3_2010/Tutorial/Ensemble/Ensemble... · Tutorial on Ensemble Learning 1 ... The following ensemble learning

Statistical Inference and Ensemble Machine Learning for ...digitalassets.lib.berkeley.edu/etd/ucb/text/Davies_berkeley_0028E... · Statistical Inference and Ensemble Machine Learning

TRAVAILLER ENSEMBLE AU SEIN DU CSE - Anact

Bayesian Hyperparameter Optimization for Ensemble Learning

Tutorial on Ensemble Learning - infochim.u-strasbg.frinfochim.u-strasbg.fr/CS3_2010/Tutorial/Ensemble/Ensem…  · Web viewThis tutorial demonstrates performance of ensemble learning

Ensemble deep learning: A review - arXiv

Machine Learning V05: Ensemble Methods - AIssays · Machine Learning V05: Ensemble Methods Meta learning Ensembles in practice AdaBoost ... Ensembles still win competitions, but Deep

Ensemble Selection in a Nutshell - GitHub Pages · Ensemble Selection is an ensemble learning method. The focus of the Ensemble Selection learning algorithm is combining a large set

Ensemble Learning: An Introduction

Machine Learning Lecture 8: Ensemble Methods

Ensemble Learning

Machine Learning (CSE 446): Learning Theory

Machine Learning CSE 681 CH2 - Supervised Learning Computational learning theory

Generative Ensemble-Regression: Learning Stochastic ... · Generative Ensemble-Regression: Learning Stochastic Dynamics from Discrete Particle Ensemble Observations Liu Yang1, Constantinos

Day 8: Ensemble methods, boosting - University of Chicagosuriya/website-intromlss2018/... · 2018. 11. 29. · Ensemble learning •Ensemble learning o Create a population of base

CSE SEMINAR: Introduction to Blended Learning

Deep ensemble learning of sparse regression models for ...adni.loni.usc.edu/adni-publications/Deep ensemble learning of sparse... · Meanwhile, deep representation learning has recently

CSE 5523: Machine Learning and Statistical Pattern Recognitionaritter.github.io/courses/5523_slides/introduction.pdf · CSE 5523: Machine Learning and Statistical Pattern Recognition

Ensemble Learning - Classesclasses.engr.oregonstate.edu/eecs/winter2011/cs434/notes/ensemble9... · Ensemble Learning • So far we ... • AdaBoost performs M boosting rounds,

L21 ensemble learning - Virginia Tech

MotherNets: Rapid Deep Ensemble Learning

Ensemble LearningEnsemble Learning