adaboost derek hoiem march 31, 2004
TRANSCRIPT
Ada
boos
t
Der
ek H
oiem
Mar
ch 3
1, 2
004
Out
line
Bac
kgro
und
Ada
boos
t Alg
orith
m
Theo
ry/In
terp
reta
tions
Pra
ctic
al Is
sues
Face
det
ectio
n ex
perim
ents
Wha
t’s S
o G
ood
Abo
ut A
dabo
ost
Impr
oves
cla
ssifi
catio
n ac
cura
cy
Can
be
used
with
man
y di
ffere
nt c
lass
ifier
s
Com
mon
ly u
sed
in m
any
area
s
Sim
ple
to im
plem
ent
Not
pro
ne to
ove
rfitti
ng
A B
rief H
isto
ry
Boo
tstra
ppin
g
Bag
ging
Boo
stin
g (S
chap
ire19
89)
Ada
boos
t (S
chap
ire19
95)
Boo
tstra
p E
stim
atio
n
Rep
eate
dly
draw
nsa
mpl
es fr
om D
For e
ach
set o
f sam
ples
, est
imat
e a
stat
istic
The
boot
stra
p es
timat
e is
the
mea
n of
the
indi
vidu
al e
stim
ates
Use
d to
est
imat
e a
stat
istic
(par
amet
er)
and
its v
aria
nce
Bag
ging
-A
ggre
gate
Boo
tstra
ppin
g
For i
= 1
.. M
Dra
w n
* <n
sam
ples
from
D w
ith re
plac
emen
tLe
arn
clas
sifie
r Ci
Fina
l cla
ssifi
er is
a v
ote
of C
1 ..
CM
Incr
ease
s cl
assi
fier s
tabi
lity/
redu
ces
varia
nce
Boo
stin
g (S
chap
ire19
89)
Ran
dom
ly s
elec
t n1 <
nsa
mpl
es fr
om D
with
out r
epla
cem
ent t
o ob
tain
D1
Trai
n w
eak
lear
ner C
1
Sel
ect n
2 <
nsa
mpl
es fr
om D
with
hal
f of t
he s
ampl
es m
iscl
assi
fied
by C
1 to
obta
in D
2Tr
ain
wea
k le
arne
r C2
Sel
ect a
llsa
mpl
es fr
om D
that
C1
and
C2
disa
gree
on
Trai
n w
eak
lear
ner C
3
Fina
l cla
ssifi
er is
vot
e of
wea
k le
arne
rs
Ada
boos
t -A
dapt
ive
Boo
stin
g
Inst
ead
of s
ampl
ing,
re-w
eigh
tP
revi
ous
wea
k le
arne
r has
onl
y 50
% a
ccur
acy
over
ne
w d
istri
butio
n
Can
be
used
to le
arn
wea
k cl
assi
fiers
Fina
l cla
ssifi
catio
n ba
sed
on w
eigh
ted
vote
of
wea
k cl
assi
fiers
Ada
boos
t Ter
ms
Lear
ner =
Hyp
othe
sis
= C
lass
ifier
Wea
k Le
arne
r: <
50%
erro
r ove
r any
di
strib
utio
n
Stro
ng C
lass
ifier
: thr
esho
lded
linea
r co
mbi
natio
n of
wea
k le
arne
r out
puts
Dis
cret
e A
dabo
ost (
Dis
cret
eAB
)(F
riedm
an’s
wor
ding
)
Dis
cret
e A
dabo
ost (
Dis
cret
eAB
)(F
reun
d an
d S
chap
ire’s
wor
ding
)
Ada
boos
t with
Con
fiden
ce
Wei
ghte
d P
redi
ctio
ns (R
ealA
B)
Com
paris
on2
Nod
e Tr
ees
Bou
nd o
n Tr
aini
ng E
rror
(Sch
apire
)
Find
ing
a w
eak
hypo
thes
is
Trai
n cl
assi
fier (
as u
sual
) on
wei
ghte
d tra
inin
g da
ta
Som
e w
eak
lear
ners
can
min
imiz
e Z
by g
radi
ent d
esce
nt
Som
etim
es w
e ca
n ig
nore
alp
ha (w
hen
the
wea
k le
arne
r ou
tput
can
be
freel
y sc
aled
)
Cho
osin
g A
lpha
Cho
ose
alph
a to
min
imiz
e Z
Res
ults
in 5
0% e
rror
rate
in la
test
wea
k le
arne
r
In g
ener
al, c
ompu
te n
umer
ical
ly
Spe
cial
Cas
e
Dom
ain-
parti
tioni
ng w
eak
hypo
thes
is (e
.g. d
ecis
ion
trees
, hi
stog
ram
s)
x x xx
x xo o
oo o
ox
P1
P3
P2
Wx1
= 5/
13, W
o1=
0/13
Wx2
= 1/
13, W
o2=
2/13
Wx3
= 1/
13, W
o3=
4/13
Z =
2 (0
+ s
qrt(2
)/13
+ 2/
13) =
.525
Sm
ooth
ing
Pre
dict
ions
Equ
ival
ent t
o ad
ding
prio
r in
parti
tioni
ng c
ase
Con
fiden
ce b
ound
by
Just
ifica
tion
for t
he E
rror
Fun
ctio
n
Ada
boos
t min
imiz
es:
Pro
vide
s a
diffe
rent
iabl
e up
per b
ound
on
train
ing
erro
r (S
hapi
re)
Min
imiz
ing
J(F)
is e
quiv
alen
t up
to 2
ndor
der T
aylo
r ex
pans
ion
abou
t F =
0 to
max
imiz
ing
expe
cted
bin
omia
l lo
g-lik
elih
ood
(Frie
dman
)
Pro
babi
listic
Inte
rpre
tatio
n (F
riedm
an)
Proo
f:
Mis
inte
rpre
tatio
ns o
f the
P
roba
bilis
tic In
terp
reta
tion
Lem
ma
1 ap
plie
s to
the
true
dist
ribut
ion
Onl
y ap
plie
s to
the
train
ing
set
Not
e th
at P
(y|x
) = {0
, 1} f
or th
e tra
inin
g se
t in
mos
t ca
ses
Ada
boos
t will
con
verg
e to
the
glob
al m
inim
umG
reed
y pr
oces
s lo
cal m
inim
umC
ompl
exity
of s
trong
lear
ner l
imite
d by
com
plex
ities
of
wea
k le
arne
rs
Exa
mpl
e of
Joi
nt D
istri
butio
n th
at C
anno
t Be
Lear
ned
with
Naï
ve W
eak
Lear
ners
x2
x1
0, 0
0, 1
1, 0
1, 1
x
xo
o
P(o
| 0,0
) = 1
P(x
| 0,1
) = 1
P(x
| 1,0
) = 1
P(o
| 1,1
) = 1
H(x
1,x2
) = a
*x1
+ b*
(1-x
1) +
c*x
2 +
d*(1
-x2)
=
(a-b
)*x1
+ (c
-d)*
x2 +
(b+d
)
Line
ar d
ecis
ion
for n
on-li
near
pro
blem
whe
n us
ing
naïv
e w
eak
lear
ners
Imm
unity
to O
verfi
tting
?
Ada
boos
t as
Logi
stic
Reg
ress
ion
(Frie
dman
)A
dditi
ve L
ogis
tic R
egre
ssio
n: F
ittin
g cl
ass
cond
ition
al p
roba
bilit
y lo
g ra
tio w
ith a
dditi
ve te
rms
Dis
cret
eAB
build
s an
add
itive
logi
stic
regr
essi
on m
odel
via
New
ton-
like
upda
tes
for m
inim
izin
g
Rea
lAB
fits
an a
dditi
ve lo
gist
ic re
gres
sion
mod
el b
y st
age-
wis
e an
d ap
prox
imat
e op
timiz
atio
n of
Eve
n af
ter t
rain
ing
erro
r rea
ches
zer
o, A
B p
rodu
ces
a “p
urer
” so
lutio
n pr
obab
ilist
ical
ly
Ada
boos
t as
Mar
gin
Max
imiz
atio
n (S
chap
ire)
Bou
nd o
n G
ener
aliz
atio
n E
rror
Num
ber o
f tra
inin
g ex
ampl
es
VC
dim
ensi
on
1/20
1/8
1/4
1/2
Con
fiden
ce o
f cor
rect
dec
isio
n
Con
fiden
ce M
argi
nB
ound
con
fiden
ce te
rm
To lo
ose
to b
e of
pr
actic
al v
alue
:
Max
imiz
ing
the
mar
gin…
But
Ada
boos
t doe
sn’t
nece
ssar
ily m
axim
ize
the
mar
gin
on th
e te
st s
et (R
atsc
h)R
atsc
hpr
opos
es a
n al
gorit
hm (A
dabo
ost*
) tha
t doe
s so
Ada
boos
t and
Noi
sy D
ata
Exa
mpl
es w
ith th
e la
rges
t gap
bet
wee
n th
e la
bel a
nd th
e cl
assi
fier p
redi
ctio
n ge
t the
mos
t wei
ght
Wha
t if t
he la
bel i
s w
rong
?
Opi
tz: S
ynth
etic
dat
a w
ith 2
0% o
ne-s
ided
noi
sy la
belin
g
Num
ber o
f Net
wor
ks in
Ens
embl
e
Wei
ghtB
oost
(Jin
)
Use
s in
put-d
epen
dent
wei
ghtin
g fa
ctor
s fo
r wea
k le
arne
rsTe
xt C
ateg
oriz
atio
n
Bro
wnB
oost
(Fre
und)
Non
-mon
oton
ic w
eigh
ting
func
tion
Exa
mpl
es fa
r fro
m b
ound
ary
decr
ease
in w
eigh
t
Set
a ta
rget
err
or ra
te –
algo
rithm
atte
mpt
s to
ach
ieve
th
at e
rror
rate
No
resu
lts p
oste
d (e
ver)
Ada
boos
t Var
iant
s P
ropo
sed
By
Frie
dman
Logi
tBoo
stS
olve
s R
equi
res
care
to a
void
num
eric
al p
robl
ems
Gen
tleB
oost
Upd
ate
is f m
(x) =
P(y
=1 |
x) –
P(y
=0 |
x) in
stea
d of
Bou
nded
[0 1
]
Com
paris
on (F
riedm
an)
Syn
thet
ic, n
on-a
dditi
ve d
ecis
ion
boun
dary
Com
plex
ity o
f Wea
k Le
arne
r
Com
plex
ity o
f stro
ng c
lass
ifier
lim
ited
by
com
plex
ities
of w
eak
lear
ners
Exa
mpl
e:W
eak
Lear
ner:
stub
s W
L D
imV
C=
2In
put s
pace
: RN
Stro
ng D
imV
C=
N+1
N+1
par
titio
ns c
an h
ave
arbi
trary
con
fiden
ces
assi
gned
Com
plex
ity o
f the
Wea
k Le
arne
r
Com
plex
ity o
f the
Wea
k Le
arne
r
Non
-Add
itive
Dec
isio
n Bo
unda
ry
Ada
boos
t in
Face
Det
ectio
n
One
Gro
up o
f
N-D
His
togr
ams
Rea
lAB
Sch
neid
erm
an
1-D
His
togr
ams
KLB
oost
KLB
oost
1-D
His
togr
ams
Floa
tBoo
stFl
oatB
oost
Stu
bsD
iscr
eteA
BV
iola
-Jon
es
Wea
k Le
arne
rA
dabo
ost
Var
iant
Det
ecto
r
Boo
sted
Tre
es –
Face
Det
ectio
n
Stu
bs (2
Nod
es)
8 N
odes
Boo
sted
Tre
es –
Face
Det
ectio
n
Ten
Stu
bs v
s. O
ne 2
0 N
ode
2, 8
, and
20
Nod
es
Boo
sted
Tre
es v
s. B
ayes
Net
Floa
tBoo
st(L
i)FP
Rat
e at
95.
5% D
et R
ate
35.0
%
40.0
%
45.0
%
50.0
%
55.0
%
60.0
%
65.0
%
70.0
%
75.0
%
80.0
%
510
1520
2530
Num
Iter
atio
ns
Tree
s - 2
0 bi
ns(L
1+L2
)Tr
ees
- 20
bins
(L2)
Tree
s - 8
bin
s (L
2)
Tree
s - 2
bin
s (L
2)
1. L
ess
gree
dy a
ppro
ach
(Flo
atB
oost
) yie
lds
bette
r res
ults
2. S
tubs
are
not
suf
ficie
nt fo
r vis
ion
Whe
n to
Use
Ada
boos
t
Giv
e it
a try
for a
ny c
lass
ifica
tion
prob
lem
Be
war
y if
usin
g no
isy/
unla
bele
d da
ta
How
to U
se A
dabo
ost
Sta
rt w
ith R
ealA
B(e
asy
to im
plem
ent)
Pos
sibl
y try
a v
aria
nt, s
uch
as F
loat
Boo
st, W
eigh
tBoo
st,
or L
ogitB
oost
Try
vary
ing
the
com
plex
ity o
f the
wea
k le
arne
r
Try
form
ing
the
wea
k le
arne
r to
min
imiz
e Z
(or s
ome
sim
ilar g
oal)
Con
clus
ion
Ada
boos
t can
impr
ove
clas
sifie
r acc
urac
y fo
r man
y pr
oble
ms
Com
plex
ity o
f wea
k le
arne
r is
impo
rtant
Alte
rnat
ive
boos
ting
algo
rithm
s ex
ist t
o de
al w
ith m
any
of A
dabo
ost’s
prob
lem
s
Ref
eren
ces
Dud
a, H
art,
ect–
Pat
tern
Cla
ssifi
catio
n
Freu
nd –
“An
adap
tive
vers
ion
of th
e bo
ost b
y m
ajor
ity a
lgor
ithm
”
Freu
nd –
“Exp
erim
ents
with
a n
ew b
oost
ing
algo
rithm
”
Freu
nd, S
chap
ire–
“A d
ecis
ion-
theo
retic
gen
eral
izat
ion
of o
n-lin
e le
arni
ng a
nd a
n ap
plic
atio
n to
boo
stin
g”
Frie
dman
, Has
tie, e
tc –
“Add
itive
Log
istic
Reg
ress
ion:
A S
tatis
tical
Vie
w o
f Boo
stin
g”
Jin,
Liu
, etc
(CM
U) –
“A N
ew B
oost
ing
Alg
orith
m U
sing
Inpu
t-Dep
ende
nt R
egul
ariz
er”
Li, Z
hang
, etc
–“F
loat
boos
tLea
rnin
g fo
r Cla
ssifi
catio
n”
Opi
tz, M
aclin
–“P
opul
ar E
nsem
ble
Met
hods
: An
Em
piric
al S
tudy
”
Rat
sch,
War
mut
h–
“Effi
cien
t Mar
gin
Max
imiz
atio
n w
ith B
oost
ing”
Sch
apire
, Fre
und,
etc
–“B
oost
ing
the
Mar
gin:
A N
ew E
xpla
natio
n fo
r the
Effe
ctiv
enes
s of
Vot
ing
Met
hods
”
Sch
apire
, Sin
ger –
“Impr
oved
Boo
stin
g A
lgor
ithm
s U
sing
Con
fiden
ce-W
eigh
ted
Pre
dict
ions
”
Sch
apire
–“T
he B
oost
ing
App
roac
h to
Mac
hine
Lea
rnin
g: A
n ov
ervi
ew”
Zhan
g, L
i, et
c –
“Mul
ti-vi
ew F
ace
Det
ectio
n w
ith F
loat
boos
t”
Ada
boos
t Var
iant
s P
ropo
sed
By
Frie
dman
Logi
tBoo
st
Ada
boos
t Var
iant
s P
ropo
sed
By
Frie
dman
Gen
tleB
oost