ethem alpaydin © the mit press, 2010ethem/i2ml2e/2e_v1-0/i2ml2e-chap3 … · apriori algorithm...

ETHEM ALPAYDIN© The MIT Press, 2010

alpaydin@boun.edu.trhttp://www.cmpe.boun.edu.tr/~ethem/i2ml2e

Lecture Slides for

Probability and Inference Result of tossing a coin is {Heads,Tails}

Random var X {1,0}

Bernoulli: P {X=1} = poX (1 ‒ po)(1 ‒ X)

Sample: X = {xt }Nt =1

Estimation: po = # {Heads}/#{Tosses} = ∑txt / N

Prediction of next toss:

Heads if po > ½, Tails otherwise

3Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Classification Credit scoring: Inputs are income and savings.

Output is low-risk vs high-risk Input: x = [x1,x2]T ,Output: C Î {0,1} Prediction:

otherwise 0

)|()|( if 1 choose

or

otherwise 0

)|( if 1 choose

C

C

C

C

,xxCP ,xxCP

. ,xxCP

2121

21

01

501

4Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Bayes’ Rule

x

xx

p

pPP

CCC

| |

110

0011

110

xx

xxx

||

||

CC

CCCC

CC

Pp

PpPpp

PP

5

posterior

likelihoodprior

evidence

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Bayes’ Rule: K>2 Classes

K

kkk

ii

iii

CPCp

CPCp

p

CPCpCP

1

|

|

||

x

x

x

xx

xx | max | if choose

and 1

kkii

K

iii

CPCPC

CPCP

10

6Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Losses and Risks Actions: αi

Loss of αi when the state is Ck : λik

Expected risk (Duda and Hart, 1973)

xx

xx

|min| if choose

||

kkii

k

K

kiki

RR

CPR

1

7Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Losses and Risks: 0/1 Loss

ki

kiik

if

if

1

0

x

x

xx

|

|

||

i

ikk

K

kkiki

CP

CP

CPR

1

1

8

For minimum risk, choose the most probable class

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Losses and Risks: Reject

10

1

1

0

otherwise

if

if

,Ki

ki

ik

xxx

xx

|||

||

iik

ki

K

kkK

CPCPR

CPR

1

1

1

otherwise reject

| and || if choose 1xxx ikii CPikCPCPC

9Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Discriminant Functions Kigi ,, , 1x xx kkii ggC max if choose

xxx kkii gg max| R

ii

i

i

i

CPCp

CP

R

g

|

|

|

x

x

x

x

10

K decision regions R1,...,RK

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

K=2 Classes Dichotomizer (K=2) vs Polychotomizer (K>2)

g(x) = g1(x) – g2(x)

Log odds:

otherwise

if choose

2

1 0

C

gC x

x

x

|

|log

2

1

CP

CP

11Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Utility Theory Prob of state k given exidence x: P (Sk|x)

Utility of αi when state is k: Uik

Expected utility:

xx

xx

| max| if Choose

||

jj

ii

kkiki

EUEUα

SPUEU

12Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Association Rules Association rule: X Y

People who buy/click/visit/enjoy X are also likely to buy/click/visit/enjoy Y.

A rule implies association, not necessarily causation.

13Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

Association measures Support (X Y):

Confidence (X Y):

Lift (X Y):

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) 14

customers

and bought whocustomers

#

#,

YXYXP

X

YX

XP

YXPXYP

bought whocustomers

and bought whocustomers

|

#

#

)(

,

)(

)|(

)()(

,

YP

XYP

YPXP

YXP

Apriori algorithm (Agrawal et al., 1996) For (X,Y,Z), a 3-item set, to be frequent (have enough

support), (X,Y), (X,Z), and (Y,Z) should be frequent.

If (X,Y) is not frequent, none of its supersets can be frequent.

Once we find the frequent k-item sets, we convert them to rules: X, Y Z, ...

and X Y, Z, ...

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) 15