•url: .../publications/courses/ece_8443/lectures/current/exam/2004
DESCRIPTION
ECE 8443 – Pattern Recognition. LECTURE 15: EXAM NO. 1 (CHAP. 2). •Spring 2004 Solutions: 1a 1b 1c 2a 2b 2c 2d 2e 2f 2g. •URL: .../publications/courses/ece_8443/lectures/current/exam/2004/. LECTURE 15: EXAM NO. 1 (CHAP. 2). PROBLEM 1. - PowerPoint PPT PresentationTRANSCRIPT
• URL: .../publications/courses/ece_8443/lectures/current/exam/2004/
ECE 8443 – Pattern Recognition
LECTURE 15: EXAM NO. 1 (CHAP. 2)
• Spring 2004
• Solutions:
1a 1b 1c 2a 2b 2c 2d 2e 2f 2g
Problem No. 1: Let for a two-category one-dimensional problem with
(a) Show that the minimum probability of error is
given by: where
),i
(N~)i
|x(p2
./)(P)(P 2121
du
a
ue
eP
22
2
1 212/a
Solution:
1 1 2 2
2 1
( ) ( ) ( / ) ( ) ( / )R R
P error P p x dx P p x dx The probability of error is given by:
1 2
LECTURE 15: EXAM NO. 1 (CHAP. 2)PROBLEM 1
Where R1 denotes class region and R2, . To determine R1 and R2 , the decision region must
be determined.
Calculate the decision rule with the Likelihood Ratio : For a two–category problem place in if
the likelihood ratio exceeds a threshold value . Otherwise decide
)(
)(
)|(
)|(
1
2
2
1
P
P
xp
xp
12
x
--------------------- ( 2 )
--------------------- ( 1 )
The class conditional densities are
222
212
)(2
1
2
)(2
1
12
1)/(,
2
1)/(
xx
expexp
and the Prior probabilities are ./)(P)(P 2121
For a given feature , the likelihood ratio (1) can be used to determine the decision rule:
2
)(
0))(2)((
0)()(2
22
1)(
)(
2/1
2/1
2
12
1
21
1212
21
2212
222
2211
2
22
21
)(2
1
)(2
1
222
212
x
x
x
xxxx
x
x
e
e
x
x
,changing signs
,taking log
, decide 1Else If
2
)( 21 x , decide 2
x
Assume , then, Let
Where,
ax
1 dadx
1 1*
( ) 2* ( ) ( / )x
P error P p x dx
21
*
1
*
1
)/(
)/(2
1*2
x
x
xp
xp
PROBLEM 1LECTURE 15: EXAM NO. 1 (CHAP. 2)
The decision region is half way between the two means. This result makes sense from an intuitive point of
view since the likelihoods are identical and differ only in their mean value.
Probability of error ( 1 ) becomes:
*
11
*
11 )/()()/()()(xx
dxxpPdxxpPerrorP
2
)(* 21
xwhere ,
211
2
2*
1( )
2
x
x
P error e dx
*
2
1
*
2
1 22
2
1
2
1)(
a
a
a
adaedaeerrorP
22* 12
121
a
Combining these two results we get:
a
adaeerrorP
2
2
1
2
1)(
1 2| |
2a
We get a similar result if 2<1.
*
2
1 2
2
1)(
a
adaeerrorP
22* 21
221
aWhere,
Where,
PROBLEM 1LECTURE 15: EXAM NO. 1 (CHAP. 2)
OK. But how to be sure if this is the minimum probability of error?
P[error] in terms of the posterior P[error|x]
The optimal decision rule will minimize P[error|x] for every value of x. At each feature value x,
P[error|x] = P[ω2|x] when ω1 was chosen.
Therefore, when we integrate over the limit, the decision rule yields a minimum P[error] . This probability of
error is the Bayes Error.
dxxpxerrorPerrorP )()|(][
(b) Use the inequality to show that P e goes to zero as
goes to infinity.
22
2
122
2
1 ae
a
dt
a
te
eP
/
12
2
2
2
2
1
2
1
2
1
2
1
2
1
02
1lim
a
a
a
e
a
a
ea
daeP
ea
and
1 2| |
2a
as the distance between the means of the two distribution
tend to infinity.
0eP 1 2| |
2a
PROBLEM 1LECTURE 15: EXAM NO. 1 (CHAP. 2)
Solution:
(c) How does your result for (b) change if the prior probabilities are not equal?
Making , puts infinite distance between the classes . The probability of
error will still tend to zero if the Prior probabilities are not equal.
2
2121&
PROBLEM 1LECTURE 15: EXAM NO. 1 (CHAP. 2)
PROBLEM 2Problem No. 2: Given a two-class two-dimensional classification problem (x = {x1,x2}) with the following parameters
(uniform distributions):
3 / 4 1/ 4 0 11 11 1
( | ) ( | )3 / 4 1/ 4 0 11 22 2
0 0
x x
p px x
elsewhere elsewhere
x x
( ) ( ) 1/ 2.1 2P P
LECTURE 15: EXAM NO. 1 (CHAP. 2)
a) Compute the mean and covariance (hint: plot the distributions).
I. Using Independent class approach, we get joint pdf as
1
2
1
2
1
2
1
2
1
2
( )
3/ 4 01/ 2
3/ 4 1/ 4
0 1/ 41/ 2
3/ 4 0
0 1/ 41
0 1/ 4
0 1/ 41/ 2
1/ 4 1
1/ 4 11/ 2
0 1
0
p
x
x
x
x
x
x
x
x
x
x
otherwise
x
1
11
1
( )
1/ 2 3/ 4 0
1 0 1/ 4
1/ 2 1/ 4 1
0
p x
x
x
x
otherwise
2
22
2
( )
1/ 2 3/ 4 0
1 0 1/ 4
1/ 2 1/ 4 1
0
p x
x
x
x
otherwise
We can obtain marginal probability distribution function as
0
2040
6080
0
2040
60800
0.2
0.4
0.6
0.8
1
PROBLEM 2LECTURE 15: EXAM NO. 1 (CHAP. 2)
1 11
2 2 2
( ) 1/ 8
1/8( )E
x p xE x
E x x p x
x
1 1 11 1 1 1
1
1 1 1 1 1 1 1
2
1 1 1 2 1 21 1 1 1 1 1 2 211 12
221 22
2 2 1 1 2 2 2 2 2 1 2 1 2 2
E x E x x E EE x x E x x
E x x E x x E x x E E E x
1 01210 12
2
2
2
11 22 2
2 2 2
// 1/ 2
// 1/ 2
EE x
p dxE x
x x x
2 2 22 2 2 2
2
2 2 2 2 2 2 2
2
1 1 1 2 1 21 1 1 1 1 1 2 211 12
221 22
2 2 1 1 2 2 2 2 2 1 2 1 2 2
E x E x x E EE x x E x x
E x x E x x E x x E E E x
1 01210 12
II. Using class-dependent approach,
1
2
1 1 1 2 1 21 1 1 1 1 1 2 211 12
221 22
2 2 1 1 2 2 2 2 2 1 2 1 2 2
43 9192 649 4364 192
E x E x x E EE x x E x x
E x x E x x E x x E E E x
1
1
1
11 11 1
2 1 2
// 1/ 4
// 1/ 4
EE x
p dxE x
x x x
We can calculate mean and covariance as,
PROBLEM 2LECTURE 15: EXAM NO. 1 (CHAP. 2)
Figure showing the decision surface and the distribution of two classes
PROBLEM 2b) Find the discriminant functions(e.g., gi (x) ).
There are infinite numbers of solutions for the case of P(w1)=P(w2) and P(x/w1)=P(x/w2) in the overlap area.
The simplest one can be defined asg(x)= x1+ x2-1/4 such that
if g(x)>0 decide class w2
else class w1
C) Write the Bayes decision rule for this case (hint: draw the decision boundary). Is this solution unique? Explain.
Since P(w1)=P(w2) and P(x/w1)=P(x/w2) in the overlap area, the posterior probability will be same in the overlap area.
Hence, the solution won’t be unique.
d) Compute the probability of error.
LECTURE 15: EXAM NO. 1 (CHAP. 2)
1 1 2 2
2 1
1 1 1 1 1( ) ( ) ( / ) ( ) ( / )
2 32 2 32 32R R
P error P p x dx P p x dx
PROBLEM 2e) How will the decision surface change if the priors are not equal? Explain.
When the priors are equal, the decision region is at the point where two distribution meet if the distributions are similar. If the priors are not equal the decision region moves away from the higher prior class.
f) How will the probability of error change if the priors are not equal?
As the prior probability changes, the decision surface changes. Hence the probability of error changes. For ex, if P(w 1)=1 and P(w2)=0. In this case, the decision line will move such that the overlapped rectangle region belongs to region 1. Hence probability of error is zero.
g) Draw the minimax decision surface. Compare and contrast this to your answer in part (c).
The requirement for the minimax decision surface is
Since P(x/w1)=P(x/w2), we need to obtain R1= R2 The minimax decision surface also will have infinite solution compared to Bayes’ decision surface.
The contrast is the overlap region needs to be divided into equal area to get R1= R2
LECTURE 15: EXAM NO. 1 (CHAP. 2)
1 2
2 1( | ) ( | )R R
p x dx p x dx