statistical approach to classification
DESCRIPTION
Statistical Approach to Classification. Naïve Bayes Classifier. Remember…. Sensors, scales, etc…. Red = 2.125 Yellow = 6.143 Mass = 134.32 Volume = 24.21. Apple. Redness. Let’s look at one dimension. For a given redness value which is the most probable fruit. Redness. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/1.jpg)
STATISTICAL APPROACH TO CLASSIFICATION
Naïve Bayes Classifier
![Page 2: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/2.jpg)
Bayesian Classifier 2
Remember…Red = 2.125
Yellow = 6.143
Mass = 134.32
Volume = 24.21
Apple
Sensors, scales, etc…
8/29/03
![Page 3: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/3.jpg)
Bayesian Classifier 3
0 2 4 6 8
0.00
000.
0005
0.00
100.
0015
0.00
20
Distribution of Redness Values
Redness
Den
sity
Fruit
ApplesPeachesOrangesLemons
Redness Let’s look at one dimension
For a given redness value which is the most probable fruit
8/29/03
![Page 4: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/4.jpg)
Bayesian Classifier 4
Redness What if we wanted to ask the question “what is the
probability that some fruit with a given redness value is an apple?”
8/29/03
0 2 4 6 8
0.00
000.
0005
0.00
100.
0015
0.00
20
Distribution of Redness Values
Redness
Den
sity
Fruit
ApplesPeachesOrangesLemons
Could we just look at how far away it is from the apple peak?
Is it the highest PDF above the X-value in question?
![Page 5: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/5.jpg)
Bayesian Classifier 5
Redness of Apples and Oranges
Redness
1 2 3 4 5 6 7
050
100
150
200
250
Probability it’s an apple If a fruit has a
redness of 4.05 do we know the probability that it’s an apple?
What do we know?
8/29/03
• We know the total number of fruit at that redness (10+25)
• We know the fraction of apples at that redness (10)
• Probability that a fruit with a redness value of 4.05 is an apple is
• If it is a histogram of counts then it straight forward• Probability it’s an apple
28.57%• Probability it’s an orange
71.43%• Getting the probability is simple
![Page 6: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/6.jpg)
Bayesian Classifier 6
But what if working PDF Probability density function Continuous Probability not count Might be tempted
to use the same approach
8/29/03
0 2 4 6 8
0.00
000.
0005
0.00
100.
0015
0.00
20
Distribution of Redness Values
RednessD
ensi
ty
Fruit
ApplesPeachesOrangesLemons
P(a fruit with redness 4.05 is apple)?=
Parametric ( and parameters)
vs. non-parametric
![Page 7: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/7.jpg)
Bayesian Classifier 7
Problem What if had trillion
oranges and only 100 apples
Might be the most common apple and have a higher value at 4.05 than oranges even though the universe would have way more oranges at that value
8/29/030 2 4 6 8
0.00
000.
0005
0.00
100.
0015
0.00
20
Distribution of Redness Values
Redness
Den
sity
Fruit
ApplesPeachesOrangesLemons
Wouldn’t change the PDFsbut…
![Page 8: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/8.jpg)
Bayesian Classifier 8
Redness of Apples and Oranges
Redness
1 2 3 4 5 6 7
050
100
150
200
250
Let’s revisit but using probabilities instead of counts
2506 apples 2486 oranges If a fruit has a
redness of 4.05 do we know the probability that it’s an apple if we don’t have specific counts at 4.05?
8/29/03
Conditional Probability
If we know it is an apple, then the…But what we want
![Page 9: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/9.jpg)
Bayesian Classifier 9
Bayes Theorem
8/29/03
Above from the book h is hypothesis, D is training Data
𝑃 (h|𝐷 )=𝑃 (𝐷|h ) 𝑃 (h)
𝑃 (𝐷)
𝑃 (𝑎𝑝𝑝𝑙𝑒|𝑟𝑒𝑑𝑛𝑒𝑠𝑠=4.05 )=𝑃 (𝑟𝑒𝑑𝑛𝑒𝑠𝑠=4.05|𝑎𝑝𝑝𝑙𝑒 )𝑃 (𝐴𝑝𝑝𝑙𝑒 )
𝑃 (𝑅𝑒𝑑𝑛𝑒𝑠𝑠=4.05)
Does this make sense?
![Page 10: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/10.jpg)
Bayesian Classifier 10
Redness of Apples and Oranges
Redness
1 2 3 4 5 6 7
050
100
150
200
250
Make Sense? 2506 apples 2486 oranges Probability that redness
would be 4.05 if know an apple About 10/2506
P(apple)? 2506/(2506+2486)
P(redness=4.05) About
(10+25)/(2506+2486)
8/29/03
𝑃 (𝑎𝑝𝑝𝑙𝑒|𝑟𝑒𝑑𝑛𝑒𝑠𝑠=4.05 )=𝑃 (𝑟𝑒𝑑𝑛𝑒𝑠𝑠=4.05|𝑎𝑝𝑝𝑙𝑒 )𝑃 (𝐴𝑝𝑝𝑙𝑒)
𝑃 (𝑅𝑒𝑑𝑛𝑒𝑠𝑠=4.05)
10(10+25)
=
102506 ∙
2506(2506+2486)
(10+25)(2506+2486)
?
![Page 11: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/11.jpg)
Bayesian Classifier 11
Can find the probability Whether have counts or PDF How do we classify?
Simply find the most probable class
8/29/03
h=argmaxh∈𝐻
𝑃 (h∨𝐷)
![Page 12: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/12.jpg)
Bayesian Classifier 12
Bayes
I think of the ratio of P(h) to P(D) as an adjustment to the easily determined P(D|h) in order to account for differences in sample size
8/29/03
𝑃 (h|𝐷 )=𝑃 (𝐷|h ) 𝑃 (h)
𝑃 (𝐷)
Prior Probabilities or Priors
Posterior Probability
![Page 13: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/13.jpg)
Bayesian Classifier 13
MAP Maximum a posteriori hypothesis (MAP)
ä-(ˌ)pō-ˌstir-ē-ˈor-ē Relating to or derived by reasoning from observed facts;
inductive A priori: relating to or derived by reasoning from self-evident
propositions; deductive Approach: Brute-force MAP learning algorithm
8/29/03
h𝑀𝐴𝑃=argmaxh∈𝐻
𝑃 (h∨𝐷)
![Page 14: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/14.jpg)
Bayesian Classifier 14
More is better
Mass (normalized)0 1 2 3 4 5 6 7 8 9 10
12
34
56
78
910
Red
Inte
nsity
(nor
mal
ized)
More dimensions can be helpful
8/29/03
Linearly Separable
![Page 15: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/15.jpg)
Bayesian Classifier 15
What if some of the dims disagree
Color (red and yellow) says apple but mass and volume say orange?
Take a vote?
8/29/03
How handle multiple dimensions?
![Page 16: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/16.jpg)
Bayesian Classifier 16
Can cheat Assume each dimension is independent
(doesn’t co-vary with any other dimension)
Can use the product rule The probability that a fruit is an apple
given a set of measurements (dimensions) is:
8/29/03
P (h|𝐷𝑟𝑒𝑑 )∗P (h|𝐷𝑦𝑒𝑙𝑙𝑜𝑤 )∗ P (h|𝐷𝑣𝑜𝑙𝑢𝑚𝑒 )∗ P (h|𝐷𝑚𝑎𝑠𝑠 )
![Page 17: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/17.jpg)
Bayesian Classifier 17
Naïve Bayes Classifier Known as a Naïve Bayes Classifier
Where vj is class and ai is an attribute Derivation
8/29/03
𝑣𝑁𝐵=argmax𝑣 𝑗∈𝑉
𝑃 (𝑣 𝑗)∏𝑖𝑃 (𝑎𝑖∨𝑣 𝑗)Where is the denominator?
![Page 18: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/18.jpg)
Bayesian Classifier 188/29/03
Example You wish to classify an instance with the following attributes
1.649917 5.197862 134.898820 16.137695 The first column is redness, then yellowness, followed by mass
then volume The training data has in the redness histogram bin in which the
instance falls 0 apples, 0 peaches, 9 oranges, and 22 lemons
In the bin for yellowness there are 235, 262, 263, and 239
In the bin for mass there are 106, 176, 143, and 239
In the bin for vol there are What 3, 57, 7, and 184
What are each of the probabilities that it is an • Apple• Peach• Orange• Lemon
![Page 19: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/19.jpg)
Bayesian Classifier 198/29/03
SolutionRed Yellow Mass Vol
Apples 0 235 106 3peaches 0 262 176 57oranges 9 263 143 7lemons 22 239 239 184Total 31 999 664 251apples 0 0.24 0.16 0.01 0peaches 0 0.26 0.27 0.23 0oranges 0.29 0.26 0.22 0.03 0.0005lemons 0.71 0.24 0.36 0.73 0.0044
![Page 20: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/20.jpg)
Bayesian Classifier 208/29/03
Zeros Is it really a zero percent chance that it’s an apple? Are these really probabilities
(hint: 0.0005 + 0.0044 not equal to 1)? What of the bin size?
Red Yellow Mass Volapples 0 235 106 3peaches 0 262 176 57oranges 9 263 143 7lemons 22 239 239 184Total 31 999 664 251apples 0 0.24 0.16 0.01 0peaches 0 0.26 0.27 0.23 0oranges 0.29 0.26 0.22 0.03 0.0005lemons 0.71 0.24 0.36 0.73 0.0044
![Page 21: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/21.jpg)
Bayesian Classifier 218/29/03
Zeros Estimating probabilities is an estimate of the probability m-estimate The choice of m is often some upper bound to n and
p is often 1/m This ensures a numerator is at least 1 (never zero) Denominator starts at upper bound and goes up to
twice that No loss of order, would be zeros are very small
![Page 22: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/22.jpg)
Bayesian Classifier 228/29/03
Curse of dimensionality Do too many dimensions hurt?
What if only some dimensions contribute to ability to classify? What would the other dimensions do to the probabilities?
![Page 23: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/23.jpg)
Bayesian Classifier 238/29/03
All about representation With imagination and innovation can
learn to classify many things you wouldn’t expect
What if you wanted to learn to classify documents, how might you go about it?
![Page 24: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/24.jpg)
Bayesian Classifier 248/29/03
Example Learning to classify text
Collect all words in examples Calculate P(vj) and P(wk|vj) Each instance will be a vector of size |vocabulary| Classes (v’s) (category) Each word (w) is a dimension
𝑣 𝑁𝐵=argmax𝑣 𝑗∈𝑉
𝑃 (𝑣𝑗) ∏𝑖∈ 𝑃𝑜𝑠𝑖𝑡𝑖𝑜𝑛𝑠
𝑃 (𝑎𝑖∨𝑣𝑗)
![Page 25: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/25.jpg)
Bayesian Classifier 258/29/03
Paper 20 News groups 1000 training documents from each group
The groups were the classes 89% classification accuracy
89 out of every 100 times could tell which newsgroup a document came from
![Page 26: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/26.jpg)
Bayesian Classifier 268/29/03
Another example: RNA
Rift Valley fever virus Basically RNA (like DNA but with an extra
oxygen – the D in DNA is deoxy) Encapsulated in a protein sheath Important protein involved in the
encapsulation process Nucleocapsid
![Page 27: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/27.jpg)
Bayesian Classifier 278/29/03
SELEX SELEX (Systematic Evolution of Ligands
by Exponential Enrichment) Identify RNA segments that have a high
affinity for nucleocapsid (aptamer vs. non-aptamer)
![Page 28: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/28.jpg)
Bayesian Classifier 288/29/03
Could we build a classifier Each known aptamer was 30 nucleotides
long A 30 character string
4 nucleotides (ACGU) What would the data
look-like How would we “bin” the
data?
![Page 29: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/29.jpg)
Bayesian Classifier 298/29/03
Results
![Page 30: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/30.jpg)
Bayesian Classifier 308/29/03
Discrete or real valued? Have seen
Fruit example Documents RNA (nucleotides)
Which is best for Bayesian?
Integers
StringsFloating Point
![Page 31: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/31.jpg)
Bayesian Classifier 318/29/03
Gene Expression Experiments
The brighter the spot, the greater the mRNA concentration
![Page 32: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/32.jpg)
Bayesian Classifier 328/29/03
Can we use expression profiles to detect disease
Thousands of genes (dimensions) Many genes not affected (distributions for
disease and normal same in that dimension)gene
patientg1 g2 g3 … gn disease
p1 x1,1 x1,2 x1,3 … x1,n Yp2 x2,1 x2,2 x2,3 … x2,n N
.
.
.
.
.
.
pm xm,1 xm,2 xm,3 … xm,n ?
![Page 33: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/33.jpg)
Bayesian Classifier 338/29/03
Rare Moss Growth Conditions Perhaps at good growth
locations pH Average temperature Average sunlight exposure Salinity Average length of day
What else? What would the data look-
like?
![Page 34: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/34.jpg)
Bayesian Classifier 348/29/03
Proof Taken from “Pattern Recognition” third edition
Sergios Theodoridis and Konstantinos Koutroumbas
The Bayesian classifier is optimal with respect to minimizing the classification error probability
Proof: let R1 be the region of the feature space in which we decide tin favor of w1 and R2 be the corresponding region for w2. Then an error is made if although it belongs to w2 of if although it belongs to w1.
𝑃𝑒=𝑃 (𝑥∈𝑅2 ,𝑤1 )+𝑃 (𝑥∈𝑅1 ,𝑤2 )
0 2 4 6 8
0.00
000.
0005
0.00
100.
0015
0.00
20
Distribution of Redness Values
Redness
Den
sity
Fruit
ApplesPeachesOrangesLemons
![Page 35: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/35.jpg)
Bayesian Classifier 358/29/03
Proof Joint probability
Using Bayes Rule
It is now easy to see that the error is minimized if the partitioning regions R1 and R2 of the feature space are chosen so that:
![Page 36: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/36.jpg)
Bayesian Classifier 368/29/03
Proof Indeed, since the union of the regions R1, R2
covers all the space, from the definition of a probability density function we have that
Combining
This suggests that the probability of error is minimized if R1 is the region of space in which . Then R2 becomes the region where the reverse is true.
![Page 37: Statistical Approach to Classification](https://reader034.vdocuments.net/reader034/viewer/2022052703/568161cc550346895dd1b554/html5/thumbnails/37.jpg)
Bayesian Classifier 378/29/03