1 peter fox data analytics – itws-4963/itws-6965 week 6a, february 25, 2014, sage 3101 knn,...
TRANSCRIPT
1
Peter Fox
Data Analytics – ITWS-4963/ITWS-6965
Week 6a, February 25, 2014, SAGE 3101
kNN, K-Means, Clustering and Bayesian Inference
Contents
2
Did you get to create the neighborhood map?
table(mapcoord$NEIGHBORHOOD)
mapcoord$NEIGHBORHOOD <- as.factor(mapcoord$NEIGHBORHOOD)
geoPlot(mapcoord,zoom=12,color=mapcoord$NEIGHBORHOOD) # this one is easier
3
4
KNN!Did you loop over k?
{
knnpred<-knn(mapcoord[trainid,3:4],mapcoord[testid,3:4],cl=mapcoord[trainid,2],k=5)
knntesterr<-sum(knnpred!=mappred$class)/length(testid)
}
knntesterr
[1] 0.1028037 0.1308411 0.1308411 0.1588785 0.1401869 0.1495327 0.1682243 0.1962617 0.1962617 0.1869159
What do you think?
5
What else could you classify?• SALE.PRICE?
– If so, how would you measure error?
# I added SALE.PRICE as 5th column in adduse…
> pcolor<- color.scale(log(mapcoord[,5]),c(0,1,1),c(1,1,0),0)
> geoPlot(mapcoord,zoom=12,color=pcolor)• TAX.CLASS.AT.PRESENT?• TAX.CLASS.AT.TIME.OF.SALE?
• measure error?6
Summing up ‘knn’• Advantages
– Robust to noisy training data (especially if we use inverse square of weighted distance as the “distance”)
– Effective if the training data is large
• Disadvantages– Need to determine value of parameter K (number of
nearest neighbors)– Distance based learning is not clear which type of
distance to use and which attribute to use to produce the best results. Shall we use all attributes or certain attributes only?
• Friday – yet more KNN: weighted KNN… 7
K-Means!> mapmeans<-data.frame(adduse$ZIP.CODE, as.numeric(mapcoord$NEIGHBORHOOD), adduse$TOTAL.UNITS, adduse$"LAND.SQUARE.FEET", adduse$GROSS.SQUARE.FEET, adduse$SALE.PRICE, adduse$'querylist$latitude', adduse$'querylist$longitude')
> mapobj<-kmeans(mapmeans,5, iter.max=10, nstart=5, algorithm = c("Hartigan-Wong", "Lloyd", "Forgy", "MacQueen"))
> fitted(mapobj,method=c("centers","classes")) 8
Return objectcluster A vector of integers (from 1:k) indicating the cluster to which each point is allocated.
centers A matrix of cluster centres.
totss The total sum of squares.
withinss Vector of within-cluster sum of squares, one component per cluster.
tot.withinss Total within-cluster sum of squares, i.e., sum(withinss).
betweenss The between-cluster sum of squares, i.e. totss-tot.withinss.
size The number of points in each cluster. 9
10
> plot(mapmeans,mapobj$cluster)
ZIP.CODE, NEIGHBORHOOD, TOTAL.UNITS, LAND.SQUARE.FEET, GROSS.SQUARE.FEET, SALE.PRICE, latitude, longitude'
ZIP.C
OD
E, N
EIG
HB
OR
HO
OD
, TO
TAL
.UN
ITS
, LA
ND
.SF, G
RO
SS
.SF, S
AL
E.P
RIC
E, la
t, lon
g
> mapobj$size[1] 432 31 1 11 56
11
> mapobj$centers adduse.ZIP.CODE as.numeric.mapcoord.NEIGHBORHOOD. adduse.TOTAL.UNITS adduse.LAND.SQUARE.FEET1 10464.09 19.47454 1.550926 2028.2852 10460.65 16.38710 25.419355 11077.4193 10454.00 20.00000 1.000000 29000.0004 10463.45 10.90909 42.181818 10462.2735 10464.00 17.42857 4.714286 14042.214 adduse.GROSS.SQUARE.FEET adduse.SALE.PRICE adduse..querylist.latitude. adduse..querylist.longitude.1 1712.887 279950.4 40.85280 -73.873572 26793.516 2944099.9 40.85597 -73.891393 87000.000 24120881.0 40.80441 -73.922904 40476.636 6953345.4 40.86009 -73.886325 9757.679 885950.9 40.85300 -73.87781
Plotting clustersrequire(cluster)
clusplot(mapmeans, mapobj$cluster, color=TRUE, shade=TRUE, labels=2, lines=0)
12
Simpler K-Means!> mapmeans<-data.frame(as.numeric(mapcoord$NEIGHBORHOOD), adduse$GROSS.SQUARE.FEET, adduse$SALE.PRICE, adduse$'querylist$latitude', adduse$'querylist$longitude')
> mapobjnew<-kmeans(mapmeans,5, iter.max=10, nstart=5, algorithm = c("Hartigan-Wong", "Lloyd", "Forgy", "MacQueen"))
> fitted(mapobjnew,method=c("centers","classes"))
13
Plot
14
Clusplot (k=17)
15
Dendogram for this = tree of the clusters:
16
Highly supported by data?
Okay, this is a little complex – perhaps something simpler?
Hierarchical clustering> d <- dist(as.matrix(mtcars))
> hc <- hclust(d)
> plot(hc)
17
Decision tree (example)> require(party) # don’t get me started!
> str(iris)
'data.frame': 150 obs. of 5 variables:
$ Sepal.Length: num 5.1 4.9 4.7 4.6 5 5.4 4.6 5 4.4 4.9 ...
$ Sepal.Width : num 3.5 3 3.2 3.1 3.6 3.9 3.4 3.4 2.9 3.1 ...
$ Petal.Length: num 1.4 1.4 1.3 1.5 1.4 1.7 1.4 1.5 1.4 1.5 ...
$ Petal.Width : num 0.2 0.2 0.2 0.2 0.2 0.4 0.3 0.2 0.2 0.1 ...
$ Species : Factor w/ 3 levels "setosa","versicolor",..: 1 1 1 1 1 1 1 1 1 1 ...
> iris_ctree <- ctree(Species ~ Sepal.Length + Sepal.Width + Petal.Length + Petal.Width, data=iris)
18
> print(iris_ctree)
Conditional inference tree with 4 terminal nodes
Response: Species
Inputs: Sepal.Length, Sepal.Width, Petal.Length, Petal.Width
Number of observations: 150
1) Petal.Length <= 1.9; criterion = 1, statistic = 140.264
2)* weights = 50
1) Petal.Length > 1.9
3) Petal.Width <= 1.7; criterion = 1, statistic = 67.894
4) Petal.Length <= 4.8; criterion = 0.999, statistic = 13.865
5)* weights = 46
4) Petal.Length > 4.8
6)* weights = 8
3) Petal.Width > 1.7
7)* weights = 46 19
plot(iris_ctree)
20
However… there is more
21
Bayes> cl <- kmeans(iris[,1:4], 3)
> table(cl$cluster, iris[,5])
setosa versicolor virginica
2 0 2 36
1 0 48 14
3 50 0 0
#
> m <- naiveBayes(iris[,1:4], iris[,5])
> table(predict(m, iris[,1:4]), iris[,5])
setosa versicolor virginica
setosa 50 0 0
versicolor 0 47 3
virginica 0 3 47 22
Using a contingency table> data(Titanic)
> mdl <- naiveBayes(Survived ~ ., data = Titanic)
> mdl
23
Naive Bayes Classifier for Discrete PredictorsCall: naiveBayes.formula(formula = Survived ~ ., data = Titanic)A-priori probabilities:Survived No Yes 0.676965 0.323035 Conditional probabilities: ClassSurvived 1st 2nd 3rd Crew No 0.08187919 0.11208054 0.35436242 0.45167785 Yes 0.28551336 0.16596343 0.25035162 0.29817159 SexSurvived Male Female No 0.91543624 0.08456376 Yes 0.51617440 0.48382560 AgeSurvived Child Adult No 0.03489933 0.96510067 Yes 0.08016878 0.91983122
Using a contingency table> predict(mdl, as.data.frame(Titanic)[,1:3])
[1] Yes No No No Yes Yes Yes Yes No No No No Yes Yes Yes Yes Yes No No No Yes Yes Yes Yes No
[26] No No No Yes Yes Yes Yes
Levels: No Yes
24
Naïve Bayes – what is it?• Example: testing for a specific item of
knowledge that 1% of the population has been informed of (don’t ask how).
• An imperfect test:– 99% of knowledgeable people test positive– 99% of ignorant people test negative
• If a person tests positive – what is the probability that they know the fact?
25
Naïve approach…• We have 10,000 representative people• 100 know the fact/item, 9,900 do not• We test them all:
– Get 99 knowing people testing knowing– Get 99 not knowing people testing not knowing– But 99 not knowing people testing as knowing
• Testing positive (knowing) – equally likely to know or not = 50%
26
Tree diagram
10000 ppl
1% know (100ppl)
99% test to know
(99ppl)
1% test not to know (1per)
99% do not know
(9900ppl)
1% test to know
(99ppl)
99% test not to know
(9801ppl)27
Relation between probabilities• For outcomes x and y there are probabilities
of p(x) and p (y) that either happened• If there’s a connection then the joint
probability - both happen = p(x,y)• Or x happens given y happens = p(x|y) or
vice versa then:– p(x|y)*p(y)=p(x,y)=p(y|x)*p(x)
• So p(y|x)=p(x|y)*p(y)/p(x) (Bayes’ Law)• E.g.
p(know|+ve)=p(+ve|know)*p(know)/p(+ve)= (.99*.01)/(.99*.01+.01*.99) = 0.5
28
How do you use it?• If the population contains x what is the
chance that y is true?
• p(SPAM|word)=p(word|SPAM)*p(SPAM)/p(word)
• Base this on data: – p(spam) counts proportion of spam versus not– p(word|spam) counts prevalence of spam
containing the ‘word’– p(word|!spam) counts prevalence of non-spam
containing the ‘word’ 29
Or..• What is the probability that you are in one
class (i) over another class (j) given another factor (X)?
• Invoke Bayes:
• Maximize p(X|Ci)p(Ci)/p(X) (p(X)~constant and p(Ci) are equal if not known)
• So: conditional indep - 30
• P(xk | Ci) is estimated from the training samples – Categorical: Estimate P(xk | Ci) as percentage of
samples of class i with value xk
• Training involves counting percentage of occurrence of each possible value for each class
– Numeric: Actual form of density function is generally not known, so “normal” density is often assumed
31
Thus..• Supervised or training set needed
• We will explore this more on Friday
32
Tentative assignments
• Assignment 4: Pattern, trend, relations: model development and evaluation. Due ~ March 7. 15% (10% written and 5% oral; individual);
• Assignment 5: Term project proposal. Due ~ March 18. 5% (0% written and 5% oral; individual);
• Term project (6). Due ~ week 13. 30% (25% written, 5% oral; individual).
• Assignment 7: Predictive and Prescriptive Analytics. Due ~ week 9/10. 20% (15% written and 5% oral; individual);
33
Coming weeks• I will be out of town Friday March 21 and 28• On March 21 you will have a lab –
attendance will be taken – to work on assignments (term (6) and assignment 7). Normal lecture on March 18.
• On March 28 you will have a lecture on SVM, thus the Tuesday March 25 will be a lab.
• Back to regular schedule in April (except 18th) 34
Admin info (keep/ print this slide)• Class: ITWS-4963/ITWS 6965• Hours: 12:00pm-1:50pm Tuesday/ Friday• Location: SAGE 3101• Instructor: Peter Fox• Instructor contact: [email protected], 518.276.4862 (do not
leave a msg)• Contact hours: Monday** 3:00-4:00pm (or by email appt)• Contact location: Winslow 2120 (sometimes Lally 207A
announced by email)• TA: Lakshmi Chenicheri [email protected] • Web site: http://tw.rpi.edu/web/courses/DataAnalytics/2014
– Schedule, lectures, syllabus, reading, assignments, etc.
35