-
Introduction to Predictive ModelsThe Bias Variance Tradeoff
Cross Validation
Some of the figures in this presentation are taken from An Introduction toStatistical Learning, with applications in R (Springer, 2013) with permission
from the authors: G. James, D. Witten, T. Hastie and R. Tibshirani
Carlos Carvalho Mladen Kolar, and Rob McCulloch
-
1. Introduction to Predictive Models2. Measuring Accuracy3. Out-of-Sample Predictions4. Bias-Variance Trade-Off5. Cross-Validation6. More on k-Nearest Neighbors, p > 17. Doing CV with a Bigger n
-
1. Introduction to Predictive Models
Simply put, the goal is to predict atarget variable Y with input variables X !
In Data Mining terminology this is know as supervised learning(also called Predictive Analytics).
In general, a useful way to think about it is that Y and X arerelated in the following way:
Yi = f (Xi ) + �i
The main purpose of this part of the course is to learn or estimatef (·) from data
1
-
Examples:
I Y: will a customer respond to a promotion (target marketing).
I Y: which customer is likely to cancel
I Y: the lifetime value of a customer (how much will theyspend).
I Y: pregnancy (from shopping behaviour) so you can target.
I Y: will a customer defect.
I Y: predict which products a customer will like (Pandora,Amazon).
I Y: predict age of death (insurance companies)
I ...
See Tables 1-9 after page 142 of “Predictive Analytics” by EricSiegel for many examples.
2
-
Y = f (X ) + �
I f (x): the part of Y you learn from X , the signal.
I �: the part of Y you don’t learn from X , the noise.
More generally,we want the conditional distribution of Y given X = x .
3
-
Example: Boston Housing
We might be interested in predicting the median house value as afunction of some measure of social economic level... here’s somedata:
Each observation corre-sponds to a town in theBoston area.
medv: median house value(data is old).
lstat: % lower status.
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●● ●
●● ●
●●
● ●● ●
●
●
●●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●● ●●
● ●
●●
● ●
●●● ●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●● ●●
● ●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
10 20 30
1020
3040
50
lstat
med
v
What should f (·) be?
4
-
How about this...
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●● ●
●● ●
●●
● ●● ●
●
●
●●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●● ●●
● ●
●●
● ●
●●● ●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●● ●●
● ●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
10 20 30
1020
3040
50
lstat
med
v
If lstat = 30 what is the prediction for medv?
5
-
or this?
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●● ●
●● ●
●●
● ●● ●
●
●
●●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●● ●●
● ●
●●
● ●
●●● ●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●● ●●
● ●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
10 20 30
1020
3040
50
lstat
med
v
If lstat = 30 what is the prediction for medv?
6
-
How do we estimate f (·)?
I Using training data:
{(X1,Y1), (X2,Y2), . . . , (Xn,Yn)}
I We use a statistical method to estimate the function f (·)I Two general methodological strategies:
1. simple parametric models (restricted assumptions about f (·))2. non-parametric models (flexibility in defining f (·))
Years of Education
Sen
iorit
y
Incom
e
Years of Education
Sen
iorit
y
Incom
e
7
-
Back to Boston Housing
Parametric Model Non-Parametric Model(Y = α + βx + �) (k-nearest neighbors)
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●● ●
●● ●
●●
● ●● ●
●
●
●●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●● ●●
● ●
●●
● ●
●●● ●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●● ●●
● ●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
10 20 30
1020
3040
50
lstat
med
v
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●● ●
●● ●
●●
● ●● ●
●
●
●●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●● ●●
● ●
●●
● ●
●●● ●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●● ●●
● ●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
10 20 3010
2030
4050
lstat
med
v
8
-
Simple parametric model:
Yi = α + β xi + �i
Using the training data,we estimate f (x) as
f̂ (x) = α̂ + β̂ x
where α̂ and β̂are the linearregression estimates.
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●● ●
●● ●
●●
● ●● ●
●
●
●●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●● ●●
● ●
●●
● ●
●●● ●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●● ●●
● ●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
10 20 30
1020
3040
50
lstat
med
v
9
-
To get this estimate we usedkNN- k-nearest neighbors.
To estimate f (xf ), average they values for the k training ob-servations with x closest to xf .
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●● ●
●● ●
●●
● ●● ●
●
●
●●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●● ●●
● ●
●●
● ●
●●● ●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●● ●●
● ●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
10 20 3010
2030
4050
lstat
med
v
What do I mean by closest?We will choose the k=50 points that are closest to the X value atwhich we are trying to predict.
10
-
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●● ●
●● ●
●●
● ●● ●
●
●
●●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●● ●●
● ●
●●
● ●
●●● ●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●● ●●
● ●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
10 20 30
1020
3040
50
k= 50
lstat
med
v
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
11
-
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●● ●
●● ●
●●
● ●● ●
●
●
●●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●● ●●
● ●
●●
● ●
●●● ●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●● ●●
● ●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
10 20 30
1020
3040
50
k= 50
lstat
med
v
●
●●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
12
-
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●● ●
●● ●
●●
● ●● ●
●
●
●●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●● ●●
● ●
●●
● ●
●●● ●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●● ●●
● ●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
10 20 30
1020
3040
50
k= 50
lstat
med
v
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ●
●●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
13
-
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●● ●
●● ●
●●
● ●● ●
●
●
●●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●● ●●
● ●
●●
● ●
●●● ●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●● ●●
● ●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
10 20 30
1020
3040
50
k= 50
lstat
med
v
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
● ●●
14
-
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●● ●
●● ●
●●
● ●● ●
●
●
●●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●● ●●
● ●
●●
● ●
●●● ●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●● ●●
● ●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
10 20 30
1020
3040
50
k= 50
lstat
med
v
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
15
-
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●●
●
●
●
●
●
●
●
●●
●●
●● ●
●● ●
●●
● ●● ●
●
●
●●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●●●
●
●
● ●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
● ●
●●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●●
●
●
●
●
●
●●● ●●
● ●
●●
● ●
●●● ●
●
●
●
●
●
●
●
●
●
●
●● ● ●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●●●
●●
●
●●
●●
●
●
●
●
●●
●
●● ●●
● ●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
10 20 30
1020
3040
50
k= 50
lstat
med
v
●
●
●●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●●●
●
16
-
Okay, that seems sensible, but, 2 neighbors or 200 neighbors?
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
● ●● ●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●● ●●
● ●
●
●
● ●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
● ●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
10 20 30
1020
3040
50
k= 2
lstat
med
v
17
-
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
● ●● ●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●● ●●
● ●
●
●
● ●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
● ●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
10 20 30
1020
3040
50
k= 10
lstat
med
v
18
-
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
● ●● ●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●● ●●
● ●
●
●
● ●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
● ●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
10 20 30
1020
3040
50
k= 50
lstat
med
v
19
-
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
● ●● ●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●● ●●
● ●
●
●
● ●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
● ●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
10 20 30
1020
3040
50
k= 100
lstat
med
v
20
-
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
● ●● ●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●● ●●
● ●
●
●
● ●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
● ●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
10 20 30
1020
3040
50
k= 150
lstat
med
v
21
-
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
● ●● ●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
