k-nearest neighborsjbhuang/teaching/ece...k-nearest neighbors jia-bin huang ece-5424g / cs-5824...
TRANSCRIPT
![Page 1: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/1.jpg)
K-Nearest Neighbors
Jia-Bin Huang
Virginia Tech Spring 2019ECE-5424G / CS-5824
![Page 2: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/2.jpg)
Administrative
• Check out review materials• Probability
• Linear algebra
• Python and NumPy
• Start your HW 0• On your Local machine: Install Anaconda, Jupiter notebook
• On the cloud: https://colab.research.google.com
• Sign up Piazza discussion forum
![Page 3: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/3.jpg)
Enrollment
• Maximum allowable capacity reached.
Students
Classroom
![Page 4: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/4.jpg)
Machine learning reading&study group
• Reading GroupTuesday 11 AM - 12:00 PMLocation: Whittmore Hall 457B• Research paper reading: machine learning, computer vision
• Study GroupThursday 11 AM - 12:00 PMLocation: Whittmore Hall 457B• Video lecture: machine learning
All are welcome. More info: https://github.com/vt-vl-lab/reading_group
![Page 5: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/5.jpg)
Recap: Machine learning algorithms
Supervised Learning
Unsupervised Learning
Discrete Classification Clustering
Continuous RegressionDimensionality
reduction
![Page 6: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/6.jpg)
Today’s plan
• Supervised learning• Setup• Basic concepts
•K-Nearest Neighbor (kNN)• Distance metric• Pros/Cons of nearest neighbor
•Validation, cross-validation, hyperparameter tuning
![Page 7: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/7.jpg)
Supervised learning
• Input: 𝑥 (Images, texts, emails)
• Output: 𝑦 (e.g., spam or non-spams)
• Data: 𝑥 1 , 𝑦 1 , 𝑥 2 , 𝑦 2 , ⋯ , 𝑥 𝑁 , 𝑦 𝑁 (Labeled dataset)
• (Unknown) Target function: 𝑓: 𝑥 → 𝑦 (“True” mapping)
• Model/hypothesis: ℎ: 𝑥 → 𝑦 (Learned model)
• Learning = search in hypothesis space Slide credit: Dhruv Batra
![Page 8: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/8.jpg)
Training set
Learning Algorithm
ℎ𝑥 𝑦
Hypothesis
![Page 9: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/9.jpg)
Training set
Learning Algorithm
ℎ𝑥 𝑦
HypothesisSize of house Estimated price
Regression
![Page 10: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/10.jpg)
Training set
Learning Algorithm
ℎ𝑥 𝑦
HypothesisUnseen image Predicted object classImage credit: CS231n @ Stanford
‘Mug’
Classification
![Page 11: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/11.jpg)
Procedural view of supervised learning
• Training Stage:• Raw data → 𝑥 (Feature Extraction)• Training data { 𝑥, 𝑦 } → ℎ (Learning)
• Testing Stage• Raw data → 𝑥 (Feature Extraction)• Test data 𝑥 → ℎ(𝑥) (Apply function, evaluate error)
Slide credit: Dhruv Batra
![Page 12: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/12.jpg)
Basic steps of supervised learning
• Set up a supervised learning problem
•Data collection: Collect training data with the “right” answer.
•Representation: Choose how to represent the data.
•Modeling: Choose a hypothesis class: 𝐻 = {ℎ: 𝑋 → 𝑌}
• Learning/estimation: Find best hypothesis in the model class.
•Model selection: Try different models. Picks the best one. (More on this later)
• If happy stop, else refine one or more of the aboveSlide credit: Dhruv Batra
![Page 13: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/13.jpg)
Nearest neighbor classifier
• Training data
𝑥 1 , 𝑦 1 , 𝑥 2 , 𝑦 2 , ⋯ , 𝑥 𝑁 , 𝑦 𝑁
• Learning
Do nothing.
• Testing
ℎ 𝑥 = 𝑦(𝑘), where 𝑘 = argmini 𝐷(𝑥, 𝑥(𝑖))
![Page 14: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/14.jpg)
Face recognition
Image credit: MegaFace
![Page 15: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/15.jpg)
Face recognition – surveillance application
![Page 16: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/16.jpg)
Music identification
https://www.youtube.com/watch?v=TKNNOMddkNc
![Page 17: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/17.jpg)
Album recognition (Instance recognition)
http://record-player.glitch.me/auth
![Page 18: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/18.jpg)
Scene Completion
(C) Dhruv Batra [Hayes & Efros, SIGGRAPH07]
![Page 19: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/19.jpg)
Hays and Efros, SIGGRAPH 2007
![Page 20: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/20.jpg)
… 200 total[Hayes & Efros, SIGGRAPH07]
![Page 21: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/21.jpg)
Context Matching
[Hayes & Efros, SIGGRAPH07]
![Page 22: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/22.jpg)
Graph cut + Poisson blending [Hayes & Efros, SIGGRAPH07]
![Page 23: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/23.jpg)
[Hayes & Efros, SIGGRAPH07]
![Page 24: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/24.jpg)
[Hayes & Efros, SIGGRAPH07]
![Page 25: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/25.jpg)
[Hayes & Efros, SIGGRAPH07]
![Page 26: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/26.jpg)
[Hayes & Efros, SIGGRAPH07]
![Page 27: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/27.jpg)
[Hayes & Efros, SIGGRAPH07]
![Page 28: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/28.jpg)
[Hayes & Efros, SIGGRAPH07]
![Page 29: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/29.jpg)
Synonyms
• Nearest Neighbors
• k-Nearest Neighbors
• Member of following families:• Instance-based Learning
• Memory-based Learning
• Exemplar methods
• Non-parametric methods
Slide credit: Dhruv Batra
![Page 30: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/30.jpg)
Instance/Memory-based Learning
1. A distance metric
2. How many nearby neighbors to look at?
3. A weighting function (optional)
4. How to fit with the local points?
Slide credit: Carlos Guestrin
![Page 31: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/31.jpg)
Instance/Memory-based Learning
1. A distance metric
2. How many nearby neighbors to look at?
3. A weighting function (optional)
4. How to fit with the local points?
Slide credit: Carlos Guestrin
![Page 32: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/32.jpg)
Recall: 1-Nearest neighbor classifier
• Training data
𝑥 1 , 𝑦 1 , 𝑥 2 , 𝑦 2 , ⋯ , 𝑥 𝑁 , 𝑦 𝑁
• Learning
Do nothing.
• Testing
ℎ 𝑥 = 𝑦(𝑘), where 𝑘 = argmini 𝐷(𝑥, 𝑥(𝑖))
![Page 33: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/33.jpg)
Distance metrics (𝑥: continuous variables )
• 𝐿2-norm: Euclidean distance 𝐷 𝑥, 𝑥′ = σ𝑖 𝑥𝑖 − 𝑥𝑖′2
• 𝐿1-norm: Sum of absolute difference 𝐷 𝑥, 𝑥′ = σ𝑖 |𝑥𝑖 − 𝑥𝑖′|
• 𝐿inf-norm 𝐷 𝑥, 𝑥′ = max( 𝑥𝑖 − 𝑥𝑖′ )
• Scaled Euclidean distance 𝐷 𝑥, 𝑥′ = σ𝑖 𝜎𝑖2 𝑥𝑖 − 𝑥𝑖′
2
• Mahalanobis distance 𝐷 𝑥, 𝑥′ = 𝑥 − 𝑥′ ⊤𝐴(𝑥 − 𝑥′)
![Page 34: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/34.jpg)
Distance metrics (𝑥: discrete variables )
• Example application: document classification
• Hamming distance
![Page 35: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/35.jpg)
Distance metrics (𝑥: Histogram / PDF)
• Histogram intersection
• Chi-squared Histogram matching distance
• Earth mover’s distance (Cross-bin similarity measure)• minimal cost paid to transform one distribution into the other
histint 𝑥, 𝑥′ = 1 −
𝑖
min(𝑥𝑖 , 𝑥𝑖′)
𝜒2 𝑥, 𝑥′ =1
2
𝑖
𝑥𝑖 − 𝑥𝑖′ 2
𝑥𝑖 + 𝑥𝑖′
[Rubner et al. IJCV 2000]
![Page 36: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/36.jpg)
Distance metrics (𝑥: gene expression microarray data)
• When “shape” matters more than values
• Want 𝐷(𝑥(1), 𝑥(2)) < 𝐷(𝑥(1), 𝑥(3))
• How?
• Correlation Coefficients• Pearson, Spearman, Kendal, etc
𝑥(1)
𝑥(3)
𝑥(2)
Gene
![Page 37: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/37.jpg)
Distance metrics (𝑥: Learnable feature)
Large margin nearest neighbor (LMNN)
![Page 38: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/38.jpg)
Instance/Memory-based Learning
1. A distance metric
2. How many nearby neighbors to look at?
3. A weighting function (optional)
4. How to fit with the local points?
Slide credit: Carlos Guestrin
![Page 39: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/39.jpg)
kNN Classification
k = 3
k = 5
Image credit: Wikipedia
![Page 40: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/40.jpg)
Classification decision boundaries
Image credit: CS231 @ Stanford
![Page 41: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/41.jpg)
Instance/Memory-based Learning
1. A distance metric
2. How many nearby neighbors to look at?
3. A weighting function (optional)
4. How to fit with the local points?
Slide credit: Carlos Guestrin
![Page 42: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/42.jpg)
Issue: Skewed class distribution
• Problem with majority voting in kNN
• Intuition: nearby points should be weightedstrongly, far points weakly
• Apply weight
𝑤(𝑖) = exp(−𝑑 𝑥 𝑖 , 𝑞𝑢𝑒𝑟𝑦
2
𝜎2)
• 𝜎2: Kernel width
?
![Page 43: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/43.jpg)
Instance/Memory-based Learning
1. A distance metric
2. How many nearby neighbors to look at?
3. A weighting function (optional)
4. How to fit with the local points?
Slide credit: Carlos Guestrin
![Page 44: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/44.jpg)
1-NN for Regression
• Just predict the same output as the nearest neighbour.
x
y
Here, this is the closest datapoint
Figure credit: Carlos Guestrin
![Page 45: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/45.jpg)
1-NN for Regression
• Often bumpy (overfits)
Figure credit: Andrew Moore
![Page 46: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/46.jpg)
9-NN for Regression
• Predict the averaged of k nearest neighbor values
Figure credit: Andrew Moore
![Page 47: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/47.jpg)
Weighting/Kernel functions
Weight
𝑤(𝑖) = exp(−𝑑 𝑥 𝑖 , 𝑞𝑢𝑒𝑟𝑦
2
𝜎2)
Prediction (use all the data)
𝑦 =
𝑖
𝑤 𝑖 𝑦 𝑖 /
𝑖
𝑤(𝑖)
(Our examples use Gaussian)
Slide credit: Carlos Guestrin
![Page 48: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/48.jpg)
Effect of Kernel Width
• What happens as σinf?
• What happens as σ0?
Slide credit: Ben Taskar
Kernel regression
![Page 49: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/49.jpg)
Problems with Instance-Based Learning• Expensive
• No Learning: most real work done during testing
• For every test sample, must search through all dataset – very slow!
• Must use tricks like approximate nearest neighbour search
• Doesn’t work well when large number of irrelevant features• Distances overwhelmed by noisy features
• Curse of Dimensionality• Distances become meaningless in high dimensions
Slide credit: Dhruv Batra
![Page 50: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/50.jpg)
Curse of dimensionality
• Consider a hypersphere with radius 𝑟 and dimension 𝑑
• Consider hypercube with edge of length 2𝑟
• Distance between center and the corners is 𝑟 𝑑
• Hypercube consist almost entirely of the “corners”
𝑑= 2
2𝑟
2𝑟
![Page 51: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/51.jpg)
Hyperparameter selection
• How to choose K?
• Which distance metric should I use? L2, L1?
• How large the kernel width 𝜎2 should be?
• ….
![Page 52: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/52.jpg)
Tune hyperparameters on the test dataset?
• Will give us a stronger performance on the test set!
• Why this is not okay? Let’s discuss
Evaluate on the test set only a single time, at the very end.
![Page 53: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/53.jpg)
Validation set
• Spliting training set: A fake test set to tune hyper-parameters
Slide credit: CS231 @ Stanford
![Page 54: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/54.jpg)
Cross-validation
• 5-fold cross-validation -> split the training data into 5 equal folds
• 4 of them for training and 1 for validation
Slide credit: CS231 @ Stanford
![Page 55: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/55.jpg)
Hyper-parameters selection
• Split training dataset into train/validation set (or cross-validation)
• Try out different values of hyper-parameters and evaluate these models on the validation set
• Pick the best performing model on the validation set
• Run the selected model on the test set. Report the results.
![Page 56: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/56.jpg)
Things to remember
• Supervised Learning• Training/testing data; classification/regression; Hypothesis
• k-NN• Simplest learning algorithm• With sufficient data, very hard to beat “strawman” approach
• Kernel regression/classification• Set k to n (number of data points) and chose kernel width• Smoother than k-NN
• Problems with k-NN• Curse of dimensionality• Not robust to irrelevant features • Slow NN search: must remember (very large) dataset for prediction
![Page 57: K-Nearest Neighborsjbhuang/teaching/ECE...K-Nearest Neighbors Jia-Bin Huang ECE-5424G / CS-5824 Virginia Tech Spring 2019. ... •K-Nearest Neighbor (kNN) •Distance metric •Pros/Cons](https://reader031.vdocuments.net/reader031/viewer/2022013023/601b04201a007f08685def44/html5/thumbnails/57.jpg)
Next class: Linear Regression
Price ($)in 1000’s
500 1000 1500 2000 2500
100
200
300
400
Size in feet^2