nathaniel e. helwig - statisticsusers.stat.umn.edu/~helwig/notes/npreg-notes.pdfin...
TRANSCRIPT
![Page 1: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/1.jpg)
Introduction to Nonparametric Regression
Nathaniel E. Helwig
Assistant Professor of Psychology and StatisticsUniversity of Minnesota (Twin Cities)
Updated 04-Jan-2017
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 1
![Page 2: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/2.jpg)
Copyright
Copyright c© 2017 by Nathaniel E. Helwig
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 2
![Page 3: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/3.jpg)
Outline of Notes
1) Need for NP RegMotivating exampleNonparametric regression
2) Local AveragingOverviewExamples
3) Local Regression:OverviewExamples
4) Kernel Smoothing:OverviewExamples
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 3
![Page 4: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/4.jpg)
Need for Nonparametric Regression
Need for NonparametricRegression
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 4
![Page 5: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/5.jpg)
Need for Nonparametric Regression Motivating Example
Results From Four Hypothetical Studies
4 6 8 10 12 14
05
1015
Study 1: y = 3 + 0.5x
x
y
R2 = 0.67
4 6 8 10 12 14
05
1015
Study 2: y = 3 + 0.5x
x
y
R2 = 0.67
4 6 8 10 12 14
05
1015
Study 3: y = 3 + 0.5x
x
y
R2 = 0.67
4 6 8 10 12 140
510
15
Study 4: y = 3 + 0.5x
x
y
R2 = 0.67
Figure: Estimated linear relationship from four hypothetical studies.
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 5
![Page 6: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/6.jpg)
Need for Nonparametric Regression Motivating Example
Implications of Four Hypothetical Studies
What do the results on the previous slide imply?
Can we conclude that there is a linear relationship between X and Y?
Is the reproducibility of the finding indicative of a valid discovery?
What have we learned about the data from these results?
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 6
![Page 7: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/7.jpg)
Need for Nonparametric Regression Motivating Example
Let’s Look at the Data
●●
●
●●
●
●
●
●
●●
4 6 8 10 12 14
05
1015
Study 1: y = 3 + 0.5x
x
y
R2 = 0.67
●●
●●●
●
●
●
●
●
●
4 6 8 10 12 14
05
1015
Study 2: y = 3 + 0.5x
x
y
R2 = 0.67
●●
●
●●
●
●●
●
●●
4 6 8 10 12 14
05
1015
Study 3: y = 3 + 0.5x
x
y
R2 = 0.67
●●
●●●
●
●
●
●
●●
8 10 12 14 16 180
510
15
Study 4: y = 3 + 0.5x
x
y
R2 = 0.67
Figure: Estimated linear relationship with corresponding data.
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 7
![Page 8: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/8.jpg)
Need for Nonparametric Regression Motivating Example
Anscombe’s (1973) Quartet
> anscombex1 x2 x3 x4 y1 y2 y3 y4
1 10 10 10 8 8.04 9.14 7.46 6.582 8 8 8 8 6.95 8.14 6.77 5.763 13 13 13 8 7.58 8.74 12.74 7.714 9 9 9 8 8.81 8.77 7.11 8.845 11 11 11 8 8.33 9.26 7.81 8.476 14 14 14 8 9.96 8.10 8.84 7.047 6 6 6 8 7.24 6.13 6.08 5.258 4 4 4 19 4.26 3.10 5.39 12.509 12 12 12 8 10.84 9.13 8.15 5.5610 7 7 7 8 4.82 7.26 6.42 7.9111 5 5 5 8 5.68 4.74 5.73 6.89
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 8
![Page 9: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/9.jpg)
Need for Nonparametric Regression Nonparametric Regression
Parametric versus Nonparametric Regression
The general linear model is a form of parametric regression, where therelationship between X and Y has some predetermined form.
Parameterizes relationship between X and Y , e.g., Y = β0 + β1XThen estimates the specified parameters, e.g., β0 and β1
Great if you know the form of the relationship (e.g., linear)
In contrast, nonparametric regression tries to estimate the form of therelationship between X and Y .
No predetermined form for relationship between X and YGreat for discovering relationships and building prediction models
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 9
![Page 10: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/10.jpg)
Need for Nonparametric Regression Nonparametric Regression
Problem of Interest
Smoothers (aka nonparametric regression) try to estimate functionsfrom noisy data.
Suppose we have n pairs of points (xi , yi) for i ∈ {1, . . . ,n}, andWLOG assume that x1 ≤ x2 ≤ · · · ≤ xn.
Also, suppose the following assumptions hold:(A1) There is a functional relationship between x and y of the form
yi = η(xi) + εi ; i ∈ {1, . . . ,n}
(A2) The εi are iid from some distribution f (x) with zero mean
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 10
![Page 11: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/11.jpg)
Local Averaging
Local Averaging
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 11
![Page 12: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/12.jpg)
Local Averaging Overview
Friedman’s (1984) Local Averaging
To estimate η at the point xi , we could calculate the average of the yjvalues corresponding to xj values that are “near” xi .
Friedman (1984) defined “near” as the smallest symmetric windowaround xi that contains s observations.
Note that s is called the spanSize of averaging window can differ for each xi
But always use s points in averaging window
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 12
![Page 13: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/13.jpg)
Local Averaging Overview
Selecting the Span
Friedman proposed using a cross-validation approach to select span s.
For a given span s, leave-one-out cross-validation:Let y(i) denote the local averaging estimate of η at the point xiobtained by holding out the i-th pair (xi , yi)
Define CV residuals ei(s) = yi − y(i); note residual is function of s
s = mins∈S(1/n)∑n
i=1 e2i (s)
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 13
![Page 14: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/14.jpg)
Local Averaging Examples
Local Averaging Example 1: sunspots data
1750 1800 1850 1900 1950
050
150
250
Raw Data
years
suns
pots
1750 1800 1850 1900 1950
050
100
150
span=0.01
sunlocavg$x
sunl
ocav
g$y
1750 1800 1850 1900 1950
2060
100
span=0.05
sunlocavg$x
sunl
ocav
g$y
1750 1800 1850 1900 1950
2040
6080
span=cv
sunlocavg$x
sunl
ocav
g$y
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 14
![Page 15: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/15.jpg)
Local Averaging Examples
Local Averaging Example 1: R code
data(sunspots)yrs=start(sunspots)yre=end(sunspots)years=seq(yrs[1]+yrs[2]/12,yre[1]+yre[2]/12,by=1/12)dev.new(width=8,height=6,noRStudioGD=TRUE)par(mfrow=c(2,2))plot(years,sunspots,type="l",main="Raw Data")sunlocavg=supsmu(years,sunspots,span=0.01)plot(sunlocavg,type="l",main="span=0.01")sunlocavg=supsmu(years,sunspots,span=0.05)plot(sunlocavg,type="l",main="span=0.05")sunlocavg=supsmu(years,sunspots)plot(sunlocavg,type="l",main="span=cv")
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 15
![Page 16: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/16.jpg)
Local Averaging Examples
Local Averaging Example 2: simulated data
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
0.0 0.2 0.4 0.6 0.8 1.0
−1
01
2
x
y
ηη
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 16
![Page 17: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/17.jpg)
Local Averaging Examples
Local Averaging Example 2: R code
> set.seed(1)> x=seq(0,1,length=50)> y=sin(2*pi*x)+rnorm(50,sd=0.5)> locavg=supsmu(x,y)> dev.new(width=6,height=6,noRStudioGD=TRUE)> plot(x,y)> lines(locavg$x,locavg$y)> lines(locavg$x,sin(2*pi*x),lty=2)> legend("bottomleft",c(expression(hat(eta)),+ expression(eta)),lty=1:2,bty="n")
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 17
![Page 18: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/18.jpg)
Local Regression
Local Regression
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 18
![Page 19: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/19.jpg)
Local Regression Overview
Cleveland’s (1979) Local Regression
To estimate η at the point xi , we could calculate the local linearregression line using the (xj , yj) points “near” xi .
LOWESS: LOcally WEighted Scatterplot SmoothingLOESS: LOcal regrESSion
Cleveland (1979) proposed using weighted regression with weightsrelated to distance of xj points to xi .
Weight function is scaled so only proportion of (xj , yj) points areused in each regressionSize of regression window can differ for each xi
But use αn points in each regression where α ∈ (0,1]
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 19
![Page 20: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/20.jpg)
Local Regression Overview
Weighted Local Regression
Cleveland proposed using a weight function W such that
W (x){> 0, |x | < 1= 0, |x | ≥ 1
Then W is modified for each index i ∈ {1, . . . ,n} by. . .Centering W at xi
Scaling W such that αn values are nonzero
R’s loess uses tricube function: W (x) = (1− |x |3)3 for |x | < 1
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 20
![Page 21: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/21.jpg)
Local Regression Overview
Weighted Local Regression (continued)
Let {w ij }nj=1 denote the weights for a particular point xi .
The weighted local regression problem minimizes
n∑j=1
w ij (yj − β i
0 − β i1xj)
2
where β i0 and β i
1 are intercept and slope between x and y inneighborhood around xi
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 21
![Page 22: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/22.jpg)
Local Regression Overview
Robust Weighted Local Regression
To reduce effect of outliers, we can perform another regression withweights based on the residuals εi = yi − yi where yi = β i
0 + β i1xi .
Bisquare weight function: B(x) = (1− x2)2, |x | < 1Residual-based weights: δi = B
(εi/(6median1≤j≤n|εj |)
)
The robust weighted local regression problem minimizes
n∑j=1
δjw ij (yj − β i ′
0 − β i ′1 xj)
2
where β i ′0 and β i ′
1 are the robust (i.e., residual corrected) intercept andslope between x and y in neighborhood around xi
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 22
![Page 23: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/23.jpg)
Local Regression Overview
Selecting the Span
Want to minimize the leave-one-out cross-validation criterion:
1n
n∑i=1
(yi − y(i))2
where y(i) is the LOESS estimate of yi obtained by holding out (xi , yi).
Rewrite the leave-one-out cross-validation criterion as
1n
n∑i=1
(yi − yi)2
(1− hii)2
where hii are diagonal entries of the hat matrix H that determines yi .Replace hii with 1
n∑n
i=1 hii = tr(H)/n for generalized CV
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 23
![Page 24: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/24.jpg)
Local Regression Examples
Local Regression Example 1: sunspots data
1750 1800 1850 1900 1950
050
150
250
Raw Data
years
suns
pots
1750 1800 1850 1900 1950
050
100
200
span=0.01
sunlocreg$x
sunl
ocre
g$fit
ted
1750 1800 1850 1900 1950
050
100
150
span=0.05
sunlocreg$x
sunl
ocre
g$fit
ted
1750 1800 1850 1900 1950
2040
6080
span=gcv
sunlocreg$x
sunl
ocre
g$fit
ted
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 24
![Page 25: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/25.jpg)
Local Regression Examples
Local Regression Example 1: R code
library(fANCOVA)data(sunspots)yrs=start(sunspots)yre=end(sunspots)years=seq(yrs[1]+yrs[2]/12,yre[1]+yre[2]/12,by=1/12)dev.new(width=8,height=6,noRStudioGD=TRUE)par(mfrow=c(2,2))plot(years,sunspots,type="l",main="Raw Data")sunlocreg=loess(sunspots~years,span=0.01)plot(sunlocreg$x,sunlocreg$fitted,type="l",main="span=0.01")sunlocreg=loess(sunspots~years,span=0.05)plot(sunlocreg$x,sunlocreg$fitted,type="l",main="span=0.05")sunlocreg=loess.as(years,sunspots,criterion="gcv")plot(sunlocreg$x,sunlocreg$fitted,type="l",main="span=gcv")
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 25
![Page 26: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/26.jpg)
Local Regression Examples
Local Regression Example 2: simulated data
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
0.0 0.2 0.4 0.6 0.8 1.0
−1
01
2
x
y
ηη
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 26
![Page 27: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/27.jpg)
Local Regression Examples
Local Regression Example 2: R code
> library(fANCOVA)> set.seed(55455)> x=seq(0,1,length=50)> y=sin(2*pi*x)+rnorm(50,sd=0.5)> locreg=loess.as(x,y)> dev.new(width=6,height=6,noRStudioGD=TRUE)> plot(x,y)> lines(locreg$x,locreg$fitted)> lines(locreg$x,sin(2*pi*x),lty=2)> legend("bottomleft",c(expression(hat(eta)),+ expression(eta)),lty=1:2,bty="n")
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 27
![Page 28: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/28.jpg)
Kernel Smoothing
Kernel Smoothing
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 28
![Page 29: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/29.jpg)
Kernel Smoothing Overview
Kernel Smoothing for General Functions
Kernel smoothing extends KDE idea to estimation a general function η.
Nadaraya (1964) and Watson (1964) independently introduced thekernel regression estimate
η(x) =∑n
i=1 yiK(x−xi
h
)∑ni=1 K
( x−xih
) =n∑
i=1
yiwi
where weights wi =K(
x−xih
)∑n
i=1 K(
x−xih
) are dependent on chosen K and h.
CV, GCV, or AIC to select bandwidth in kernel regression problems.
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 29
![Page 30: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/30.jpg)
Kernel Smoothing Examples
Kernel Smoothing Example 1: sunspots data
1750 1800 1850 1900 1950
050
150
250
Raw Data
years
suns
pots
1750 1800 1850 1900 1950
050
150
250
bw=0.01
years
fitte
d(su
nker
reg)
1750 1800 1850 1900 1950
050
150
250
bws=0.05
years
fitte
d(su
nker
reg)
1750 1800 1850 1900 1950
050
150
bws=cv (0.09)
years
fitte
d(su
nker
reg)
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 30
![Page 31: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/31.jpg)
Kernel Smoothing Examples
Kernel Smoothing Example 1: R code
library(np)data(sunspots)yrs=start(sunspots)yre=end(sunspots)sunspots=as.vector(sunspots)years=seq(yrs[1]+yrs[2]/12,yre[1]+yre[2]/12,by=1/12)dev.new(width=8,height=6,noRStudioGD=TRUE)par(mfrow=c(2,2))plot(years,sunspots,type="l",main="Raw Data")sunkerreg=npreg(bws=0.01,txdat=years,tydat=sunspots)plot(years,fitted(sunkerreg),type="l",main="bw=0.01")sunkerreg=npreg(bws=0.05,txdat=years,tydat=sunspots)plot(years,fitted(sunkerreg),type="l",main="bws=0.05")sunkerreg=npreg(txdat=years,tydat=sunspots)plot(years,fitted(sunkerreg),type="l",main="bws=cv (0.09)")
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 31
![Page 32: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/32.jpg)
Kernel Smoothing Examples
Kernel Smoothing Example 2: simulated data
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
0.0 0.2 0.4 0.6 0.8 1.0
−1
01
2
x
y
ηη
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 32
![Page 33: Nathaniel E. Helwig - Statisticsusers.stat.umn.edu/~helwig/notes/npreg-Notes.pdfIn contrast,nonparametric regressiontries to estimate the form of the relationship between X and Y](https://reader033.vdocuments.net/reader033/viewer/2022042112/5e8da313c09fc73bb17c3f87/html5/thumbnails/33.jpg)
Kernel Smoothing Examples
Kernel Smoothing Example 2: R code
> library(np)> set.seed(55455)> x=seq(0,1,length=50)> y=sin(2*pi*x)+rnorm(50,sd=0.5)> kerreg=npreg(txdat=x,tydat=y)
> dev.new(width=6,height=6,noRStudioGD=TRUE)> plot(x,y)> lines(x,fitted(kerreg))> lines(x,sin(2*pi*x),lty=2)> legend("bottomleft",c(expression(hat(eta)),+ expression(eta)),lty=1:2,bty="n")
Nathaniel E. Helwig (U of Minnesota) Introduction to Nonparametric Regression Updated 04-Jan-2017 : Slide 33