statistical significance and population controls presented to the new jersey sdc annual network...
TRANSCRIPT
![Page 1: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/1.jpg)
Statistical Significance and Population Controls
Presented to the New Jersey SDC Annual Network Meeting
June 6, 2007
Tony Tersine, U.S. Census Bureau
![Page 2: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/2.jpg)
2
Overview of the Session
• Basic Concepts
• Margin of Error
• Confidence Intervals
• Standard Error Formulas
• Statistical Testing
• Population Controls
![Page 3: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/3.jpg)
3
Basic Concepts - 1• Sampling error is introduced due to
sampling, selection of a subset of the population to draw inferences about the entire population.
• Standard error is an estimate of the precision of the estimates. It measures the variability of an estimate due to sampling.
![Page 4: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/4.jpg)
4
Basic Concepts - 2
• The sampling error is often reported as the estimate “plus or minus” the margin of error, a measure of how precise the estimate is.
• The margin of error describes the precision of the estimate at a given confidence level.
![Page 5: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/5.jpg)
5
Basic Concepts - 3
• The confidence level measures the likelihood that the true value is within the margin of error of the sample estimate.
• The Census Bureau statistical standard for published data is to use the 90 percent confidence level.
![Page 6: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/6.jpg)
6
Margin of Error• The margin of error is important
because relying on statistical inference can save you from drawing incorrect conclusions from data based on a sample.
• It can help prevent you from interpreting small or nonexistent differences as important.
![Page 7: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/7.jpg)
7
Margin of Error (MOE)
MOE = 1.65 * Standard Error1.65 is used for the 90 percent confidence level
Standard Error = MOE / 1.65
Starting in 2006 ACS will use 1.645
![Page 8: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/8.jpg)
8
Confidence Interval• Confidence Interval
Estimate ± Margin of Error
• 90 percent confidence levelMargin of Error = 1.645 * Std Error
• 95 percent confidence levelMargin of Error = 1.96 * Std Error
![Page 9: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/9.jpg)
9
Confidence Interval
• The confidence interval tells you the upper and lower bounds of a range of values that may contain the true value.
• It provides important information about the true value or the population parameter.
• It tells you the limitations on using the estimates.
![Page 10: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/10.jpg)
10
MOE / Confidence Interval
Median Family Income – $30,000Standard Error – $1,500
90% MOE = 1.645 * $1,500 = $2,468
90% CI = $30,000 ± $2,468 = $27,532 to $32,468
![Page 11: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/11.jpg)
11
Standard Error – Sum/Difference
• Standard Error of X + Y or X – Y
• SE(X+Y) = SE(X-Y)
22 [SE(Y)] [SE(X)]
![Page 12: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/12.jpg)
12
Standard Error – Sum
• SE(X1+X2+…+Xn)
2n
22
21 )][SE(X ...)][SE(X )][SE(X
![Page 13: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/13.jpg)
13
Standard Error – Proportions
• P= X / Y – X is a subset of Y
• SE(P)
22
22 [SE(Y)]
Y
X [SE(X)]
Y
1
![Page 14: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/14.jpg)
14
Standard Error – Ratios
• X / Y – X is not a subset of Y
• SE(X / Y)
22
22 [SE(Y)]
Y
X [SE(X)]
Y
1
![Page 15: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/15.jpg)
15
Statistical Testing
• Two estimates are "significantly different" at the 90 percent confidence level if the difference between them is large enough to infer that there was a less than 10 percent chance that the difference was purely random.
• Users may want to compare estimates across years or geographies.
• It is important to note that small differences, which may be statistically significant, may not have any practical significance.
![Page 16: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/16.jpg)
16
Statistical Testing - Steps
1. State that two estimates are statistically different if the difference between the two estimates is statistically different from zero.
2. Calculate the standard error of the difference.
![Page 17: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/17.jpg)
17
Statistical Testing - Steps
3. Calculate the margin of error of the difference.
4. Compare the original difference between the estimates to the margin of error of the difference.
![Page 18: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/18.jpg)
18
Statistical Testing - Steps
5. If the difference is greater than the margin of error, then you conclude that the two estimates are significantly different.
6. If the difference is less than the margin of error, you conclude that the two estimates are not significantly different.
![Page 19: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/19.jpg)
19
Statistical Testing - Example
Percent with Bachelor’s Degree or Higher
Geography Percent MOE CIArea 1 20.0 ± 5.0 15.0-25.0Area 2 12.3 ± 4.7 7.6-17.0
Difference = 20.0 – 12.3 = 7.7
![Page 20: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/20.jpg)
20
Statistical Testing - Example
MOE of the Difference - Standard Errors for Each EstimateSE = MOE / 1.65
SE(Area 1) = 5.0 / 1.65 = 3.03
SE(Area 2) = 4.7 / 1.65 = 2.85
![Page 21: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/21.jpg)
21
Statistical Testing - Example
• Standard Error of the Difference
• Margin of Error of the Difference
MOE(X - Y) = 1.65 * 4.16 = 6.9
16.42.85 3.03 Y)SE(X 22
![Page 22: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/22.jpg)
22
Statistical Testing - Example
• Compare the Difference to MOE– Difference = 7.7%– MOE = 6.9%
• Difference > MOE
• Conclude that the two estimates are significantly different with 90 percent confidence
![Page 23: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/23.jpg)
23
Census 2000 – Example
• Percent Bachelor’s Degree or Higher – Alexandria, VA• 51,982 / 95,730 = 54.3%• DF = 1.2 – (13.4% in sample)
• 90% MOE =1.65 * 0.4 = 0.7 • 90% CI = 54.3 ± 0.7 = 53.6 to 55.0
4.0)3.54100(*3.54*730,95
5*2.1
SE
![Page 24: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/24.jpg)
24
Rules to Remember
• Don’t make a big deal of small differences.
• If the confidence intervals overlap you cannot conclude the difference is not statistically significant.
• Always talk to subject matter experts before making any conclusions.
![Page 25: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/25.jpg)
25
Population Controls - Rational
• Correct for coverage– Higher undercoverage in surveys than in
census
• Reduce variance estimates
![Page 26: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/26.jpg)
26
2005 ACS Coverage Rates - US
Total Pop
Male Female Hispanic
95.1 93.9 96.2 93.6
Non-HispanicWhite Black AIAN Asian NHOPI
96.3 90.7 97.9 94.5 84.0
![Page 27: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/27.jpg)
27
Population Controls
• Intercensal estimates are produced by updating the previous census results using various administrative records data and ACS data on foreign-born
• In a multi-stage process, HU, GQ, and population adjustment ratios are applied to the weights
![Page 28: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/28.jpg)
28
GQ Controls
• GQ population controls applied at the STATE level by 7 major types.
• Collapsing across types if not enough sample
• Always control to at least Institutional / Non-Institutional Population
![Page 29: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/29.jpg)
29
Housing Unit Controls
• Applied at a weighting area level
• New step to make all 3 agree– Households– Householders– Occupied Housing Units
• Housing Units will not be controlled
![Page 30: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/30.jpg)
30
Weighting Areas
• Controls applied at the weighting area (county or group of counties) level
• 1343 weighting areas consist of a single county– All 21 New Jersey counties are weighting
areas
• The other 607 weighting areas are made up of 1798 counties
![Page 31: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/31.jpg)
31
HU Population Controls
• Controls applied by race/ethnicity and age/sex groups
• ACS GQ estimates subtracted from population estimates to obtain controls
• Collapsing of race/ethnicity and age/sex groups
![Page 32: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/32.jpg)
32
Why Do Place Estimates Differ
• ACS does not control subcounty areas
• 1-person households– Lower response rate
• Multi-Unit Structures– Conversion of single to multi-unit
![Page 33: Statistical Significance and Population Controls Presented to the New Jersey SDC Annual Network Meeting June 6, 2007 Tony Tersine, U.S. Census Bureau](https://reader034.vdocuments.net/reader034/viewer/2022051819/5515564c55034674578b45da/html5/thumbnails/33.jpg)
33
Room: 4H477Phone: 301-763-1994Email: [email protected]
U.S. DEPARTMENT OF COMMERCEU.S. Census BureauWashington, DC 20233
Anthony Tersine
Contact Information