![Page 1: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/1.jpg)
1
Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates
Joanne Clements
Ruth Fulton
Alison Whitworth
![Page 2: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/2.jpg)
2
Context
• Improving Migration and Population Statistics (IMPS) Project
• Quality Strand
• “Establish quality measures for population statistics”
• No international precedent for this work
![Page 3: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/3.jpg)
3
Issues
• Estimates compiled from a wide range of administrative sources plus some survey and Census data
• Source data subject to sampling and non-sampling errors
• Lack of independent data with which to corroborate
• How to estimate each potential error and combine these in one measure?
![Page 4: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/4.jpg)
4
Aim and Objectives
• AimImprove understanding, measurement and reporting
of the quality of population estimates
• Objectives– Describing the sources of uncertainty– Developing methods for measuring uncertainty for
each issue and combining them into one measure– Eventually feeding findings into ONS quality
reports
![Page 5: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/5.jpg)
5
Presentation Outline
• Summarise population estimates methodology
• Summarise previous research on quality• Detail proposed error measurement
methodology– Illustrate by applying to local authority (LA) mid-
2006 population estimates
• Outline emerging proposals for further work to achieve robust quality measures
![Page 6: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/6.jpg)
6
Calculating LA Population Estimates
e.g. Southampton UA
![Page 7: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/7.jpg)
7
Calculating LA Population Estimates (cont)
Adjustment
![Page 8: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/8.jpg)
8
Calculating LA Population Estimates
![Page 9: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/9.jpg)
9
Previous Research
Quality of Population Estimates• Past experience of inter-censal errors • Sampling error and expert opinion of non-
sampling error in components of estimates
Quality of Population Projections• Accuracy of past projections• Use of variant projections• Simulation methods using error distributions
for the components of projections (stochastic forecasting)
![Page 10: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/10.jpg)
10
Proposed Methodology: Initial Assessment of Quality Issues
• Map out the procedures and data sources used to derive population estimates
• Identify associated quality issues
• Identify the importance of these issues
![Page 11: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/11.jpg)
11
LA Population Estimates:Initial Assessment of Quality Issues
• Brief assessment of the evidence for each component
• For example: Internal Migration– Relies on GP registration data– Assumes patients reregister within a month of
moving (known issue for students leaving university)
![Page 12: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/12.jpg)
12
Proposed Methodology (cont):Detailed Investigation of Quality Issues
• Quantify uncertainty using statistical theory, empirical evidence and / or expert opinion
• Both sampling and non-sampling errors
![Page 13: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/13.jpg)
13
LA Population Estimates:Detailed Investigation of Quality Issues
• Attributing a potential uncertainty range and distribution to each component
• Not each quality issue
• Made relatively simplistic distribution assumptions (Normal or Uniform)
• Assumed same level of uncertainty across LAs
![Page 14: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/14.jpg)
14
LA Population Estimates:Detailed Investigation of Quality Issues
• Estimating uncertainty relative to size of local authority component
• For example: Could assume potential error in annual local authority births estimate
N(0, X% of estimated births)
– Assume similar error distributions by year
![Page 15: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/15.jpg)
15
Proposed Methodology (cont):Overall Quality Measure
• Mathematically complicated to combine a large number of potential error measures into one quality measure– Errors may be correlated– Distributions not all normally distributed
• Developed a Simulation methodology
![Page 16: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/16.jpg)
16
LA Population Estimates: Simulation
• For each local authority randomly generate errors for each component – Using previously developed error distributions
Mid-2001 error estimate+ Births 01/02 error estimate - Deaths 01/02 error estimate+ Internal In-Migrants 01/02 error estimate - Internal Out-Migrants 01/02 error estimate+ …..+ Births 02/03 error estimate - Deaths 02/03 error estimate+ …
![Page 17: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/17.jpg)
17
LA Population Estimates: Simulation
• Calculate error in mid-2006 estimate by combining the errors generated for each component in each year up to 2006
• Repeat process 1000 times
• Obtain distribution of potential error in mid-2006 local authority estimate
![Page 18: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/18.jpg)
18
Findings: Potential Error Distribution
![Page 19: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/19.jpg)
19
Findings: Measuring Uncertainty in Population Estimates
• Simulation methodology allows measures of uncertainty to be calculated for population estimates.
• But, in reality, there is uncertainty in these measures of uncertainty, as…– Only as good as the error assumptions made for
each issue / component of change– Very difficult to exactly measure non-sampling
error
![Page 20: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/20.jpg)
20
Findings: Key Components of Uncertainty
• Uncertainty in population estimates related to:– Size of each component– Error distribution assumptions
• Key components driving uncertainty in LA estimates:– Mid-2001 base population estimate– Internal Migration– International Migration (IPS)– Specific components important in specific LAs
e.g. Foreign Armed Forces
![Page 21: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/21.jpg)
21
Extending the Methodology
• Current assumptions in the estimation of uncertainties are inadequate– Need to examine issues within each component– Consider LA variation in uncertainties within each
component
• Currently focussing on refining error distributions for key drivers of uncertainty within LA estimates– Internal Migration– International Migration
![Page 22: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/22.jpg)
22
Estimating Uncertainty - Internal Migration:Emerging Proposals for Further Work
Building upon previous work, investigate
uncertainty in estimates related to:• Time lags between moving and reregistering• Moves not captured by GP registers because
patients were not registered when data were extracted
• The scale of constraining GP register data to NHSCR
![Page 23: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/23.jpg)
23
Estimating Uncertainty - International Migration:Emerging Proposals for Further Work
• Calculating sampling error of IPS estimates• Investigating uncertainty around migrant and
visitor switcher estimates • Investigate uncertainty within methods used
to calculate LA migration estimates from the IPS– For example, in LA emigration model used
![Page 24: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/24.jpg)
24
Future Outcomes of this Work
• Increased understanding of sources of error in the population estimates and their relative importance
• Ability to focus resources for research on key sources of uncertainties
• Additional information which could feed into Quality Reports
This work is intended to improve our understanding of the uncertainty in population estimates, rather than provide exact estimates of uncertainty
![Page 25: 1 Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates Joanne Clements Ruth Fulton Alison Whitworth](https://reader036.vdocuments.net/reader036/viewer/2022062515/56649f445503460f94c6593d/html5/thumbnails/25.jpg)
25
SummaryMeasuring Uncertainty of Population Estimates
• Estimating their error margin is complex• Detailed quality assessment of each
component required to obtain a robust measurement
• Simulation methods are a plausible approach to approximately measure the overall quality of an estimate
• Ongoing work on estimating uncertainty in migration components