1

Understanding and Measuring Uncertainty Associated with the Mid-Year Population Estimates

Joanne Clements

Ruth Fulton

Alison Whitworth

2

Context

• Improving Migration and Population Statistics (IMPS) Project

• Quality Strand

• “Establish quality measures for population statistics”

• No international precedent for this work

3

Issues

• Estimates compiled from a wide range of administrative sources plus some survey and Census data

• Source data subject to sampling and non-sampling errors

• Lack of independent data with which to corroborate

• How to estimate each potential error and combine these in one measure?

4

Aim and Objectives

• AimImprove understanding, measurement and reporting

of the quality of population estimates

• Objectives– Describing the sources of uncertainty– Developing methods for measuring uncertainty for

each issue and combining them into one measure– Eventually feeding findings into ONS quality

reports

5

Presentation Outline

• Summarise population estimates methodology

• Summarise previous research on quality• Detail proposed error measurement

methodology– Illustrate by applying to local authority (LA) mid-

2006 population estimates

• Outline emerging proposals for further work to achieve robust quality measures

6

Calculating LA Population Estimates

e.g. Southampton UA

7

Calculating LA Population Estimates (cont)

Adjustment

8

Calculating LA Population Estimates

9

Previous Research

Quality of Population Estimates• Past experience of inter-censal errors • Sampling error and expert opinion of non-

sampling error in components of estimates

Quality of Population Projections• Accuracy of past projections• Use of variant projections• Simulation methods using error distributions

for the components of projections (stochastic forecasting)

10

Proposed Methodology: Initial Assessment of Quality Issues

• Map out the procedures and data sources used to derive population estimates

• Identify associated quality issues

• Identify the importance of these issues

11

LA Population Estimates:Initial Assessment of Quality Issues

• Brief assessment of the evidence for each component

• For example: Internal Migration– Relies on GP registration data– Assumes patients reregister within a month of

moving (known issue for students leaving university)

12

Proposed Methodology (cont):Detailed Investigation of Quality Issues

• Quantify uncertainty using statistical theory, empirical evidence and / or expert opinion

• Both sampling and non-sampling errors

13

LA Population Estimates:Detailed Investigation of Quality Issues

• Attributing a potential uncertainty range and distribution to each component

• Not each quality issue

• Made relatively simplistic distribution assumptions (Normal or Uniform)

• Assumed same level of uncertainty across LAs

14

LA Population Estimates:Detailed Investigation of Quality Issues

• Estimating uncertainty relative to size of local authority component

• For example: Could assume potential error in annual local authority births estimate

N(0, X% of estimated births)

– Assume similar error distributions by year

15

Proposed Methodology (cont):Overall Quality Measure

• Mathematically complicated to combine a large number of potential error measures into one quality measure– Errors may be correlated– Distributions not all normally distributed

• Developed a Simulation methodology

16

LA Population Estimates: Simulation

• For each local authority randomly generate errors for each component – Using previously developed error distributions

Mid-2001 error estimate+ Births 01/02 error estimate - Deaths 01/02 error estimate+ Internal In-Migrants 01/02 error estimate - Internal Out-Migrants 01/02 error estimate+ …..+ Births 02/03 error estimate - Deaths 02/03 error estimate+ …

17

LA Population Estimates: Simulation

• Calculate error in mid-2006 estimate by combining the errors generated for each component in each year up to 2006

• Repeat process 1000 times

• Obtain distribution of potential error in mid-2006 local authority estimate

18

Findings: Potential Error Distribution

19

Findings: Measuring Uncertainty in Population Estimates

• Simulation methodology allows measures of uncertainty to be calculated for population estimates.

• But, in reality, there is uncertainty in these measures of uncertainty, as…– Only as good as the error assumptions made for

each issue / component of change– Very difficult to exactly measure non-sampling

error

20

Findings: Key Components of Uncertainty

• Uncertainty in population estimates related to:– Size of each component– Error distribution assumptions

• Key components driving uncertainty in LA estimates:– Mid-2001 base population estimate– Internal Migration– International Migration (IPS)– Specific components important in specific LAs

e.g. Foreign Armed Forces

21

Extending the Methodology

• Current assumptions in the estimation of uncertainties are inadequate– Need to examine issues within each component– Consider LA variation in uncertainties within each

component

• Currently focussing on refining error distributions for key drivers of uncertainty within LA estimates– Internal Migration– International Migration

22

Estimating Uncertainty - Internal Migration:Emerging Proposals for Further Work

Building upon previous work, investigate

uncertainty in estimates related to:• Time lags between moving and reregistering• Moves not captured by GP registers because

patients were not registered when data were extracted

• The scale of constraining GP register data to NHSCR

23

Estimating Uncertainty - International Migration:Emerging Proposals for Further Work

• Calculating sampling error of IPS estimates• Investigating uncertainty around migrant and

visitor switcher estimates • Investigate uncertainty within methods used

to calculate LA migration estimates from the IPS– For example, in LA emigration model used

24

Future Outcomes of this Work

• Increased understanding of sources of error in the population estimates and their relative importance

• Ability to focus resources for research on key sources of uncertainties

• Additional information which could feed into Quality Reports

This work is intended to improve our understanding of the uncertainty in population estimates, rather than provide exact estimates of uncertainty

25

SummaryMeasuring Uncertainty of Population Estimates

• Estimating their error margin is complex• Detailed quality assessment of each

component required to obtain a robust measurement

• Simulation methods are a plausible approach to approximately measure the overall quality of an estimate

• Ongoing work on estimating uncertainty in migration components