A Strategy for Prioritising Non-response Follow-up to Reduce Costs Without

Reducing Output Quality

Gareth James Methodology Directorate

UK Office for National Statistics

Outline of presentation

• Introduction response-chasing in ONS business surveys

• Understanding non-responseeffects, patterns and reasons

• Strategy for response-chasingscoring methods – current investigations and future strategies

2

Introduction

• Non-response … the failure of a business to respond in part or full to a survey. Effect on:

– bias and standard error, – perception of output quality, – business behaviour

Improve response rates by:– better questionnaire design, sample rotation rates, …– response-chasing - necessary, but expensive

• Quality improvements and efficiency targets– effective targeting needed

3

Current practice at ONS

• Use of % targets (mainly counts, occasionally other variables)

• Written reminders to all. Then targeted phone calls, … could lead to enforcement

• Businesses identified as ‘key’ (by survey area) chased intensively first

• After ‘keys’, principle to chase large-employment businesses next

• Methods differ between surveys

4

Current practice at ONS

• Areas for improvement:– Methods for ‘key’ businesses:

make more consistent, transparent, scientific

– Effective use of response-chasing tools– Team structure and knowledge

(Area undergoing restructure)

• Efficiency initiatives– save resources: some changes already implemented– effects being monitored; evaluation needed

5

Efficiency initiatives – removal of second reminders

Stocks Inquiry (50-99 sizeband)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%0

10

20

30

40

50

60

70

80

90

10

0

Days since despatch

Res

pons

e ra

te

With 2nd reminder Without 2nd reminder

6

UNDERSTANDING NON-RESPONSE

Patterns of non-response

• Industrial sector - identified those with lower response rates (e.g. catering, hotels)

• High correlation between industry response rates at early and final results

• Size of business – larger businesses take longer to respond. Chasing strategy ensures responses are received later though

8

Intensive Follow-Up (IFU) exercise

• Dual aims:– to estimate non-response bias (work in progress – see final

paper)– to establish reasons for non-response and (later) cost response-

chasing

• Used the Monthly Inquiry into the Distribution and Services Sector (MIDSS):

– dedicated team for the IFU– contacted c.600 non-responders per month in chosen industries– businesses to receive up to 5 phone calls– reason for initial non-response; nature of call; length of call

9

IFU results – returned data

• c.80% of all businesses selected for IFU returned questionnaire, but

• many businesses returned questionnaire just after deadline – no call needed!

• Only c.60% of those contacted returned questionnaire

10

IFU results – reasons for non-response

Reason for initial non-response

Number who gave a reason

Returned data after IFU calls

Still didn’t return data

after IFU calls

Forgot, missed date

667 77% 23%

Too busy, too low priority

361 67% 33%

Actively decided not to

67 33% 67%

11

BUILDING A RESPONSE-CHASING STRATEGY

Dealing with businesses that don’t respond

• Aim to make response-chasing more efficient

• Create a scoring system to prioritise/categorise non-responders

• Focus on reducing non-response bias

13

Estimation in ONS business surveys

We impute/construct where there is non-response.

Then estimate totals as

where

*y i i

i S

t w y

* if

ˆ if i R

ii NR

y i Sy

y i S

14

Bias in ONS business surveys

• Total potential non-response bias (= total imputation error) given by

• We will concentrate on

(i.e. the absolute error of imputation for each business)

ˆi i ii S

w y y

ˆi i iw y y

15

Scoring - principles

• Reduce imputation error by attempting to predict

(Large value means increased risk if business is imputed – therefore target these)

• May also wish to score to encourage good response behaviour from businesses – e.g. new-to-sample

• Need a system that is easy to use and justify.

ˆ| |i i iw y y

16

Scoring methods

• (McKenzie) Calculate imputation error from previous returns; then rank into deciles: 0, 1, …, 9.

(Smallest – Largest)

New-to-sample or long-term non-responders = 10

Tested on MIDSS in 2001-2; implementation issues

• (Daoust) Calculate weighted contribution to estimates – categorise into 3 groups for follow-up

• New investigations with adapted methods

17

Current investigations in MIDSS

• Predict imputation error in monthly turnover (= y)

– Various predictors available– Rank businesses then group – No imputation score?

Use stratum average.

• Assess actual error against predicted.

1. imputation error(t-1)

2. register turnover

3. ( ,imp.error(t-1),

imp.error(t-2), ...,

reg. turnover ,

reg. employment , ...)

i

i i

i

i

i

POSSIBLE PREDICTORS

w

w

f w

18

Results (5 groups)

ˆR

i i ii S

w y y

Actual

Score Imputation error

4 88

3 8

2 3

1 1

0 << 1

• Percentage of within each priority score group

19

Results

ˆR

i i ii S

w y y

Actual Weighted

prediction

Score Imputation error

Previous imp. error

4 88 73

3 8 12

2 3 10

1 1 3

0 << 1 2

• Percentage of within each priority score group

19

Results

ˆR

i i ii S

w y y

Actual Weighted

prediction

Score Imputation error

Previous imp. error

Register turnover

4 88 73 68

3 8 12 15

2 3 10 8

1 1 3 5

0 << 1 2 4

• Percentage of within each priority score group

19

Results

ˆR

i i ii S

w y y

Actual Weighted

prediction

Unweighted prediction

Score Imputation error

Previous imp. error

Register turnover

Register employment

Register employment

4 88 73 68 42 40

3 8 12 15 20 15

2 3 10 8 11 12

1 1 3 5 9 18

0 << 1 2 4 18 15

• Percentage of within each priority score group

19

Conclusions

• Significant gains available in response chasing

Future plans:• Refinements to scores:

– optimum predictor– individual adjustments (e.g. long-term non-responders)– overall or by separate industry groups?– multivariate surveys

• Dynamic updating of scores• Live testing

20

References

• Daoust, P., (2006), 'Prioritizing Follow-Up of Non-respondents Using Scores for the Canadian Quarterly Survey of Financial Statistics for Enterprises', Conference of European Statisticians

• McKenzie, R., (2000) 'A Framework for Priority Contact of Non Respondents', Proceedings of the Second International Conference of Establishment Surveys

21