analysis of the characteristics of internet respondents to the 2011 census to inform 2021 census...
TRANSCRIPT
![Page 1: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/1.jpg)
Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design
Orlaith Fraser & Cal Ghee
![Page 2: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/2.jpg)
Overview
1. The census and modes of response
2. Census Quality Survey
3. Propensity score analysis
I. Census quality survey results
II. 2011 Census results
4. Conclusions
![Page 3: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/3.jpg)
The census and modes of response
Mode effect = Same respondent gives different response when using different modes
X
Y
2011 Census: Paper by default, Internet option- 81% responded by paper, 19% by internet
2021 Census: Internet by default, possible paper option.
![Page 4: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/4.jpg)
Census Quality Survey (CQS)
What is your date of birth?
01 01 1977
Face-to-face CAPI sample survey.Sample of census respondents asked majority of census questions again.
Answers compared to calculate agreement rates.
CQS answers assumed correct as face-to-face likely to be more accurate than self-completion.
![Page 5: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/5.jpg)
Census Quality Survey
• Stratified Sample region, hard to count, mode
• No adult proxy responses• Individuals weighted
age, sex, ethnic group, mode
• Household representative interviewed
5170 matched
households
9,650matched usual
residents5170
matched households
Paper/CQS agreement rate
Internet/CQS agreement rate
Co
mp
are
agre
emen
t ra
tes
![Page 6: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/6.jpg)
Internet agreement rates significantly higher than paper
![Page 7: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/7.jpg)
Paper agreement rates higher than internet
![Page 8: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/8.jpg)
Possible reasons for differences
Age Scanning errors?
Marital status Social desirability bias?
Disability Recall error?
Group most likely to change answer
Religion
Possible reasons for other differences:• Use of radio buttons• Help information• Scrolling distance• Paper format easier to look ahead
![Page 9: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/9.jpg)
Propensity score method
Direct comparison between internet and paper responders difficult because of differences in
respondent characteristics
Distribution of internet responders matched to that of paper responders
![Page 10: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/10.jpg)
Proportion of internet responders
Propensity Score Method
• Propensity score = Propensity towards exposure to a treatment (responding by internet) given a set of observed characteristics
Proportion of paper respondersAdjustment factor for each subgroup =
1. Individual propensity scores derived from logistic regression model
2. Respondents split into ten subgroups based on propensity scores
3. Internet distribution standardised by applying adjustment factors
STEPS
![Page 11: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/11.jpg)
Propensity Score Analysis of CQS data
VARIABLECQS/Census Agreement
Paper %
Internet %
Internet - Paper
Unadjusted Adjusted
Unpaid Care Agree 82.50 80.46 2.04 -0.26
Disability Agree 85.68 91.99 -6.31 -1.72
Workplace address Agree 43.97 49.31 -5.34 -5.91
Address one year ago Agree 25.51 32.17 -6.66 -3.39
Variables included in logistic regression model:Sex, student status, disability, English as a main language (English or Welsh in Wales), good health and whether working
![Page 12: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/12.jpg)
Propensity Score Analysis of CQS data
VARIABLE
Internet - Paper
Unadjusted Adjusted
Unpaid Care -2.04 0.26
Disability 6.31 1.72
Workplace address 5.34 5.91
Address one year ago 6.66 3.39
Variables included in logistic regression model:Sex, student status, disability, English as a main language (English or Welsh in Wales), good health and whether working
Easier to check postcodes online
![Page 13: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/13.jpg)
Limitations
Household responses may not be independentA highly educated young person may respond online for an older
less well educated person
Proxy effectProxy may not have responded in the same way as the
individual they were representing
Chicken and egg dilemmaAny mode effects included in the model may be considered
as actual predictors
Small CQS sample
Can’t restrict sample to one response per household
![Page 14: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/14.jpg)
Propensity Score Analysis of Census data
More robust analysis using 10% microdata household sample of census data
- stratified by output area
Only household reference persons included Mainly household level variables included as model
predictors Direct comparison of paper/internet proportions for
variable categories rather than CQS/Paper and CQS/Internet agreement rates
![Page 15: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/15.jpg)
Propensity Score Analysis of Census data
Variables included in new logistic regression model:
Age of household reference personCountry of birthDeprivation indicators of a householdEthnic groupHousehold languageHousehold reference person social gradeLiving arrangementsNumber of cars and vans in householdRegionSize of householdTenureUrban Rural classification
![Page 16: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/16.jpg)
Results
Variable Category Paper % Internet %Internet - Paper
Unadjusted AdjustedPensionable age indicator Of pensionable age (65+) 29.32 8.95 -20.37 -1.11Sex Male 58.73 66.34 7.61 2.09Disability Day-to-day not limited at all 73.58 86.06 12.48 -1.99Health Very good health 32.19 43.96 11.77 0.86Hours Works 30-49 hrs/wk 63.52 67.32 3.80 1.30Activity Last Week Not working 39.48 19.24 -20.23 -1.08
Main LanguageEnglish as main language (or Welsh in Wales) 93.83 89.60 -4.23 -0.38
Marital status Widowed 12.66 4.09 -8.57 1.32Marital Status Single 24.83 30.02 5.19 -1.19Level of highest qualifications
Level 2: 5 GCSEs (A* - C) or 1 A level or equivalent 12.42 39.78 27.37 -0.57
All internet/paper differences negligible after adjustment – all explained by differences in population characteristics
![Page 17: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/17.jpg)
Conclusions
• Propensity score analysis can be a useful tool for identifying true mode effects
• Most differences in internet / paper responses attributable to characteristics of respondent group
• Useful tool for testing mode effects prior to 2021 census New mode effects: tablet/mobile/desktop... More detailed knowledge of internet/paper
respondent profiles will help to target support for digitally excluded / reluctant internet respondents
![Page 18: Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design Orlaith Fraser & Cal Ghee](https://reader034.vdocuments.net/reader034/viewer/2022051417/5697c0111a28abf838ccb7ad/html5/thumbnails/18.jpg)
Further information
Please contact:
2021 Statistical Design,Census Transformation Programme,Office for National Statistics