1 - burton understanding society 20160708 - burton... · 2016. 10. 17. · dutch platform for...
TRANSCRIPT
-
An initiative by the Economic and Social Research Council, with scientific leadership by the Institute for Social and Economic Research, University of Essex, and survey delivery by NatCen Social Research and TNS BMRB
Understanding SocietyDr Jonathan BurtonUnderstanding Society Associate Director, Surveys
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Understanding Society
• What is it? • Features of the study
• How is it organised?• Survey design• Modes
• Response• Response rates• Activities to improve response
• Future plans
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Understanding Society: origin story• Launched in 2008• Built on the strong foundation of the British Household Panel
Survey (BHPS)• 1991-2008: 18 years of data• Most used data by academics• More than 3000 publications• BHPS – part of Understanding Society from Wave 2
• 2016 – 25th year of data collection!• Funded by the Economic and Social Research Council (ESRC) and
drawing on co-funding from Government Departments and Devolved Administrations
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
What is Understanding Society?• The UK Household Longitudinal Study (UKHLS)
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
What is Understanding Society?• The UK Household Longitudinal Study (UKHLS)
• UK – covers • England• Scotland• Wales• Northern Ireland
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
What is Understanding Society?• The UK Household Longitudinal Study (UKHLS)
• Collects data on all household members• Seeks to interview all aged 16+• Self-completion questionnaire for 10-15 year olds
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
What is Understanding Society?• The UK Household Longitudinal Study (UKHLS)
• Follows the same original sample of people across time within the UK
• Includes other people living with original sample members after first wave
• Measures of individual-level change
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Why collect longitudinal data?
• Provide a dynamic analysis of change rather than a snapshot
-
Cross-sectional and longitudinal change
57 58
4 3
39 39No change
-1
+1
Change: 2 percentage points
Employed Inactive
2015 2016
-
Cross-sectional and longitudinal change
57 58
4 3
39 39
57
4
39 1 5
6
2
2
39
3
58
No change
-1
+1
Change: 2 percentage points Change: 16 percentage points
Employed Inactive
2015 2016 2015 2016
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Why collect longitudinal data?
• Provide a dynamic analysis of change rather than a snapshot• Better able to disentangle age and cohort effects• Model transitions into and out of particular states• Better able to make causal inferences; A→B rather than
A↔B
-
Key features of Understanding
Society
Panel study of all ages
Annual interviews
Household-focus
Geographic data
A focus on ethnicity
Multi-topicInnovation
Biosocial study
Data linkage
Policy Engagement
Large sample size
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Key features: Large sample size• A large sample size gives an opportunity to explore issues
where other longitudinal surveys are too small. • Permits analysis of small subgroups• Analysis at regional and sub-regional• High-resolution analysis of events in time – single-year
age cohorts
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Key features: All ages
• Collects survey data from people aged 10+• (compared to birth cohorts)• All adults aged 16+ asked to do full interview• Young people aged 10-15; self-completion questionnaire
• Collects information about everyone• Basic information on all household members (age 0+)• Key child development measures at ages 3, 5, 8.
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Key features: Annual interviews• Participants interviewed annually• Collect detailed information on areas of life which are subject to
change (health, education, employment, marital status, fertility)• Regular data collection allows us to capture information on changes
close to the point that they happen – before they get forgotten• … and then see what affect these changes have• Less regular data collection risks some events being forgotten or
mis-remembered, we might miss the impact they have on their everyday life, and we cannot react to changes in society or government in a timely way
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Key features: Whole household• Data collected on all individuals living together in households• Important for research e.g.
• consumption and income, where within-household sharing of resources is important,
• demographic change, where the household itself is often the object of study.
• Can investigate family factors and context in decision making• Observing multiple generations allows examination of long-
term transmission processes • Comparative analysis of sibling outcomes
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Key features: Geography
• Covers all 4 countries of the UK• Large sample size allows for sub-national analyses• Link to information from Local Education Authorities, Local
Health Authorities, parliamentary constituencies, police areas etc
• More secure data access for post-code or grid-reference information information about air pollution, level of deprivation, amount of greenspace etc
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Key features: Ethnicity
• Original “Ethnic Minority boost” (Wave 1) and recent “New Immigrant and ethnic minority boost” (Wave 6)
• Allows sufficient sample size for robust analysis within ethnic groups and across generations
• Survey materials/questionnaire translated into 9 languages• Additional 5 minutes of questionnaire time • Increasing prominence of research into ethnic difference for
understanding the make-up of UK society • Focus on issues of diversity and commonality
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Key features: Multi-topic
• Covers range of social, economic and behavioural factors• Potential for research from a wide range of disciplines• Look at cross-domain impacts: e.g., health on employment
(and vice versa)• “Topic Champions” and user consultations review content to
keep it relevant
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Key features: Innovation
“Innovation Panel” (IP) of around 1500 households• Similar longitudinal design to main survey• Methodological sample, test:
• New questions• New ways to ask old questions• Different fieldwork designs
• Annual competition allows researchers from around the world to propose experiments
“Associated Studies” allow more in-depth, qualitative research
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Types of experiments in IP• Procedural
• Incentives, design of letters, mode and time of contact, tailoring ‘interesting questions’
• Mixed mode – incentives, attrition, measurement issues• General measurement
• Branched v unbranched questions• Layouts of grids• Ways of reducing item missingness (in web surveys)
• Topic-specific measurement• Asking consent• Item Count technique (sexual orientation)• Comparison quick question vs. in-depth module
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
• Mixed method research• K Gush et al, ISER: ‘Understanding the impact of recession on labour
market behaviour in Britain’• A Sacker et al, UCL: ‘Needing help and accessing services in later life’• Mok, LSE (PhD): ‘Making ethnic choices’
• Experimentation• M. Galizzi, LSE: Time and Risk Preferences• A Delavande et al, Essex: between wave web survey of attitudes to risk and
ambiguity
• New forms of data• J. Gershuny, et al Oxford: Paper time use diaries• Discussing ….Smartphone data collection, different data linkage
Associated Studies – to date
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Key features: Biosocial
• Combination of the “social” with the “biological”• 2-way relationship between social and economic
circumstances, and health• Collection of bio-markers and genetics data• Wave 3 – cognitive functions tests• Nurse visits early in the study
• Height, weight, body fat, waist circumference, blood pressure, lung function, grip strength
• Drug coding• Whole blood
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Key features: Linked administrative data• Recently linked to a wide range of education information• Discussions underway for linkage to health, benefits, tax,
vehicle information, energy use…• Allows researchers to link detailed, thorough administrative
data to survey data • Reduces the burden on participants, provides much more
information than we’d be able to collect in an interview• Can fill in the gaps in the survey, but also fill in the gaps in
administrative data – e.g., unmet need? People who need health treatment but don’t get it, won’t appear in administrative data
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Key features: Policy engagement• Establishment of a “Policy Unit”• Directly support government departments to use
Understanding Society data• Work with other organisations (charities, think tanks) to
promote policy learning• Recent announcement of “voucher scheme” for charities
• Invite innovative research ideas from third sector organisations that help further their mission, drive social change and support social innovation.
-
SURVEY DESIGN
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
The Innovation Panel sample to enable
methodological research.
How many households at Wave 1?
A new equal probability sample representative of the UK
A new Ethnic Minority Boost sample: designed to achieve 1000 interviews in
each of five key groups
The British Household Panel Survey sample
(Wave 18)
39,805 in total
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
How many individuals in the sample at Wave 1?
101,087 in total
*Individuals – includes children and non-responding adults
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Structure of the study
Plus “Immigrant and Ethnic Minority Boost” (IEMB) from Wave 6• 2,496 households• 8,626 individuals• 4,460 adult respondents
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Structure of the study
Practically, there are two surveys• “main-stage” – GPS, EMB, IEMB, BHPS• Innovation Panel (IP)
These differ in terms of fieldwork timing and content• Main-stage: continuous survey, same questionnaire (with
routing)• IP: May-September
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Main-stage fieldwork
• Each wave of fieldwork is issued as 24 monthly samples• GPS and EMB are issued across the 24 months• BHPS are issued in the first year of each wave• IEMB are issued in the second year of each wave
• Each monthly sample is in the field for 6 months• So each wave takes almost 2½ years
• A new wave starts each year, so households are issued annually
An over-lapping design
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Over-lapping design2014 2015 2016 2017 2018
Wave 5GPS/EMB Y2
BHPS
IEMB Y2
Wave 6GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
Wave 7GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
Wave 8GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
Wave 9GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Over-lapping design2014 2015 2016 2017 2018
Wave 5GPS/EMB Y2
BHPS
IEMB Y2
Wave 6GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
Wave 7GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
Wave 8GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
Wave 9GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
Individual issued here…
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Over-lapping design2014 2015 2016 2017 2018
Wave 5GPS/EMB Y2
BHPS
IEMB Y2
Wave 6GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
Wave 7GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
Wave 8GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
Wave 9GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
Individual issued here…
…also issued here
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Over-lapping design2014 2015 2016 2017 2018
Wave 5GPS/EMB Y2
BHPS
IEMB Y2
Wave 6GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
Wave 7GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
Wave 8GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
Wave 9GPS/EMB Y1 Y2
BHPS Y1
IEMB Y2
Individual issued here…
…also issued here
…and here
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Time in the field
2014 2015 2016 2017 2018
Wave 5
GPS/EMB
BHPS
IEMB
IP
Wave 6
GPS/EMB
BHPS
IEMB
IP
Wave 7
GPS/EMB
BHPS
IEMB
IP
Wave 8
GPS/EMB
BHPS
IEMB
IP
Wave 9
GPS/EMB
BHPS
IEMB
IP
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Household design
• First task on making contact and gaining co-operation: Enumerate the household• Who is there? Any new people? Any movers?
• Household questionnaire: 10-15 minutes, 1 per household• Adult interviews (age 16+): average 40 minutes each• Youth self-completions (age 10-15)
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Content of the survey
Enumeration grid• Basic demographic of the household members (age 0+)
• Name, sex, date of birth, employment status, relationship status, relationship to others in the household
• Movers/leavers – date of change, reason for changeHousehold questionnaire• Information about accommodation and household-level expenditure
• Housing tenure, rent/mortgage, house value• Spending on food, non-food, utilities (gas, electricity)• Ownership of consumer durables, cars• Difficulties making household payments
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Content of the survey
Individual (adult) interview• Core content – update on changes in the last year:
• Residential mobility• Marital status• Cohabitation status• Fertility history (including information on childbirth and
behaviour during and after pregnancy)• Periods of full-time education• Periods of training• Employment status changes• Health changes
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Building up a picture of change over time
2012 2013 2014 2015Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
Point of interviewPoint of interview
Retrospective data
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Content of the survey
Individual (adult) interview• Rotating content: modules asked every 2, 3, or 4 years, e.g.,
• Wealth, assets and debts (4 yr)• Social/friendship networks (3 yr)• Political and social engagement (3 yr)• Identity measures (3 yr)• Social support (3 yr)• Local neighbourhood (3 yr)• Family relationships and parenting (2 yr)• Family networks (2 yr)• Environmental behaviour (2 yr)• Leisure participation (3 yr) • Financial behaviour and attitudes (2 yr)• Employment conditions (2 yr)• Well-being and sleep quality (2 yr)
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Content of the survey
Individual (adult) interview• Also, age-related
• Retirement planning (45, 50, 55, 60 yr olds)• Child development (when child is 3, 5, 8 yrs)• Educational aspirations (young adults and parents)
• Or time-triggered• General election• Scottish referendum• Olympic Games
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Survey structure – possible paths
Household grid
Household questionnaire
Individual questionnaire 1
Individual questionnaire 2
Individual questionnaire 1
Household questionnaire
Individual questionnaire 2
Household questionnaire
Individual questionnaire 2
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
How is the survey conducted?
Wave 1 and 2• Face-to-face onlyWave 3-6• Face-to-face• Telephone mop-up for non-responders
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
How is the survey conducted?
Wave 7• Face-to-face for Wave 6 responding households• Web for Wave 6 non-responding households
• Face-to-face follow-up for non-responders (“Web-first”)• Telephone mop-up
Wave 8• Face-to-face for 60% of Wave 7 responding households
• Non-respondents invited to participate on-line (“CAPI-first”)• Web-first for 40% of Wave 7 responding households + Wave 7 non-
responding households• Telephone mop-up
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
The rise of mixed-modeProportion of the sample issued to each mode
0%
20%
40%
60%
80%
100%
Wave 6 Wave 7 Wave 8 Wave 9 Wave 10 Wave 11
CAPI-first Web-first
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Why mixed-modes?
• Requirement from funders to reduce costs to make the study sustainable
• Innovate• Try to increase response by allowing people to take part in a
way which they prefer
• Sequential mixed-mode design• Non-respondents to initial mode re-issued to follow-up
mode
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Mixed-mode design requirements• Reduction of costs
• Whole-households complete in cheaper mode• The effort required by interviewers is reduced for partial-
completion households• Increase in response
• The cheaper mode brings in people who would not have been interviewed otherwise
• Having invited people to the cheaper mode, non-response doesn’t affect their propensity to respond in the more expensive mode
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Our design can lead to some complications• Household
• Want all adults to participate• …even adults we don’t know about yet
• Longitudinal• Need specific adults to participate• …even if they have moved house
Requirement to: Enumerate the household first Deal with movers in and out of the household Follow-up non-respondents in another mode
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Experience of mixed modes?
• Use of the Innovation Panel• IP2 – experiment with CATI-CAPI
• Lower response than CAPI alone• Lower response among those who require more effort• Lower response ‘scarred’ the sample for the next two
CAPI-only waves
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
CATI reduced response and the effect lasted to the next wave
Response Rate
Wave 2 Wave 3(All F2F)
Wave 4(All F2F)
Face-to-face 75.8% 66.3% 57.3%
CATI-CAPI 67.4% 59.9% 53.7%
Difference -8.4 -6.4 -3.6
Odds Ratio (P) 0.67 (0.001) 0.79 (0.07) 0.88 (0.29)
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
IP5 onwards – use of web
2/3 sample –allocated to WEB
1/3 sample allocated to F2F
Non-responders allocated to F2F
2 week “web-only” period. Advance letter + email
plus 1 letter and 2 email reminders
Non-responders allocated CATI
mop-up
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
IP5 Household Response Rates
F2F MM P
Household response rate 84.1 81.1 0.29
Complete Households 63.9 55.7 0.02
Partial Households 20.3 25.4 0.07
Non‐contact 5.6 2.9 0.04
Refusal 8.7 14.1 0.05
N 322 621
* IP4 responding households
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
IP6 Household Response Rates
£10 incentive IP5 responding
F2F MM P
HH response rate 85.2 75.8 0.03
Complete HHs 65.4 64.2 0.74
Partial HHs 19.8 11.6 0.004
Non‐contact 7.2 11.9 0.03
Refusal 7.6 12.3 0.04
N 445 268
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Effects of Incentives: IP6
Incentive £10 unc £10 unc+ £20 con
£30 unc P
HH response rate 75.7 84.8 86.2 0.03
Complete HHs 64.2 71.1 74.6 0.03
Partial HHs 11.6 13.7 11.6 0.68
Refusal 12.3 7.6 6.0 0.03
Other NR 11.9 7.6 7.8 0.14
N 268 277 268
* Mixed mode sample, IP5 respondents
F2F £10 household response: 85.2%Require £30 incentive to get similar response using mixed-modes
Annette Jäckle, Peter Lynn, Jonathan Burton: “Going online with a face-to-face household panel: initial results from an experiment on the Understanding Society Innovation Panel”, Understanding Society Working Paper Series: 2013-03
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Mixed Modes over Multiple Waves
Percentage of completed interviews completed by web:
.Source: Professor Peter Lynn’s own calculations
Household interview %
Individual interview %
Wave 5 31.3 30.0
Wave 6 36.8 38.3
Wave 7 42.8 43.6
Wave 8 44.8 45.0
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Attrition: Web + CAPI
• Achieved higher response rates for t-1 non-respondents than CAPI only Implemented in main survey from Wave 7 (2015)
• Household attrition rates compared to CAPI only 3-6% higher with standard £10 unconditional
incentive Comparable/lower with higher unconditional or
conditional incentives• Proportion of Web completers is increasing
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Measurement Issues• NatCen Social Research commissioned to code all questions
proposed for waves 7 – 10 with respect to 13 characteristics associated with increased risk of measurement differences between modes
• Summary measure of ‘risk of mode effect’ derived for all 1,486 questions
• Many of the more sensitive questions (health, attitudes, satisfaction) are already asked by self-completion within CAPI; mode effects likely to be negligible for these
• Other questions most at risk of mode effects are:• Questions about harassment, prejudice etc, asked as part of the
ethnic minority research strand;• Questions that benefit from interviewer encouragement/ support to
answer fully and accurately, including income.
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Measurement / data comparability: Web + CAPI• No concerns about Break-offs Missing or duplicate instruments HH grid identification of leavers/joiners
• Web respondents are not representative More affluent
• Web + CAPI follow-up sample is similar to CAPI only Web-only not an option!
• Accounting for selection into mode Few differences in response distributions
Forthcoming research from Annette Jäckle
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Web Conclusions
• Initial response rate lower when switching to web-first, but recovers after a couple of waves
• Previous wave non-respondents are more likely to respond with web-first design than with CAPI-only
• Cost savings should soon accrue as around 25% of responding households respond fully by web
• Percent responding by web appears to increase over waves• May be possible to avoid potential increases in item response
rates• Measurement error issues remain a concern; our focus is to
minimise impact and provide analysis guidance
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Adaptive design on the main-stage (Wave 8)• More of a risk?• Try to restrict to adaptations that have at worse “no effect”• Either cost-saving or cost-neutral• Experimental allocation from Quarter 2• Allocation to mode non-random
• All previous-wave non-responding households: Web-first• CAPI-first:
• Ring-fenced sample (20% households)• Lowest propensity to complete online (20%)
• Of the remaining 60%• 20% CAPI-first• 40% Web-first
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Month 1 – standard £10 unconditional incentive
0%
5%
10%
15%
20%
25%
30%
35%
40%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
January
18.6% completed households
Days in the field
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Month 2 – addition of £10 conditional incentive if complete in first 2 weeks
0%
5%
10%
15%
20%
25%
30%
35%
40%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
JanuaryFebruary
37.4% completed householdsDeadline for
bonus
Days in the field
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Month 3 – incentives same as month 2, half sample get additional reminder letter
0%
5%
10%
15%
20%
25%
30%
35%
40%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
JanuaryFebruaryMarch
37.5% completed households
Days in the field
-
Month 3: Additional reminder letter increased whole-household web completion
Invite + 1 week + 1 week
Control Letter + Email Email Email
Treatment Letter + Email Letter + Email Email
16% 17%
33%
44%
0%
10%
20%
30%
40%
50%
Control Treatment
Before 1streminder
End of web-only
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Months 4-6 – ‘trigger’ experiments
• Experiment with the ‘trigger’ for the bonus• Adult completion• Whole-household completion
34,1% 31,7%
22,7%19,9%
0%
10%
20%
30%
40%
50%
60%
Individual trigger Household trigger
Partial household
Whole-household
-
Months 1-9: Different ‘web-only’ periods
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35
% fully complete web
January
February
March
April
May
June
July ‐ 5 week
July ‐ 3 week
Aug (5 week)
Sept ‐ 5 week
Days in the field
-
SURVEY RESPONSE
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Response rates
• Response rates an important indicator to the quality of the study for users, funders
• Response also affects sample size• Particularly important in a longitudinal study
• Measures over time required• Missing a wave, or drop-out, affects longitudinal sample
for analysis
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Fieldwork – response rates
57,6% 56,0% 55,8% 55,5% 54,8%
0%
20%
40%
60%
80%
100%
UnderstandingSociety GPS
CitizenshipSurvey
ESS 2008 Labour ForceSurvey (W1)
Wealth &Assets (W1)
Wave 1 – below the 60% target, but in line with other surveys at the same time
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Response over timeHousehold response rate for previous-wave
responding households
60%
65%
70%
75%
80%
85%
90%
95%
100%
Wave 2 Wave 3 Wave 4 Wave 5 Wave 6 Wave 7* Wave 8*
GPSEMBBHPS
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Response over timeHousehold response rate for previous-wave
responding households
60%
65%
70%
75%
80%
85%
90%
95%
100%
Wave 2 Wave 3 Wave 4 Wave 5 Wave 6 Wave 7* Wave 8*
GPSEMBBHPS
?
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Wave 6 fall in response?
• At Wave 6, the fieldwork agency changed• Almost all sample members were contacted by a different
interviewer at Wave 6• Figures for Wave 7 (and 8) look as if the upward trend has
recovered
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Activities to increase response
• Survey design under constant review to try and improve response
• …or maintain response at a lower cost• Advantage of having the Innovation Panel
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Response: Fieldwork design
• Use of different modes to allow sample members to participate in a mode that is convenient to them
• Email reminders for break-offs• Minimum number of call attempts before making contact (inc.
evenings/weekends, over at least 3 weeks)• Long fieldwork period to allow interviewer flexibility• Translate participant materials/questionnaire• From Wave 8, allow interviewers to telephone first to make an
appointment• Fieldwork monitoring reports available daily• Non-respondents re-issued to another interviewer• Telephone mop-up at end of fieldwork period
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Response: Incentives
• Unconditional £10 gift voucher for all issued adults, in advance letter• From Wave 6: £20 unconditional for adults in previous-wave
refusal households, £20 conditional for adults in previous-wave non-contact households
• From Wave 8: Previous-wave adult non-responders offered conditional £10 incentive
• From Wave 8: Additional £10 bonus for early-completion online• Unconditional £5 gift voucher for young people (10-15 years)• £5 voucher for returning a change-of-address (COA) card• Entry into quarterly prize draw to win iPad for 16-year olds
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Response: Tracing/panel maintenance• Maintain Freepost, Freephone, email address, web-form to enable
sample members to contact us• Provide sample members with COA cards• £5 incentive for returning COA cards• Collect multiple contact details (home phone, work phone, mobiles,
email addresses)• Collect stable contact information• Use of tracing letters• Weekly tracing file sent to ISER
• Email untraced movers• Use of online databases• Use of out-of-hours telephone tracing staff
• Additional mailing of ‘likely movers’
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Response: Participant Communications
• Advance letters/emails tailored to different sample status• Introduction leaflet for new entrants• Between-wave mailing with findings from study (with COA)• “likely mover” mailing• Participant website, regularly updated• Use of social media for dissemination of news/research• Regular (3 x year) Participant Update emails• Birthday cards for those turning 16• Case studies for use on doorstep
-
UNDERSTANDING SOCIETY:THE FUTURE…?
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Currently…
• Scripting/testing Wave 9 pilot (October-November)• Scripting IP10• Developing Wave 10 content• Preparing Wave 6 data for release
• Starting a new fieldwork contract for Waves 9-11• Working with TNS BMRB and NatCen Social Research
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Coming up…
• Further increase of Web-first sample• Further experimentation on Wave 8 adaptive design
• With successful protocol adopted for Wave 9• Review of content across key topic areas (Topic Champions)• Increasing focus on scientific and policy impact• Feasibility of between-wave online surveys
• Sub-groups of sample• Event-triggered (e.g., elections)
• Feasibility of using new data collection technologies
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Survey
Store card data
Clickstream data
Barcode scanners, till receipts
Wearables
Financial aggregators
Social media
Smartphone apps
Post
New data collection technologies
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Example: Collect household budget
• Improve Data Quality by imposing accounting identities
Income – Expenditure = Change in Assets and Debts
Resources available
Net disposable
income
Borrowing / dissaving
Repayment of loans/ savings
Expenditure
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
• Improve measurement of net monthly income:1. Additional questions2. Income summary screen
• Benefit Unit budget reconciliation:3. New module for couples
• Data linkage: Financial Conduct Authority • Pilot new technologies
4. App: scan till receipts
• Between wave web survey (development)5. Web survey: gamified tool to capture income & expenditure6. Benefit Unit budget reconciliation 7. (Possible) Preload data collected with new technologies
IP: New technologies, data linkage
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Accessing the data and findingsStudy documentation:• Protocols, fieldwork reports, questionnaires, variable level metadata• User guides, Quality Profile, technical reports• Working Papers and publications from methodological researchSee www.understandingsociety.ac.uk
Data are available at the UK Data Service, University of Essex:• Wave 1 - 5 data, including biomarkers; wave 6 and immigrant &
ethnic minority boost sample to be deposited November 2016• Innovation Panel data waves 1-8• Linked education data, geographical identifiers available by special
licence or via secure labSee UK Data Service website: http://ukdataservice.ac.uk/
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Thank you
Jonathan BurtonISER, University of Essex
[email protected]@jburton123
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Data – structure
• Understanding Society is a complex data-set• Hierarchical
• All sample households, all sample individuals• Responding households• All adults• Responding adults• Loops with adults: employment spells, information about
children• Plus, paradata – call records, timings, interviewer
characteristics• 15 data-sets per wave
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Data – content
Questionnaire data, plus:• Derived variables
• Pointers to parents/spouse/children, Socio-economic and occupational classifications of jobs, Summary scores such as BMI, GHQ, SF-12 and SDQ, household typologies, household incomes, net and gross incomes … and lots more
• Weights• Design weights, individual and household response weights,
cross-sectional and longitudinal • Imputed values
• to replace missing values on individual components of earned and benefit incomes that contribute to total income
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Data – help
• “Getting started” guide• User guides• On-line data documentation• Regular training courses• Support forum
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
Main-stage
0
200
400
600
800
1000
1200
1400
1600
2010 2011 2012 2013 2014 2015
UKDS Downloads
Total
Linear (Total)
0
100
200
300
400
500
600
700
800
900
2010 2011 2012 2013 2014 2015
New Unique UKDS Users
Total
Linear (Total)
-
IP
0
50
100
150
200
250
2011 2012 2013 2014 2015
UKDS Downloads
Total
Linear (Total)
020406080
100120140160180200
2011 2012 2013 2014 2015
New Unique UKDS Users
Total
Linear (Total)
-
Special License
0
50
100
150
200
250
2011 2012 2013 2014 2015
UKDS Downloads
Total
Linear (Total)
0
10
20
30
40
50
60
70
2011 2012 2013 2014 2015
New Unique UKDS Users
Total
Linear (Total)
-
HE users
Government users
Charities
Think tanks
Membership orgs
-
Dutch Platform for Survey Research8th September 2016. Amsterdam
UKDS UK HE Users by discipline
0
100
200
300
400
500
600
2010 2011 2012 2013 2014 2015
EconomicsSociologyHealth SciencesGeographyStatsBusinessPoliticsEducation