1 - burton understanding society 20160708 - burton... · 2016. 10. 17. · dutch platform for...

95
An initiative by the Economic and Social Research Council, with scientific leadership by the Institute for Social and Economic Research, University of Essex, and survey delivery by NatCen Social Research and TNS BMRB Understanding Society Dr Jonathan Burton Understanding Society Associate Director, Surveys

Upload: others

Post on 03-Feb-2021

1 views

Category:

Documents


0 download

TRANSCRIPT

  • An initiative by the Economic and Social Research Council, with scientific leadership by the Institute for Social and Economic Research, University of Essex, and survey delivery by NatCen Social Research and TNS BMRB

    Understanding SocietyDr Jonathan BurtonUnderstanding Society Associate Director, Surveys

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Understanding Society

    • What is it? • Features of the study

    • How is it organised?• Survey design• Modes

    • Response• Response rates• Activities to improve response

    • Future plans

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Understanding Society: origin story• Launched in 2008• Built on the strong foundation of the British Household Panel

    Survey (BHPS)• 1991-2008: 18 years of data• Most used data by academics• More than 3000 publications• BHPS – part of Understanding Society from Wave 2

    • 2016 – 25th year of data collection!• Funded by the Economic and Social Research Council (ESRC) and

    drawing on co-funding from Government Departments and Devolved Administrations

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    What is Understanding Society?• The UK Household Longitudinal Study (UKHLS)

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    What is Understanding Society?• The UK Household Longitudinal Study (UKHLS)

    • UK – covers • England• Scotland• Wales• Northern Ireland

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    What is Understanding Society?• The UK Household Longitudinal Study (UKHLS)

    • Collects data on all household members• Seeks to interview all aged 16+• Self-completion questionnaire for 10-15 year olds

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    What is Understanding Society?• The UK Household Longitudinal Study (UKHLS)

    • Follows the same original sample of people across time within the UK

    • Includes other people living with original sample members after first wave

    • Measures of individual-level change

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Why collect longitudinal data?

    • Provide a dynamic analysis of change rather than a snapshot

  • Cross-sectional and longitudinal change

    57 58

    4 3

    39 39No change

    -1

    +1

    Change: 2 percentage points

    Employed Inactive

    2015 2016

  • Cross-sectional and longitudinal change

    57 58

    4 3

    39 39

    57

    4

    39 1 5

    6

    2

    2

    39

    3

    58

    No change

    -1

    +1

    Change: 2 percentage points Change: 16 percentage points

    Employed Inactive

    2015 2016 2015 2016

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Why collect longitudinal data?

    • Provide a dynamic analysis of change rather than a snapshot• Better able to disentangle age and cohort effects• Model transitions into and out of particular states• Better able to make causal inferences; A→B rather than

    A↔B

  • Key features of Understanding

    Society

    Panel study of all ages

    Annual interviews

    Household-focus

    Geographic data

    A focus on ethnicity

    Multi-topicInnovation

    Biosocial study

    Data linkage

    Policy Engagement

    Large sample size

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Key features: Large sample size• A large sample size gives an opportunity to explore issues

    where other longitudinal surveys are too small. • Permits analysis of small subgroups• Analysis at regional and sub-regional• High-resolution analysis of events in time – single-year

    age cohorts

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Key features: All ages

    • Collects survey data from people aged 10+• (compared to birth cohorts)• All adults aged 16+ asked to do full interview• Young people aged 10-15; self-completion questionnaire

    • Collects information about everyone• Basic information on all household members (age 0+)• Key child development measures at ages 3, 5, 8.

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Key features: Annual interviews• Participants interviewed annually• Collect detailed information on areas of life which are subject to

    change (health, education, employment, marital status, fertility)• Regular data collection allows us to capture information on changes

    close to the point that they happen – before they get forgotten• … and then see what affect these changes have• Less regular data collection risks some events being forgotten or

    mis-remembered, we might miss the impact they have on their everyday life, and we cannot react to changes in society or government in a timely way

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Key features: Whole household• Data collected on all individuals living together in households• Important for research e.g.

    • consumption and income, where within-household sharing of resources is important,

    • demographic change, where the household itself is often the object of study.

    • Can investigate family factors and context in decision making• Observing multiple generations allows examination of long-

    term transmission processes • Comparative analysis of sibling outcomes

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Key features: Geography

    • Covers all 4 countries of the UK• Large sample size allows for sub-national analyses• Link to information from Local Education Authorities, Local

    Health Authorities, parliamentary constituencies, police areas etc

    • More secure data access for post-code or grid-reference information information about air pollution, level of deprivation, amount of greenspace etc

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Key features: Ethnicity

    • Original “Ethnic Minority boost” (Wave 1) and recent “New Immigrant and ethnic minority boost” (Wave 6)

    • Allows sufficient sample size for robust analysis within ethnic groups and across generations

    • Survey materials/questionnaire translated into 9 languages• Additional 5 minutes of questionnaire time • Increasing prominence of research into ethnic difference for

    understanding the make-up of UK society • Focus on issues of diversity and commonality

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Key features: Multi-topic

    • Covers range of social, economic and behavioural factors• Potential for research from a wide range of disciplines• Look at cross-domain impacts: e.g., health on employment

    (and vice versa)• “Topic Champions” and user consultations review content to

    keep it relevant

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Key features: Innovation

    “Innovation Panel” (IP) of around 1500 households• Similar longitudinal design to main survey• Methodological sample, test:

    • New questions• New ways to ask old questions• Different fieldwork designs

    • Annual competition allows researchers from around the world to propose experiments

    “Associated Studies” allow more in-depth, qualitative research

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Types of experiments in IP• Procedural

    • Incentives, design of letters, mode and time of contact, tailoring ‘interesting questions’

    • Mixed mode – incentives, attrition, measurement issues• General measurement

    • Branched v unbranched questions• Layouts of grids• Ways of reducing item missingness (in web surveys)

    • Topic-specific measurement• Asking consent• Item Count technique (sexual orientation)• Comparison quick question vs. in-depth module

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    • Mixed method research• K Gush et al, ISER: ‘Understanding the impact of recession on labour

    market behaviour in Britain’• A Sacker et al, UCL: ‘Needing help and accessing services in later life’• Mok, LSE (PhD): ‘Making ethnic choices’

    • Experimentation• M. Galizzi, LSE: Time and Risk Preferences• A Delavande et al, Essex: between wave web survey of attitudes to risk and

    ambiguity

    • New forms of data• J. Gershuny, et al Oxford: Paper time use diaries• Discussing ….Smartphone data collection, different data linkage

    Associated Studies – to date

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Key features: Biosocial

    • Combination of the “social” with the “biological”• 2-way relationship between social and economic

    circumstances, and health• Collection of bio-markers and genetics data• Wave 3 – cognitive functions tests• Nurse visits early in the study

    • Height, weight, body fat, waist circumference, blood pressure, lung function, grip strength

    • Drug coding• Whole blood

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Key features: Linked administrative data• Recently linked to a wide range of education information• Discussions underway for linkage to health, benefits, tax,

    vehicle information, energy use…• Allows researchers to link detailed, thorough administrative

    data to survey data • Reduces the burden on participants, provides much more

    information than we’d be able to collect in an interview• Can fill in the gaps in the survey, but also fill in the gaps in

    administrative data – e.g., unmet need? People who need health treatment but don’t get it, won’t appear in administrative data

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Key features: Policy engagement• Establishment of a “Policy Unit”• Directly support government departments to use

    Understanding Society data• Work with other organisations (charities, think tanks) to

    promote policy learning• Recent announcement of “voucher scheme” for charities

    • Invite innovative research ideas from third sector organisations that help further their mission, drive social change and support social innovation.

  • SURVEY DESIGN

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    The Innovation Panel sample to enable

    methodological research.

    How many households at Wave 1?

    A new equal probability sample representative of the UK

    A new Ethnic Minority Boost sample: designed to achieve 1000 interviews in

    each of five key groups

    The British Household Panel Survey sample

    (Wave 18)

    39,805 in total

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    How many individuals in the sample at Wave 1?

    101,087 in total

    *Individuals – includes children and non-responding adults

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Structure of the study

    Plus “Immigrant and Ethnic Minority Boost” (IEMB) from Wave 6• 2,496 households• 8,626 individuals• 4,460 adult respondents

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Structure of the study

    Practically, there are two surveys• “main-stage” – GPS, EMB, IEMB, BHPS• Innovation Panel (IP)

    These differ in terms of fieldwork timing and content• Main-stage: continuous survey, same questionnaire (with

    routing)• IP: May-September

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Main-stage fieldwork

    • Each wave of fieldwork is issued as 24 monthly samples• GPS and EMB are issued across the 24 months• BHPS are issued in the first year of each wave• IEMB are issued in the second year of each wave

    • Each monthly sample is in the field for 6 months• So each wave takes almost 2½ years

    • A new wave starts each year, so households are issued annually

    An over-lapping design

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Over-lapping design2014 2015 2016 2017 2018

    Wave 5GPS/EMB Y2

    BHPS

    IEMB Y2

    Wave 6GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

    Wave 7GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

    Wave 8GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

    Wave 9GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Over-lapping design2014 2015 2016 2017 2018

    Wave 5GPS/EMB Y2

    BHPS

    IEMB Y2

    Wave 6GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

    Wave 7GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

    Wave 8GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

    Wave 9GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

    Individual issued here…

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Over-lapping design2014 2015 2016 2017 2018

    Wave 5GPS/EMB Y2

    BHPS

    IEMB Y2

    Wave 6GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

    Wave 7GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

    Wave 8GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

    Wave 9GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

    Individual issued here…

    …also issued here

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Over-lapping design2014 2015 2016 2017 2018

    Wave 5GPS/EMB Y2

    BHPS

    IEMB Y2

    Wave 6GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

    Wave 7GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

    Wave 8GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

    Wave 9GPS/EMB Y1 Y2

    BHPS Y1

    IEMB Y2

    Individual issued here…

    …also issued here

    …and here

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Time in the field

    2014 2015 2016 2017 2018

    Wave 5

    GPS/EMB

    BHPS

    IEMB

    IP

    Wave 6

    GPS/EMB

    BHPS

    IEMB

    IP

    Wave 7

    GPS/EMB

    BHPS

    IEMB

    IP

    Wave 8

    GPS/EMB

    BHPS

    IEMB

    IP

    Wave 9

    GPS/EMB

    BHPS

    IEMB

    IP

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Household design

    • First task on making contact and gaining co-operation: Enumerate the household• Who is there? Any new people? Any movers?

    • Household questionnaire: 10-15 minutes, 1 per household• Adult interviews (age 16+): average 40 minutes each• Youth self-completions (age 10-15)

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Content of the survey

    Enumeration grid• Basic demographic of the household members (age 0+)

    • Name, sex, date of birth, employment status, relationship status, relationship to others in the household

    • Movers/leavers – date of change, reason for changeHousehold questionnaire• Information about accommodation and household-level expenditure

    • Housing tenure, rent/mortgage, house value• Spending on food, non-food, utilities (gas, electricity)• Ownership of consumer durables, cars• Difficulties making household payments

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Content of the survey

    Individual (adult) interview• Core content – update on changes in the last year:

    • Residential mobility• Marital status• Cohabitation status• Fertility history (including information on childbirth and

    behaviour during and after pregnancy)• Periods of full-time education• Periods of training• Employment status changes• Health changes

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Building up a picture of change over time

    2012 2013 2014 2015Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4

    Point of interviewPoint of interview

    Retrospective data

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Content of the survey

    Individual (adult) interview• Rotating content: modules asked every 2, 3, or 4 years, e.g.,

    • Wealth, assets and debts (4 yr)• Social/friendship networks (3 yr)• Political and social engagement (3 yr)• Identity measures (3 yr)• Social support (3 yr)• Local neighbourhood (3 yr)• Family relationships and parenting (2 yr)• Family networks (2 yr)• Environmental behaviour (2 yr)• Leisure participation (3 yr) • Financial behaviour and attitudes (2 yr)• Employment conditions (2 yr)• Well-being and sleep quality (2 yr)

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Content of the survey

    Individual (adult) interview• Also, age-related

    • Retirement planning (45, 50, 55, 60 yr olds)• Child development (when child is 3, 5, 8 yrs)• Educational aspirations (young adults and parents)

    • Or time-triggered• General election• Scottish referendum• Olympic Games

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Survey structure – possible paths

    Household grid

    Household questionnaire

    Individual questionnaire 1

    Individual questionnaire 2

    Individual questionnaire 1

    Household questionnaire

    Individual questionnaire 2

    Household questionnaire

    Individual questionnaire 2

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    How is the survey conducted?

    Wave 1 and 2• Face-to-face onlyWave 3-6• Face-to-face• Telephone mop-up for non-responders

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    How is the survey conducted?

    Wave 7• Face-to-face for Wave 6 responding households• Web for Wave 6 non-responding households

    • Face-to-face follow-up for non-responders (“Web-first”)• Telephone mop-up

    Wave 8• Face-to-face for 60% of Wave 7 responding households

    • Non-respondents invited to participate on-line (“CAPI-first”)• Web-first for 40% of Wave 7 responding households + Wave 7 non-

    responding households• Telephone mop-up

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    The rise of mixed-modeProportion of the sample issued to each mode

    0%

    20%

    40%

    60%

    80%

    100%

    Wave 6 Wave 7 Wave 8 Wave 9 Wave 10 Wave 11

    CAPI-first Web-first

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Why mixed-modes?

    • Requirement from funders to reduce costs to make the study sustainable

    • Innovate• Try to increase response by allowing people to take part in a

    way which they prefer

    • Sequential mixed-mode design• Non-respondents to initial mode re-issued to follow-up

    mode

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Mixed-mode design requirements• Reduction of costs

    • Whole-households complete in cheaper mode• The effort required by interviewers is reduced for partial-

    completion households• Increase in response

    • The cheaper mode brings in people who would not have been interviewed otherwise

    • Having invited people to the cheaper mode, non-response doesn’t affect their propensity to respond in the more expensive mode

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Our design can lead to some complications• Household

    • Want all adults to participate• …even adults we don’t know about yet

    • Longitudinal• Need specific adults to participate• …even if they have moved house

    Requirement to: Enumerate the household first Deal with movers in and out of the household Follow-up non-respondents in another mode

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Experience of mixed modes?

    • Use of the Innovation Panel• IP2 – experiment with CATI-CAPI

    • Lower response than CAPI alone• Lower response among those who require more effort• Lower response ‘scarred’ the sample for the next two

    CAPI-only waves

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    CATI reduced response and the effect lasted to the next wave

    Response Rate

    Wave 2 Wave 3(All F2F)

    Wave 4(All F2F)

    Face-to-face 75.8% 66.3% 57.3%

    CATI-CAPI 67.4% 59.9% 53.7%

    Difference -8.4 -6.4 -3.6

    Odds Ratio (P) 0.67 (0.001) 0.79 (0.07) 0.88 (0.29)

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    IP5 onwards – use of web

    2/3 sample –allocated to WEB

    1/3 sample allocated to F2F

    Non-responders allocated to F2F

    2 week “web-only” period. Advance letter + email

    plus 1 letter and 2 email reminders

    Non-responders allocated CATI

    mop-up

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    IP5 Household Response Rates

    F2F MM P

    Household response rate 84.1 81.1 0.29

    Complete Households 63.9 55.7 0.02

    Partial Households 20.3 25.4 0.07

    Non‐contact 5.6 2.9 0.04

    Refusal 8.7 14.1 0.05

    N  322 621

    * IP4 responding households

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    IP6 Household Response Rates

    £10 incentive  IP5 responding

    F2F MM P

    HH response rate 85.2 75.8 0.03

    Complete HHs 65.4 64.2 0.74

    Partial HHs 19.8 11.6 0.004

    Non‐contact 7.2 11.9 0.03

    Refusal 7.6 12.3 0.04

    N  445 268

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Effects of Incentives: IP6

    Incentive £10 unc £10 unc+ £20 con

    £30 unc P

    HH response rate 75.7 84.8 86.2 0.03

    Complete HHs 64.2 71.1 74.6 0.03

    Partial HHs 11.6 13.7 11.6 0.68

    Refusal 12.3 7.6 6.0 0.03

    Other NR 11.9 7.6 7.8 0.14

    N 268 277 268

    * Mixed mode sample, IP5 respondents

    F2F £10 household response: 85.2%Require £30 incentive to get similar response using mixed-modes

    Annette Jäckle, Peter Lynn, Jonathan Burton: “Going online with a face-to-face household panel: initial results from an experiment on the Understanding Society Innovation Panel”, Understanding Society Working Paper Series: 2013-03

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Mixed Modes over Multiple Waves

    Percentage of completed interviews completed by web:

    .Source: Professor Peter Lynn’s own calculations

    Household interview %

    Individual interview %

    Wave 5 31.3 30.0

    Wave 6 36.8 38.3

    Wave 7 42.8 43.6

    Wave 8 44.8 45.0

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Attrition: Web + CAPI

    • Achieved higher response rates for t-1 non-respondents than CAPI only Implemented in main survey from Wave 7 (2015)

    • Household attrition rates compared to CAPI only 3-6% higher with standard £10 unconditional

    incentive Comparable/lower with higher unconditional or

    conditional incentives• Proportion of Web completers is increasing

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Measurement Issues• NatCen Social Research commissioned to code all questions

    proposed for waves 7 – 10 with respect to 13 characteristics associated with increased risk of measurement differences between modes

    • Summary measure of ‘risk of mode effect’ derived for all 1,486 questions

    • Many of the more sensitive questions (health, attitudes, satisfaction) are already asked by self-completion within CAPI; mode effects likely to be negligible for these

    • Other questions most at risk of mode effects are:• Questions about harassment, prejudice etc, asked as part of the

    ethnic minority research strand;• Questions that benefit from interviewer encouragement/ support to

    answer fully and accurately, including income.

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Measurement / data comparability: Web + CAPI• No concerns about Break-offs Missing or duplicate instruments HH grid identification of leavers/joiners

    • Web respondents are not representative More affluent

    • Web + CAPI follow-up sample is similar to CAPI only Web-only not an option!

    • Accounting for selection into mode Few differences in response distributions

    Forthcoming research from Annette Jäckle

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Web Conclusions

    • Initial response rate lower when switching to web-first, but recovers after a couple of waves

    • Previous wave non-respondents are more likely to respond with web-first design than with CAPI-only

    • Cost savings should soon accrue as around 25% of responding households respond fully by web

    • Percent responding by web appears to increase over waves• May be possible to avoid potential increases in item response

    rates• Measurement error issues remain a concern; our focus is to

    minimise impact and provide analysis guidance

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Adaptive design on the main-stage (Wave 8)• More of a risk?• Try to restrict to adaptations that have at worse “no effect”• Either cost-saving or cost-neutral• Experimental allocation from Quarter 2• Allocation to mode non-random

    • All previous-wave non-responding households: Web-first• CAPI-first:

    • Ring-fenced sample (20% households)• Lowest propensity to complete online (20%)

    • Of the remaining 60%• 20% CAPI-first• 40% Web-first

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Month 1 – standard £10 unconditional incentive

    0%

    5%

    10%

    15%

    20%

    25%

    30%

    35%

    40%

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

    January

    18.6% completed households

    Days in the field

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Month 2 – addition of £10 conditional incentive if complete in first 2 weeks

    0%

    5%

    10%

    15%

    20%

    25%

    30%

    35%

    40%

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

    JanuaryFebruary

    37.4% completed householdsDeadline for

    bonus

    Days in the field

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Month 3 – incentives same as month 2, half sample get additional reminder letter

    0%

    5%

    10%

    15%

    20%

    25%

    30%

    35%

    40%

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

    JanuaryFebruaryMarch

    37.5% completed households

    Days in the field

  • Month 3: Additional reminder letter increased whole-household web completion

    Invite + 1 week + 1 week

    Control Letter + Email Email Email

    Treatment Letter + Email Letter + Email Email

    16% 17%

    33%

    44%

    0%

    10%

    20%

    30%

    40%

    50%

    Control Treatment

    Before 1streminder

    End of web-only

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Months 4-6 – ‘trigger’ experiments

    • Experiment with the ‘trigger’ for the bonus• Adult completion• Whole-household completion

    34,1% 31,7%

    22,7%19,9%

    0%

    10%

    20%

    30%

    40%

    50%

    60%

    Individual trigger Household trigger

    Partial household

    Whole-household

  • Months 1-9: Different ‘web-only’ periods

    0%

    5%

    10%

    15%

    20%

    25%

    30%

    35%

    40%

    45%

    1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35

    % fully complete web

    January

    February

    March

    April

    May

    June

    July ‐ 5 week

    July ‐ 3 week

    Aug (5 week)

    Sept ‐ 5 week

    Days in the field

  • SURVEY RESPONSE

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Response rates

    • Response rates an important indicator to the quality of the study for users, funders

    • Response also affects sample size• Particularly important in a longitudinal study

    • Measures over time required• Missing a wave, or drop-out, affects longitudinal sample

    for analysis

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Fieldwork – response rates

    57,6% 56,0% 55,8% 55,5% 54,8%

    0%

    20%

    40%

    60%

    80%

    100%

    UnderstandingSociety GPS

    CitizenshipSurvey

    ESS 2008 Labour ForceSurvey (W1)

    Wealth &Assets (W1)

    Wave 1 – below the 60% target, but in line with other surveys at the same time

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Response over timeHousehold response rate for previous-wave

    responding households

    60%

    65%

    70%

    75%

    80%

    85%

    90%

    95%

    100%

    Wave 2 Wave 3 Wave 4 Wave 5 Wave 6 Wave 7* Wave 8*

    GPSEMBBHPS

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Response over timeHousehold response rate for previous-wave

    responding households

    60%

    65%

    70%

    75%

    80%

    85%

    90%

    95%

    100%

    Wave 2 Wave 3 Wave 4 Wave 5 Wave 6 Wave 7* Wave 8*

    GPSEMBBHPS

    ?

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Wave 6 fall in response?

    • At Wave 6, the fieldwork agency changed• Almost all sample members were contacted by a different

    interviewer at Wave 6• Figures for Wave 7 (and 8) look as if the upward trend has

    recovered

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Activities to increase response

    • Survey design under constant review to try and improve response

    • …or maintain response at a lower cost• Advantage of having the Innovation Panel

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Response: Fieldwork design

    • Use of different modes to allow sample members to participate in a mode that is convenient to them

    • Email reminders for break-offs• Minimum number of call attempts before making contact (inc.

    evenings/weekends, over at least 3 weeks)• Long fieldwork period to allow interviewer flexibility• Translate participant materials/questionnaire• From Wave 8, allow interviewers to telephone first to make an

    appointment• Fieldwork monitoring reports available daily• Non-respondents re-issued to another interviewer• Telephone mop-up at end of fieldwork period

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Response: Incentives

    • Unconditional £10 gift voucher for all issued adults, in advance letter• From Wave 6: £20 unconditional for adults in previous-wave

    refusal households, £20 conditional for adults in previous-wave non-contact households

    • From Wave 8: Previous-wave adult non-responders offered conditional £10 incentive

    • From Wave 8: Additional £10 bonus for early-completion online• Unconditional £5 gift voucher for young people (10-15 years)• £5 voucher for returning a change-of-address (COA) card• Entry into quarterly prize draw to win iPad for 16-year olds

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Response: Tracing/panel maintenance• Maintain Freepost, Freephone, email address, web-form to enable

    sample members to contact us• Provide sample members with COA cards• £5 incentive for returning COA cards• Collect multiple contact details (home phone, work phone, mobiles,

    email addresses)• Collect stable contact information• Use of tracing letters• Weekly tracing file sent to ISER

    • Email untraced movers• Use of online databases• Use of out-of-hours telephone tracing staff

    • Additional mailing of ‘likely movers’

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Response: Participant Communications

    • Advance letters/emails tailored to different sample status• Introduction leaflet for new entrants• Between-wave mailing with findings from study (with COA)• “likely mover” mailing• Participant website, regularly updated• Use of social media for dissemination of news/research• Regular (3 x year) Participant Update emails• Birthday cards for those turning 16• Case studies for use on doorstep

  • UNDERSTANDING SOCIETY:THE FUTURE…?

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Currently…

    • Scripting/testing Wave 9 pilot (October-November)• Scripting IP10• Developing Wave 10 content• Preparing Wave 6 data for release

    • Starting a new fieldwork contract for Waves 9-11• Working with TNS BMRB and NatCen Social Research

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Coming up…

    • Further increase of Web-first sample• Further experimentation on Wave 8 adaptive design

    • With successful protocol adopted for Wave 9• Review of content across key topic areas (Topic Champions)• Increasing focus on scientific and policy impact• Feasibility of between-wave online surveys

    • Sub-groups of sample• Event-triggered (e.g., elections)

    • Feasibility of using new data collection technologies

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Survey

    Store card data

    Clickstream data

    Barcode scanners, till receipts

    Wearables

    Financial aggregators

    Social media

    Smartphone apps

    Post

    New data collection technologies

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Example: Collect household budget

    • Improve Data Quality by imposing accounting identities

    Income – Expenditure = Change in Assets and Debts

    Resources available

    Net disposable

    income

    Borrowing / dissaving

    Repayment of loans/ savings

    Expenditure

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    • Improve measurement of net monthly income:1. Additional questions2. Income summary screen

    • Benefit Unit budget reconciliation:3. New module for couples

    • Data linkage: Financial Conduct Authority • Pilot new technologies

    4. App: scan till receipts

    • Between wave web survey (development)5. Web survey: gamified tool to capture income & expenditure6. Benefit Unit budget reconciliation 7. (Possible) Preload data collected with new technologies

    IP: New technologies, data linkage

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Accessing the data and findingsStudy documentation:• Protocols, fieldwork reports, questionnaires, variable level metadata• User guides, Quality Profile, technical reports• Working Papers and publications from methodological researchSee www.understandingsociety.ac.uk

    Data are available at the UK Data Service, University of Essex:• Wave 1 - 5 data, including biomarkers; wave 6 and immigrant &

    ethnic minority boost sample to be deposited November 2016• Innovation Panel data waves 1-8• Linked education data, geographical identifiers available by special

    licence or via secure labSee UK Data Service website: http://ukdataservice.ac.uk/

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Thank you

    Jonathan BurtonISER, University of Essex

    [email protected]@jburton123

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Data – structure

    • Understanding Society is a complex data-set• Hierarchical

    • All sample households, all sample individuals• Responding households• All adults• Responding adults• Loops with adults: employment spells, information about

    children• Plus, paradata – call records, timings, interviewer

    characteristics• 15 data-sets per wave

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Data – content

    Questionnaire data, plus:• Derived variables

    • Pointers to parents/spouse/children, Socio-economic and occupational classifications of jobs, Summary scores such as BMI, GHQ, SF-12 and SDQ, household typologies, household incomes, net and gross incomes … and lots more

    • Weights• Design weights, individual and household response weights,

    cross-sectional and longitudinal • Imputed values

    • to replace missing values on individual components of earned and benefit incomes that contribute to total income

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Data – help

    • “Getting started” guide• User guides• On-line data documentation• Regular training courses• Support forum

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    Main-stage

    0

    200

    400

    600

    800

    1000

    1200

    1400

    1600

    2010 2011 2012 2013 2014 2015

    UKDS Downloads

    Total

    Linear (Total)

    0

    100

    200

    300

    400

    500

    600

    700

    800

    900

    2010 2011 2012 2013 2014 2015

    New Unique UKDS Users

    Total

    Linear (Total)

  • IP

    0

    50

    100

    150

    200

    250

    2011 2012 2013 2014 2015

    UKDS Downloads

    Total

    Linear (Total)

    020406080

    100120140160180200

    2011 2012 2013 2014 2015

    New Unique UKDS Users

    Total

    Linear (Total)

  • Special License

    0

    50

    100

    150

    200

    250

    2011 2012 2013 2014 2015

    UKDS Downloads

    Total

    Linear (Total)

    0

    10

    20

    30

    40

    50

    60

    70

    2011 2012 2013 2014 2015

    New Unique UKDS Users

    Total

    Linear (Total)

  • HE users

    Government users

    Charities

    Think tanks

    Membership orgs

  • Dutch Platform for Survey Research8th September 2016. Amsterdam

    UKDS UK HE Users by discipline

    0

    100

    200

    300

    400

    500

    600

    2010 2011 2012 2013 2014 2015

    EconomicsSociologyHealth SciencesGeographyStatsBusinessPoliticsEducation