online opinions – a pilot to extend ons social data collection capabilities ed dunn ons social...
TRANSCRIPT
Online Opinions – A Pilot to Extend ONS Social Data Collection Capabilities
Ed DunnONS Social Survey Division
Overview
• Background and context setting• The pilot study; our key objectives and design• Findings • Next steps
Pilot Key Objectives – ‘Dipping our Toes’
• Need to examine web collection• Investigate unit and item response rates• Investigate respondent characteristics• Evaluate the hardware/software capabilities
Examine the effect of offering an incentive• Design and administration of web surveys• Recommendations for further work
• Deliver to a short timescale (by December 2008)
Sample Design and Respondent Selection – Web Pilot
• Simple random sample of 2,000 households from English Postal Address File
• Advance letter • Half offered a £10 Amazon gift certificate to
take part online• Letter asked adult with most recent birthday
to complete the survey• Unique username and password provided to
access survey from ONS website• Follow-up letter
Sample Design and Respondent Selection- Opinions Survey (OPN)
• OPN is multi-stage cluster design • Respondent selection via KISH grid• November OPN sampled 2,010 households• Book of postal stamps sent with advance
letter
Questionnaire Design (1)
• Based upon ONS Opinions Survey (OPN)• Used question blocks equivalent to a ‘normal’
month of OPN• Questions reviewed for suitability for web• Some modifications necessary• Questionnaire tested internally
Questionnaire Design (2)
• Design and layout reviewed for web survey*
• Page Design
• Next/Back Button
• Progress Bar
* See Dillman, D.A (2007) Mail and internet surveys - the tailored design method (2nd edition) J Wiley & Sons, New Jersey
Unit ResponseInternet pilot
• 364 fully complete responses received (18%)• 32 partial responses (2%)• Little difference between incentive (18%) and non-
incentive (19%) group
• But, Full sample = 2000
10% unknown eligibility and only 61% have internet access at home (LCF)Available sample = 1098 New Response = 33% complete responses
November OPN
• 59% overall response (with known eligibility)
Metrics
• 80% of respondents completed web survey in under 25 minutes
• 75% completed 8am-6pm• Most popular hours to begin survey were
4pm-5pm (10%) and 7pm-8pm (11%)• 25% completed on weekend• Wednesdays and Fridays most popular
• OPN face to face takes c.45mins (excl. admin time)
Results (1)
• Weighted results; OPN routinely weighted but pilot required some methodological work. OPN was then re-weighted to same population totals.
• 2-tailed t test applied to differences
• Some clear differences in profile of internet pilot respondents compared to OPN and key question responses
Results (2)
• The pilot respondents were more likely to be:• Aged 25-44 and 55-64 • Married and living with partner• White • Better educated (with degree level
qualification)• Managers or supervisors • In good health
Results (3)
• In terms of key question responses pilot respondents were less likely to:
• Smoke• Have a disability• Think charities played an important role in
society• Think HM Revenue & Customs treated them
fairly
• Pilot respondents were more likely to:• Eat healthily (5 portions of fruit or vegetables a
day)
Next Steps
• Limitations of this pilot widely accepted• Pilot has demonstrated viability of conducting a major
ONS social survey online and receiving a significant, if minority, response
• Pilot has highlighted logistical and technological issues
• Pilot has highlighted risk of significant bias• Identified areas for further work via more
sophisticated pilot designs• Plan to run a further pilot on OPN in 2010 and pilot
on LFS early in 2011
References
• Dillman, D.A. 2007. Mail and Internet Surveys – The Tailored Design Method (2nd ed). J Wiley and Sons, New Jersey
• Flatley, J. 2001. The Internet as a Mode of Data Collection in Government Social Surveys: Issues and Investigation Social Survey Methodology Bulletin, ONS 49(7)
• Fricker, R and Schonlau, M. 2002. Advantages and Disadvantages of Internet Research Surveys: Evidence from the Literature. Field Methods 14(4)
• Solomon, D. 2001. Conducting Web-Based Surveys. Practical Assessment, Research and Evaluation 7(19)
Internet and the UK LFS
•Current Modes:
•Wave 1 Face to Face Wave 2-5 TU or ftf
•Response on UK LFS all time low 65% wave 1, 57% overall.
•Achieved sample size decreasing or much more costly to obtain
•Efficiency savings face to face costs 3 times TU•Pilot shows some groups with current high non-response will take part in the internet.•TU costs 3 times Internet.
Internet and the UK LFS
• Pilot showed technology exists.• 4 main Problems:• Collecting close to the Reference week • Problem of integrating the internet with the
other modes to prevent bias• Mode effects• Collecting from Households not persons
Face to Face W1 TU W2-5
Internet and the UK LFS
• Analysed other approaches but derived duel sample approach after considering results of the pilot
Initial Sample
Registered for internet
Respond to internet
FtF W1 TU W2-5
Internet and the UK LFS
• Currently reviewing questionnaire for internet suitability based on pilot results and others research to minimise mode effects.
• Household collection seems feasible• Use Confirmit to test ideas• Use BLAISE IS? To run pilot in 2011 in a few
NUTS2 areas• If successful run full design by 2012• Modify collection by 2013
Internet and the UK LFS
• Issues
• Internet pushes up refusals
• Unequal response
• Changes to internet over time
Internet and the UK LFS
• Also monitoring wave 2-5 work by other NSi’s to implement if shown to be successful
• But current systems do not allow internet, TU and ftf integration.
• Cost savings and non-response mainly achieved for UK at Wave 1.
Internet and the UK LFS
• So far we have had:
• OPN Pilot
• Input at International Workshop
• Prof. Mick Couper’s work and training
• Literature review
• But interested to have design scrutinised for issues.