iaos 2014 da nang: an agile approach to question testing and satisfying a new data requirement
DESCRIPTION
IAOS 2014 Da Nang: An agile approach to question testing and satisfying a new data requirement. Pete Brodie ONS, UK. Overview. Background Drivers Questionnaire requirements Delivery dilemma Solution and outcome New output requirement “Agile” methodology Outcome Results Next steps - PowerPoint PPT PresentationTRANSCRIPT
IAOS 2014 Da Nang: An agile approach to question testing and satisfying a new data requirement
Pete Brodie
ONS, UK
Overview
• Background• Drivers• Questionnaire requirements• Delivery dilemma• Solution and outcome• New output requirement• “Agile” methodology• Outcome• Results• Next steps• Conclusion
Background
• ONS delivers a very broad spectrum of statistics
• Financial pressures on all NSIs• Difficult to react quickly to new
requirements• Scarcity of specialist skills
Drivers
• Changes due to ESA 10 and BPM6 - updated regulations for producing National Accounts
• Changes in the labour market – the way people are employed
Questionnaire requirements
• Early in 2014 legislative changes required substantial changes to most ONS business statistics
• First tranch of 28 Finance Surveys• Total redesign of questionnaires• Additional questions• Covering new sectors of the economy
Delivery dilemma
• Standard approach for DCM uses multiple waves of cognitive interviews with responders
• Transcription of interviews• Analysis using thematic framework • Detailed report of issues and
recommendations• Time and available resource – impossible!
Solution and outcome I
• Brand new approach needed trialled for two surveys
• Condensed usual waves of testing into one week “stints”
• Three stages to each stint
Monday Tuesday Wednesday Thursday Friday
Stage 1 Stage 2 Stage 3
Fieldwork Collaborative meetings – DCM, Clients, Stakeholders
Changes agreed
Solution and outcome II
• Traffic light document
Solution and outcome III
• Extensive changes implemented for two surveys – agreed between all stakeholders
• Real improvements made• All questions rated “green” at end• Client feedback very positive• Re-use methodology with some tweaks
New output requirements
• Autumn 2013 demand for measurement of the number of employees on “Zero Hours” Contracts
• Definition unclear• No information from businesses• Promise to produce estimates early in 2014• Conventional approach of qualitative and
quantitative research likely to take around three years
“Agile” methodology I
• Staged approach – pilot followed by live survey
• Combined qualitative and quantitative work
• Novel (for ONS) approach of telephone survey of businesses
• Used flexible “Statistical Computing” for all quantitative production
“Agile” methodology II
• Pilot survey of around 1000 businesses (30 strata)• Supplemented frame information with contact
numbers• Collaborative working between different parts of
the office• In field for four weeks achieved 60% response• Qualitative interviews with selected responders• Bespoke code for editing, outlier detection and
estimation
“Agile” methodology III
• Quantitative estimates from pilot used to optimise survey design for main survey – aim for 10% cv at top level
• Questionnaires updated using qualitative research
• Main survey of 5000 businesses included all pilot sample and 4000 new members
• In field for six weeks achieved over 60% response
Outcome
• Published results on 30th April – first business survey estimate of the number of live contracts of 1.4 million (c.v. 8.5%)
• Greater understanding of term -“contracts with no guaranteed minimum hours”
• Comparison with estimates from Labour Force Survey
• Published some more detailed estimates• Some data available for longitudinal analysis
Results I
Proportion of business using ZHC by size
Results II
Proportion by broad industry
Next Steps
• The LFS questionnaire was updated in the light of findings and to answer detailed demographic questions
• Follow up business survey of 5000 now in the field
• Considering longitudinal design• Further embedded qualitative work• Considering longer term strategy
Conclusions
• These two examples illustrate what is possible when using an agile approach.
• Too often constrained by artificial boundaries.
• Opportunities for development of staff and methods through collaborative working.
• Necessity is the mother of invention!
Thank you!
Any questions ?