mixed mode in lfs: questionnaire design and mode-effects
DESCRIPTION
Mixed mode in LFS: questionnaire design and mode-effects. Liisa Larja, Statistics Finland 9th Workshop on Labour Force Survey Methodology Rome, 15 th of May 2014. StatFin/EssNet web-project:. Designing LFS-CAWI questionnaire + iterative testing with cognitive interviews (N = 22) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/1.jpg)
Mixed mode in LFS: questionnaire design and mode-effectsLiisa Larja, Statistics Finland9th Workshop on Labour Force Survey MethodologyRome, 15th of May 2014
![Page 2: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/2.jpg)
StatFin/EssNet web-project:
1. Designing LFS-CAWI questionnaire + iterative testing with cognitive interviews (N = 22)
2. Pilot survey (N = 8 000), experimental design (”split-plot”) with DK-option, working time and job search questions. Comparison to CAWI data from the same period. Paradata, respondent feedback.
![Page 3: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/3.jpg)
Content of the presentation:
1. Questionnaire design: results & recommendations
2. Data collection process
3. Mode-effects: WSTATOR (employment status)
4. Mode-effects: HWACTUAL (actual working hours)
5. Qualitative feedback
![Page 4: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/4.jpg)
1. Questionnaire design: layout
BlaiseIS default layout was not perceived as attractive Number of question / page Navigation bars Instructions Error messages Don’t know –option (DK)
![Page 5: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/5.jpg)
Error message, DK
Instructions, info-button
Status barNumber of questions, updating the remaining questions
Scrolling down for navigation bars
![Page 6: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/6.jpg)
When the cursor points the info-symbol, the instructions appear in a box below.
![Page 7: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/7.jpg)
Navigation bars:Takaisin = Back Jatka = ContinueJatka myöhemmin = Continue later/pause
Kyllä = YesEn = No(Don’t know not visible)
![Page 8: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/8.jpg)
Kyllä = yesEi = noEn osaa sanoa = Don’t know
An error message: Please respond to this question or choose the “Don’t know” option.
DK-option, version 1
![Page 9: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/9.jpg)
Kyllä = yesEi = noEn osaa sanoa = Don’t know
DK-option, version 2
![Page 10: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/10.jpg)
2. Data collection process
Response rate 30 % potential for savings in data collection costs when the sample is large enough
CAWI-data more ”distorted” ? CAPI non-response more evenly distriputed with regard to educational background, gender and age than CAWI-data.
Speed: 40 % responses were received withing 3 days after that switch to CAPI?
![Page 11: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/11.jpg)
3. Mode-effects: WSTATOR (employment status)
![Page 12: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/12.jpg)
3. Mode-effects: WSTATOR (employment status)
No significant mode-effects, but….• media attention remains if unemployment rate 8,2 % instead of 7,4 %• sample for pilot study was small• CAWI respondents answer more often that they have been looking for job• = mode effect or selection bias?
![Page 13: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/13.jpg)
4. Mode-effects: HWUSUAL (actual working hours)
”One question” –format for HWACTUAL:
“How many hours did you work in your main job during the week 7.10.-13.10.2013?”
___ hours ___ minutes
![Page 14: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/14.jpg)
”Grid” –format for HWACTUAL:“How many hours did you work in your main job during the week 7.10.-13.10.2013?”
Date Hours Minutes
Monday 7.10.2013 ___ ___
Tuesday 8.10.2013 ___ ___
Wednesday 9.10.2013 ___ ___
Thursday 10.10.2013 ___ ___
Friday 11.10.2013 ___ ___
Saturday 12.10.2103 ___ ___
Sunday 13.10.29013 ___ ___
Total ___ ___
![Page 15: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/15.jpg)
Average number of hours worked (CATI-data and two different CAWI-datas)
![Page 16: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/16.jpg)
5. Qualitative feedback
Too intimate information on employer Questions do not fit for people with multiple jobs
without main job or with scholarship All kind of entrepreneurial activity is not covered Working time items do not fit for people in atypical
employment Pensioners do not understand point of reference week
![Page 17: Mixed mode in LFS: questionnaire design and mode-effects](https://reader034.vdocuments.net/reader034/viewer/2022051402/56815a8e550346895dc80411/html5/thumbnails/17.jpg)
Conclusions and lessons learnt Response rate of 30 % and speed of data collection
encourage to continue towards mixed-mode LFS.
Visual design is important: balanced layout, intuitive navigation bars, informative error messages, a few questions per page, hidden instructions, hidden DK.
CAWI data more ”distorted”
In working time questions different format for CATI and CAWI may be needed to reduce mode-effects (”functional equivalency”). Differences in WSTATOR need to be further analysed.