The Philosophy of French experiments

on Internet and mixed modes data collection

Tiaray RazafindranovonaSeptember 26th, 2013

2

Introduction

• Better quality, more complexity …

• … in a general context of budgetary reduction !

• Mixed modes to face these challenges ?

• Experimental surveys will help to answer

3

Principles of French experiments

• An experiment for (almost) each new household survey– General results on mixed modes surveys not sufficient – Experiments in parallel of each new CAPI survey

• No interference with main survey– Workload– Sampling frames

• Each survey must test something new– Accumulation of experience– Test of new specific points not already handled

4

• Sample frames– Name and adress for massmailing– Available in taxpayers files

• Questionnaire– Fitted subset of the CAPI questionnaire– Length around 10/15 minutes

• Notification letter, reminders– First, a postal mail with a notification letter– 2 reminders (one with paper questionnaire)

Standard protocol

5

• Housing 2010 (Amiel and Denoyelle, 2012)– Comparison with results of Housing 2006 (face-to-face)– Internet respondents less satisfied (calibration techniques)

• Safety survey (SASU) 2013– First experimental survey on victimization– 2 protocols to control the problem of self-selection

• Housing 2014– Focus on rental amount– External documents without the interviewer ?

• Wealth 2014– Focus on value of real estate property– Deviations from market value without the interviewer ?

Examples of experiments

6

• Working conditions survey (face-to-face survey)

Survey adresses many topics surrounding working conditions : – working hours and organization of working time– psychosocial constraints– relations with the public …

• Experimental parallel survey : Quality of life at work

Tackles same topics : shorter version of the questionnaire.

Sample drawn in a base built on taxpayers files : only people that have declared earnings (in 2010) from work can be selected.

Working conditions Quality of life at work (QLW)

7

• Aims– Comparison of

• Response rates between internet and internet+paper• Quality and content of responses between traditional and experimental surveys• Responses between 2 protocols involving different notification letters

– Test questions of the future survey on psychosocial risks (2015)– Open questions to raise topics for this survey

• Protocols– 2 different models of notification letters (psychosocial risks)

– For half of the sample, internet and paper (standard protocol), for the other half, only internet

QLW – Aims and protocols

8

At the end of the data collection

Subsample 1 - model A letter, internet and paper available as modes of responseSubsample 2 - model B letter, internet and paper available as modes of responseSubsample 3 - model A letter and only internet available as mode of responseSubsample 4 - model B letter and only internet available as mode of response

(model B letter insists on psychosocial risks)

QLW – Response rates

9

Comparison on sample frame variables

QLW – Sample vs internet / paper

10

QLW – Internet vs paper respondants

Comparison on survey variables

11

• Differentiation on responses scale (satisficing)– Control of selection with kernel matching on propensity score– Index of response differentiation (McCarty and Shrum, 2000)

– Result : differentiation more pronounced on internet than on paper

• Emotional well-being– Control of selection with kernel matching on propensity score– WHO-5 well-being index (de Wit et al., 2007)

– Result : emotional well-being higher on paper

QLW – Examples of mode effects

12

• Mixed modes : adaptations to technological changes

• Not to be forced without prior evaluation of quality

• Future results of experiments will help to re-design

• A standard scheme ? Trade-off with society requests

Conclusion

13

References

Amiel, M. H,. and Denoyelle, T. (2012), Enquêtes en ligne : comparaison de modes de questionnement sur le thème du logement, XIèmes Journées de Méthodologie Statistique de l’Insee

de Peretti, G., and Razafindranovona, T. (2013), Les enquêtes multimode : multi-solution ou multi-problème ?, 45e Journées de Statistique de la SFDS.

de Wit, M., Pouwer, F., Gemke, R., Delemarre-van de Waal, H. and Snoek, F. (2007), Validation of the WHO-5 Well-Being Index in Adolescents With Type 1 Diabetes, Diabetes care, 30.

Heckman, J., Ichimura, H., and Todd, P. (1998), Matching as an Econometric Evaluation Estimator, Review of Economic Studies,65.

Krosnick, J. (1991), Response strategies for coping with the cognitive demands of attitude measures in surveys, Applied Cognitive Psychology. Vol 5.

Lugtig, P. J., Lensvelt-Mulders, G. J., Frerichs, R., and Greven, F. (2011), Estimating nonresponse bias and mode effects in a mixed mode survey, International Journal of Market Research, 53(5).

Lyberg L. (2012), La qualité des enquêtes, Techniques d’enquête, 38(2).

McCarty, J. A., and Shrum L. J. (2000), The measurement of personal values in survey research: A test of alternative rating procedures, Public Opinion Quarterly, 64.