© federal statistical office (fso), institute for research and development in federal statistics...

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 1

Quality of Pretesting: Instruments for Evaluation and Standardization

Session 23:Survey measurement issues

Q2010 in Helsinki

May 3-6, 2010

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics

Contents

Pretesting at the FSO

Quality standards in qualitative pretesting

Future prospects


Institutional background Code of Practice (2005), principle 8: “Questionnaires

are systematically tested prior to the data collection.”

Eurostat QDET (2006): systematic testing in the following cases

a new survey new or modified questions additional or modified data collection instrument poor data quality

Pretesting: Increase in data quality Decrease in respondents’ burden


Methods of pretesting Quantitative testing methods:

Multitude of probands (N > 100) Under field conditions Frequency of problems with the questionnaire

Qualitative testing methods: Limited number of probands (N ≤ 20) Under laboratory conditions Reasons for problems with the questionnaire First ideas for improvement

Three step approach


Step I: Observation Sources of information:

Gestures, facial and short verbal expressions (“reality without words”)

Remarks in the questionnaire

Gain of knowledge: Independent and unaffected behavior without any

advance information


Step II: Cognitive interview Sources of information:

Insights in the response process by the use of cognitive methods

Narrative description of personal situation

Gain of knowledge: Reasons for incorrect or

missing answers Individual reality

questionnaire Suggestions for improvement

Judgment

Comprehension

Information Retrieval

Response

(Tourangeau/Rips/Rasinski 2000)


Step III: Evaluation of the questionnaire

Sources of information: Entries in the questionnaire Remarks, question marks, etc.

Gain of knowledge: Actual handling of the questionnaire beyond what

respondents thought they had understood


Contents

Pretesting at the FSO

Quality standards in qualitative pretesting

Future prospects


Need for quality standards

Qualitative methods are often criticized as being unreliable, unrepresentative and insignificant

Statistical offices traditionally work quantitatively

new development to elaborate standards forqualitative data and to improve their

explanatory power


Criteria for high quality of qualitative data

Checking for generalization without verification Checking for representative probands Checking for researcher effects Triangulation Balancing the evidence

(Miles/Huberman 1994)


Checking: generalization without verification

Avoid to regard conclusions for one or two very striking probands as typical (“You see what you want to see.”)

Safeguards: Consider positive and negative evidence Quantify qualitative data by the use of QDA

software and matrices Double-check codings and conclusions in team


Checking for representative probands Approximately 20 probands who represent

the ordinary respondent in official statistics; group shall be as heterogeneous as possible

Safeguards: Select probands adequate for the target

population Invite probands with different social

background by different ways of recruitment Establish a data base with information on

probands


Checking for effects on probands Intimidated by the test situation Social desirability or acquiescence Concerns about providing information to the

“government”

Safeguards: Create a comfortable atmosphere Warming-up (course of the test, expectations

towards the probands) Underline anonymization and confidentiality


Checking for effects on interviewer Leading questions Losing distance (”going native“)

Safeguards: React in an adequate way, remain neutral Avoid additional remarks on personal opinion

or survey question Ask for mutual feedback in team


Triangulation

Confirming results by replicating them Taking different perspectives on the questionnaire Gain an overall picture

Safeguards: Data triangulation (probands, places, points in time) Researcher triangulation (teamwork) Methods triangulation (three step approach)


Methods triangulation

Overallpicture

Observation

Questionnaire

Cognitiveinterview


Balancing the evidence “Stronger data can be given more weight in

the conclusion.” (Miles/Huberman 1994)

Safeguards: Make a note of cases with poor data quality Remember theses cases during data analysis Exclude these cases from the final report, if

necessary


Contents

Pretesting in official statistics

Selected results

Future prospects


Future prospects Quality standards for qualitative pretesting

(e. g. checklists) Online questionnaires Business statistics

Elaborated guidelines for cognitive interviewing

Exchange of experience between statistical offices


Thank you for your attention.

[email protected]

[email protected]

© federal statistical office (fso), institute for research and development in federal statistics...

Documents