How do we assess whether we are improving instrument design?

Alice McGee

How do we assess whether we are improving instrument design?

Using multiple methods to evaluate whether a re-designed travel record

was 'better' than the existing one

Presentation outline

• Background to NTS and Travel Record• Rationale for re-design• Research stages

A - Evaluate the existing recordB - Record re-designC - Evaluate the new record

• Findings• Conclusions

Background to study

• National Travel Survey (NTS)• Measures travel patterns and behaviour

Face to face interview - household levelSeven-day Travel Record - individual level

Adult and child versionsNature of information - very detailed

Existing Travel Record

Rationale

• Concerns surrounding data qualityHigh proportions of item missingAnecdotal evidence suggesting measurement

error• International evidence supporting alternative

designsAxhausen:

Column based approach

Research stages

• A - Evaluate the existing record• B - Re-design the record• C - Evaluate the new record

to assess whether the new was ‘better’ than the original

A - Evaluation methods

Quantitative

1. Secondary analysis of survey data

Qualitative

2. Workshops with interviewers

3. Cognitive interviews4. Respondent suggestions5. Data editor comments

B - Re-designing the record

• Two underpinning elementsExisting literature: theoretical basis for how

people approach self-completion documentsEmpirical evidence from Stage A

• Existing literatureJenkins et al:

Readers and Skimmers typologyPrinciples of good questionnaire design

B - Aims of the re-design

• Make the task easier and more straightforward for all types of respondent through:providing navigational guides to help

respondents find their way around the recordimprove the way information was organised and

displayed• NB: unable to address the more conceptual

problems

B - Main changes to record

• Colour and layout more user-friendly• Instructions brought together• Visual guides (arrows) and clearer signposting• Pictorial images• Tick boxes• Column heading posed as questions

Re-designed Travel Record

C - Evaluation methods

Quantitative

1. Comparison of sample of data from existing Record and pilot data (100 respondents)

2. Respondent debriefing questionnaires

Qualitative

3. Cognitive interviews4. Pilot interviewer

comments5. Data editor comments

Hypotheses

• All evidence sources reviewed and triangulated• Hypotheses tested at Stage C:

Cognitive respondents would experience narrower range of problems

Problems relating to information organisation and navigation would diminish among cognitive respondents

Systematic review of respondent error would show the new Record produced lower error rates

Findings and conclusions

• All three hypotheses proved correct• Triangulation of quantitative and qualitative

evaluation methods showed new Record to be ‘better’ than the original

• Study demonstrates how a combination of evaluation methods can be brought together to substantively conclude the best approach

Re-designed Travel Record

Future work

• Evaluation of Event History Calendar approach (EHC)England Longitudinal Study of Ageing (ELSA)Retrospective interview following Wave 3Interactive calendar administered through CAPI

• Package of work involving multiple evaluation methods (quantitative and qualitative)

EHC - Evaluation methods

Quantitative

1. Behaviour coding 2. Secondary analysis

of survey data3. Respondent

debriefing questionnaires (closed responses)

Qualitative4. Interviewer diary5. Respondent

debriefing questionnaires (open responses)

6. Quasi transcripts7. Behaviour coding

(supporting notes)

References

• Axhausen, K (1995) “Travel Diaries: An annotated catalogue, 2nd edition”

• Jenkins, C.R., Ciochetto, S. and Davies, W. (1992) 'Results of cognitive research on the public school 1991-92 field test questionnaire for the schools and staffing survey'. Unpublished

• Jenkins, C.R. and Dillman, D.A. (1995) 'Towards a theory of self-administered questionnaire design' in Lyberg et al (1997) 'Survey Measurement and Process Quality' Wiley

• McGee, A, Gray, M and Collins, D (2006) “NTS Travel Record Review Stage 1” DfT, web only

• McGee, A, Gray, M, Andrews, F, Legard, R, Wood, N and Collins, D (2006) “NTS Travel Record Review Stage 2” DfT, web only