translation of survey instruments alisú schoua-glusberg, ph.d. research support services dc/aapor...

Translation of survey instruments

Alisú Schoua-Glusberg, Ph.D.Research Support Services

DC/AAPOR Presentation – 7/7/04

Research Support Services - 2004

2

Translation methods for survey instruments

This presentation will focus on:

problems involved in questionnaire translation methods and approaches for the translation of survey

instruments,

use of qualitative methods to assess translation quality and the performance of the translated instrument,

providing survey translators the information they need.


3

When is Questionnaire Translation Necessary?

Cross-national studies

Cross-cultural studies

National studies by Federal Statistical Agencies

Studies of special populations

Studies where sample falls in areas with significant concentrations of speakers of other languages


4

Factors to Consider to Define the Target Audience of Translated Instruments Language minorities Immigrant populations Monolingual (or at least not

bilingual) Language different from designers’ Lower level of education Age at immigration Hybrid culture Different degrees of acculturation


5

Source & Target Question Characteristics to Consider

Meaning Style Complexity Source flexibility Cultural aspects Existence of equivalent realities


6

Using Previous Translations of Questions vs. Translating Anew

Issues:

Maintaining longitudinal comparability vs. improving questions

Is priority to compare with past translations or with English questionnaire?


7

Questionnaires are a Complex Text Type

Little context to clarify ‘intended meaning’

Ambiguous

Measurement properties lead to ‘surveyspeak’ and ‘scalespeak’ (Harkness 1996)

Translators thus often not versed in questionnaire discourse ‘rules’

Special kind of conversation


8

Question ‘Equivalence’ is Expected in Translation

Expectation that the translated question… Says the same thing Means the same thing Measures the same thing Measures equivalently (e.g. scales) Imposes same burden on respondents Meets reliability and validity requirements


9

Translators’ Job

Translators make decisions about: Semantics Syntax (structure) Pragmatics

They need to understand: Intended “meaning” in order to translate Covert (measurement) intention and

requirements


10

Translators therefore…

Have little context and co-text

Have little supporting documentation

Often have no questionnaire ‘author’ to consult

May experience uncertainly about what to “match”

May make decisions based on their experience with other types of text


11

Risks of Overly Close Translations Focus on meaning of words rather than

meaning of questions

Rs are inadvertently asked a different question

Processing is more complex

Translated questionnaire sounds unidiomatic


12

Steps in producing and testing quex translations Translation Translation review Decisions/Adjudication Quality control Qualitative research Pretesting Documentation


13

Survey Translation Approaches One translator - one translation

(direct translation)

Multiple translators – one translation (split committee)

Multiple translators – multiple parallel translations (parallel committee)


14

Committee Approach

Three translators prepare translations independently (split or parallel)

Reconciliation meeting with referee

Qualitative research with monolinguals (focus groups and/or cognitive interviews)


15

Committee Approach: Reconciliation Meeting Question-by-question review

Reaching consensus when possible

Providing alternatives if no consensus possible

Identifying terms/items for qualitative research


16

Committee Approach: Referee’s Role

Resolves style disagreements

Manages interaction

Brings survey researcher perspective

Keeps an eye on source version

Pushes for global decisions

Liaison with research team for consultation and documentation


17

Committee Approach: Cognitive Interviews Allow to administer all or part of

instrument

Give a glimpse into thought processes

Allow to see how different alternative terms work


18

Committee Approach: Focus Groups

Permit to distinguish what is idiosyncratic

Allow us to listen to how Rs. use language

Allow to include more people in a shorter time

Allow to include different national origin Rs and see if they reach consensus


19

Advantages of Committee/ Team Approaches

Group process benefits

Include different varieties of language in translation team

Qualitative research that follows allows to incorporate the target population into process

Relatively low cost

Relatively quick


20

Selected Committee Translations: 2000-2004

NSFG (U. Mich.) ATUS (BLS) NHSDA (RTI) ICARIS (Battelle) NMHS (Battelle) SLAITS Asthma Q. (Abt) Job Corps Student Q. (Battelle) Survey of Consumer Attitudes (U.

Mich.) Catholic Voters Political Attitudes

(BRS) McNair Program Evaluation (DIR) L.A. Latino Eye Study (USC)

Women’s Health Initiative Q. (U. Mich.)

WHO Health & Performance Q. (Harvard)

PHDCN (Harvard) Project Bread (UMass-Boston) California Safe Schools

(RAND) CAHPS Dialysis Center

(RAND) National Children’s Study Pilot

(Battelle) Survey of Bioterrorism Pre-

paredness (N.Y. Academy of Medicine)


21

Textual assessments Translation appraisalHolistic approaches, e.g., TRAPD, committeeBack translation

Pretesting with bilinguals, e.g.,SplitsDouble administrations

Debriefings Probe interviews Think alouds with Respondent or Translator Focus groups with sample population

Assessing Translations: some procedures(Harkness, Pennell, Schoua-Glusberg – ASA, 2003)


22

Backtranslation

One person translates from source into target language.

A second person translates the target language version back into the source language.

A third person compares the original and the backtranslated source language versions.

Discrepancies are investigated.


23

What is wrong with backtranslation?

It is a black box: we know what went in, we know what came out, yet we know nothing about the adequacy of the target language version.


24

Why give Q-by-Q specs to translators?

In the absence of question-by-question specifications, translators make their own decisions…

to resolve ambiguities to figure what “they” mean

These are most often not documented or even explicit. They happen in the translator’s mind.


25

Research Conducted

Goal: to examine translators’ decision processes

Vehicle: Recorded a translators’ Committee meeting

Method: Listened to tape searching for decisions made during discussions


26

Why this design?

Committee discussion involves:

–Review

–Translation–Adjudication

Makes decisions verbally explicit


27

Transcription Detail:

2 hours of committee meeting

11 pages of questionnaire

110 questions


28

Example 1

How would your parents feel if they found out you drank alcohol sometimes?

Not at all upset

A little upset

Pretty upset Very upset


29

Discussion about “upset”Did “they” mean…

Angry?

Perturbed / Bothered?

Given context, committee decided to translate upset as angry.


30

Example 2

I think sometimes it’s okay to cheat at school.

Strongly Agree

Agree

Disagree

Strongly Disagree


31

Discussion about ‘cheat’Original Item: I think sometimes it’s okay to cheat at school.

First translation:

Creo que a veces está bien hacer trampa en la escuela.

Referee: “¿Hacer trampa? Everyone agrees?”

Orig. Tr.: To me it refers to copying in exams.

Tr2: “You can also copy when you are doing homework or other schoolwork, that is cheating too, not only in exams.” “And you could cheat in sports, so we need to be specific. In this case this refers to school work, not to playing volleyball.”

Resolution: Creo que a veces está bien copiar o hacer trampa en mis exámenes y tareas escolares.

(I think sometimes it’s okay to copy or cheat in my exams and schoolwork.)


32

Example 3

How do you feel about your ability to care for your teen when they are sick or upset?

Discussion about ‘upset’.

In this case, the discussion centered around whether ‘upset’ meant angry, not feeling well, or bothered by something.

The committee went with the latter option.(‘upset’: molesto)


33

Example 4How do you feel about your ability to discipline your

teen?

Discussion about ‘discipline’. Do “they” mean:• Punish?• Teach them to behave properly? • Set rules for them?

Checked with client who opted for ‘punish’.


34

Example 5

How do you feel about your ability to obtain needed resources for your teen?

Discussion about ‘resources’. Did “they” mean...

Material resources?Assistance/help?

Client asked to preserve the ambiguity as much as possible. Translated as ‘recursos’.


35

Example 6

During the past week, how often did you let this teen know you really care about him/her?

Discussion about ‘care’. Did “they” mean…love?concern?

Committee decide to include both:… how much you love him/her and are concerned about him/her


36

Providing Specifications:Ideal ModelThree steps:

1. Deliver question-by-question specifications

2. Review original text with translation team member

3. Ask translators to make their decisions explicit and submit them for review.


37

Format for Documentation

Q# Original Q. Issue D/QSpecify decision or

question


38

Recent/Current Efforts to Standardize Translation Procedures

U.S. Census Bureau translation guidelines

International Social Survey Programme’s translation methods’ work

European Social Survey Implemented Procedures in 20+ countries

European Social Survey and Survey of Health, Aging and Retirement in Europe analyzing translation and documentation outputs

CAHPS Cultural Comparability Task Force

NCHS Translation Issues Forum

CSDI (Comparative Survey Design and Implementation Translation Task Force)

translation of survey instruments alisú schoua-glusberg, ph.d. research support services dc/aapor...

Documents

survey translators

translation quality

translators jobtranslators

questionnaire author

english questionnaire

thing measures

past translations

qualitative methods