cognitive testing of survey translations: does respondent ... · cognitive testing: one-on-one...
TRANSCRIPT
Cognitive Testing of Survey
Translations: Does Respondent
Language Proficiency Matter?
Patricia Goerman and Mikelyn Meyers, U.S. Census BureauHyunjoo Park and Mandy Sha, RTI International
Alisu Schoua Glusberg, RSS
Presented at the 70th annual conference of the American Association for Public Opinion Research (AAPOR)
Hollywood, FL: May 14-17, 2015
1
Overview of the Issue
Rule of thumb: survey translations should be tested with monolingual respondents
Assumptions:
Monolinguals are the intended users
Bilinguals maybe more likely to understand “bad” or overly literal translations:
For example: group home (hogar de grupo)
home schooling (enseñanza en el hogar)
2
Why Does it Matter?
More costly and time consuming to restrict testing to only monolingual respondents
Difficulties with recruiting/interviewing monolinguals
Distrust/lack of understanding of purpose
Correlation with lower income status
3
Review of the Literature
Survey Pretesting with monolinguals v. bilinguals -very little empirical research
(Park, et al. 2014)
Monolinguals: Difficulty with cognitive interview process (Park, et al. 2013; Pan et al. 2010; Goerman, 2006)
Differences between monolinguals and bilinguals in cognitive processes
(Bialystok, 2007; Hernandez and Bates 2001; Marian et al. 2009)
Monolingual status may be a correlate of other variables, e.g., educational attainment
(Ridolfo and Schoua Glusberg, 2011)
Important to include bilinguals in testing to account for differences in acculturation and other demographic characteristics
(Willis and Zahnd 2007)
4
Definitions
Cognitive testing: One-on-one interviews to evaluate whether respondents interpret, comprehend and respond to survey questions as intended.
Usability testing: One-on-one interview to study whether online questionnaire can be answered effectively, efficiently and with satisfaction by target respondents (Wang, 2015)
5
2014 Census Test Spanish Internet
Cognitive/Usability Testing
Origin of present study
Quick turnaround project: Test Spanish-language Census 2014 test instrument with 10 Spanish speakers
Recruitment goals Different national origins
Internet experience
Monolingual or Spanish-dominant, bilingual respondents to best test the translation
6
Preliminary Findings/ Impressions
Correlation between bilingual ability, education level and internet/computer experience
Accidental recruitment:
Non-computer literate, monolingual
Computer savvy, fluent English speakers
Issues of mode interacting with other respondent characteristics. How does language proficiency fit in with other considerations?
7
Design of Empirical Research
Decennial Census testing across modes and languages, large ongoing contract
Double the number of respondents so that half are monolingual, half bilingual
Spanish testing:
Paper, internet, interviewer administered form
Chinese, Korean, Vietnamese, Russian, Arabic Fillable forms: internet/paper, self administered mode
8
Research Questions for Overall
Project
Do monolingual and bilingual respondents help to “uncover” same number and types of issues?
Are there differences by mode?
Are there differences by language?
9
Early Results: Spanish, Interviewer-
Administered Instrument
Respondents asked about speaking, reading ability, dominant language in screening
39 Spanish-speaking respondents
19 monolingual
20 bilingual
Coding of interview summaries (3 coders)
Whether probe administered
Whether understood concept as intended
10
Respondent CharacteristicsDemographic Characteristics Monolinguals Bilinguals
GenderMale 9 8
Female 10 12
Age
18-34 6 9
35-44 3 6
45-54 8 3
44-64 1 2
65+ 1 0
Education*
Less than HS 10 0
HS grad/GED 7 16
College + 1 4
Years in US**
Less than 5 7 4
6-15 years 10 7
16+ 2 6
*Education level was missing for 1 monolingual respondent**Years in US was missing for 3 bilingual respondents
11
Results: Concepts Monolinguals
Misunderstood More Frequently
12
Concept
Monolinguals Bilinguals
% Misunderstood N % Misunderstood N
Foster child 94% 18 53% 19
Military assignment 60% 10 25% 12
Afroamericano 47% 15 19% 16
Own with mortgage 41% 17 15% 20
Live/stay somewhere else 21% 14 0% 15
Group home 87% 15 80% 10
Owner/renter name 40% 15 36% 11
% Misunderstood is calculated as : # who misunderstood a# who were asked probe (N)
Concepts Bilinguals
Misunderstood More Frequently
13
Concept
Monolinguals Bilinguals
% Misunderstood N % Misunderstood N
Housemate/roommate 43% 14 73% 15
Own free and clear 0% 18 20% 20
Seasonal home 50% 14 55% 11
Tenure question 16% 19 21% 19
Confidentiality 17% 6 21% 14
Indigena de las americas 50% 14 54% 13
Understanding of “Foster Child”
by English-Speaking Ability
10%
63%
90%
37%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Understood, N = 10* Misunderstood, N = 27
Speak English well /very wellSpeak English not well /not at all
14
*Data regarding understanding of this concept were unavailable for 2 respondents out of 39
NOTE: “Foster child” was translated as “Hijo(a) de crianza del programa Foster del gobierno”
Summary of Findings
Bilinguals caught most of the same problems as monolinguals but often less frequently
Depending on the size of a bilingual sample, researchers may or may not see an issue
One term was more problematic for bilinguals than monolinguals (Free and clear). Testing with only monolinguals may have masked this issue
Including only bilinguals in interviewer administered (CAPI) testing may work as a cost saving measure but some issues maybe lost
15
Next Steps
Calculate intercoder reliability and recode as necessary
More analysis of nuances in the types of problems identified
Analysis of additional modes in Spanish: Internet (data collected)
Paper questionnaires (pending funding)
Examination of monolingual v. bilingual respondents in 5 additional languages (pending funding)
Add analysis of English speakers for comparison
Look at “levels” of bilingualism
16
Cognitive Testing of Survey
Translations: Does Respondent
Language Proficiency Matter?
Patricia Goerman and Mikelyn Meyers, U.S. Census BureauHyunjoo Park and Mandy Sha, RTI International
Alisu Schoua Glusberg, RSS
For more information: E-mail: [email protected]
Disclaimer: This presentation is intended to inform people about research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau.
17