1

Culture and

Survey Measurement

Timothy Johnson

Survey Research Laboratory

University of Illinois at Chicago

2

What is Culture?

the “shared elements that provide the standards for perceiving, believing, evaluating, communicating, and acting among those who share a language, a historic period, and a geographic location” (Triandis, American Psychologist, 1996).

3

Unpackaging Culture

Research has focused on racial, ethnic, or cross-national differences

With few exceptions, mechanisms responsible for these identity group differences have been unexplored

4

Some Models of Culture

Hofstede (2001)

Schwartz (1992)

Triandis (1996)

Ingelhart (1997)

Trompennars and Hampen-Turner (1998)

5

Geert Hofstede:

Culture’s Consequences

Individualism-Collectivism

Power Distance

Uncertainty Avoidance

Masculinity and Femininity

Long- vs. Short-Term Orientations

6

Individualism vs. Collectivism

Self identity and personal goals vs. norms, obligations and duties

In-groups vs. out-groups

Individualists more commonly make cost-benefit decisions

Collectivists more likely to monitor other’s behaviors and feelings

7 8

Individualist

9

Individualist Collectivist

10

Shalom Schwartz:

Cultural Value Orientations

Identifies 3 bipolar dimensions of culture; each represents an alternative resolution to problems that confront all societies:

Embeddedness vs. autonomy

Hierarchy vs. egalitarianism

Mastery vs. harmony

11 12

Harry Triandis:

Cultural Tightness

Reflects emphasis on heterogeneity and surveillance

13

Horizontal vs. Vertical

Social Structures

Vertical cultures

Emphasize social hierarchies

Horizontal cultures

Emphasize egalitarianism

14

Other Dimensions of Culture

Analytic-linear vs. holistic-dialectical cognitive styles

Communication processesContext requirements

Nonverbal behavior

Self-disclosure

Social participationHistorical experience

Social distance

15

1. Reliability

2. Validity

3. Equivalence

Elements of Social Measurement

in Cross-Cultural Research

16

1. Calibration

2. Category

3. Complete

4. Conceptual

5. Construct

6. Construct operationalization

7. Content

8. Contextual

9. Credible

10. Criterion

11. Cross-cultural

12. Cross-language

13. Cross-level

14. Cross-national

15. Cultural

16. Data

17. Definitional

18. Direct

19. Ethnographic

20. Exact

21. Experiential

22. External

23. Factor

24. Factorial

25. False

26. Formal

27. Foreign language

28. Full

29. Full measurement

30. Functional

31. Grammatical-syntactical

32. Group

33. Idiomatic

34. Interpretive

35. Indicator

36. Instrument

37. Item

38. Language

39. Lexical

40. Linguistic

41. Literal

42. Logical

43. Meaning

44. Measure

45. Measure equivalence

46. Measurement equivalence

47. Measurement unit

48. Metaphorical

49. Metric

50. Motivational

51. Normative

52. Operational

53. Operationalization

54. Partial

55. Procedural

56. Pseudo

57. Psychological

58. Psychometric

59. Relational

60. Relative

61. Response

61. Response

62. Rough

63. Sampling

64. Scale

65. Scalar

66. Scalar/metric

67. Situational

68. Stimulus

69. Structural

70. Substantive

71. Syntactic

72. Technical

73. Text

74. Theoretical

75. Translation

76. Translational/ Linguistic

77. True-score

78. Verbal

79. Vignette

80. Vocabulary

Types of Equivalence

17

Question Comprehension/

Interpretation

Emic (culture specific)

Etic (pancultural)

18

Category Fallacy

Assuming a question or concept is universally understood when in fact understanding is culturally conditioned

19

The meaning of words may

evolve faster than most other

aspects of culture.

Triandis (2004) chapter in Comparing Cultures, Dimensions of Culture in a Comparative Perspective

20

“In this question, what does the

word ‘stress’ mean to you?”

0

5

10

15

20

25

30

African

American

Mexican

American

Puerto

Rican

White

Health Problems

Social Problems

21

Probability of Comprehension Difficulty by

Race/Ethnicity

0.066

0.086*0.099*** 0.093**

0.00

0.02

0.04

0.06

0.08

0.10

0.12

White African American Mexican

American

Puerto Rican

Race/Ethnicity

Pro

ba

bil

ity

of

Co

mp

reh

en

sio

n D

iffi

cu

lty

* p<.05; **p<.01; ***p<.001 significantly different from White

22

Memory Retrieval

Episodic vs. semantic search strategies

Memory cues

23

Ji, Schwarz & Nisbett (2000)

Collectivist societies attend more closely to the behavior of others, resulting in memories for behaviors of others that Americans can only estimate

24

Accessibility

Anchoring

Response formatting

Response styles

Judgment Formation/Mapping

25

Comprehension Difficulty by Response Format and

Ethnicity

0.00

0.05

0.10

0.15

0.20

0.25

White African

American

Mexican

American

Puerto Rican

Race/Ethnicity

Pro

bab

ilit

y o

f

Co

mp

reh

en

sio

n

Dif

ficu

ltie

s

Yes-No Vague Quantifier Numeric

26

Extreme response styles

Acquiescence

Non-differentiation

Measurement Artifacts in

Survey Research

27

Self presentation

Social desirability

Interviewer effects

Response Editing

28

% Uncomfortable Discussing Alcohol Use

with Interviewers from Same/Different

Cultural Groups

0

5

10

15

20

25

30

35

40

African

American

Mexican

American

Puerto

Rican

White

Same Culture

Different Culture

29

A. Question Development Phase

B. Questionnaire Pretesting Phase

C. Data Collection Phase

D. Data Analysis Phase

Available Methods for Addressing

Cross-Cultural Equivalence

30

Question Development Stage

1. Expert consultation/collaboration

2. Ethnographic and other qualitative approaches

3. “Good” question-wording practices

4. “Good” translation practices

31

Good Question Wording Practices

Avoid vague quantifiers

Avoid items with ambiguous or dual meanings

Avoid hypothetical questions

Use simple terms that are similarly understood

Use clear and precise time references

Avoid questions with highly abstract concepts

See also recommendations by Brislin (1986)

32

Other Strategies

Use multiple indicators

Use both etic and emic items

Use KISS method: dichotomous response options

Likely results in loss of precision

Substitute numerical or other nonverbal scales, although:

Many numeric scales are also difficult to use

Numeric scales also not invariant in meaning across groups

Beware of lucky and unlucky numbers

Generally do not completely eliminate word use (e.g., semantic differentials still label end-points)

Alternative numbering schemes can influence reporting

Translation: When things go wrong…

• Pepsi's "Come Alive With the Pepsi Generation" translated into "Pepsi Brings Your Ancestors Back From the Grave" in Chinese.

• Frank Perdue's chicken slogan, "It takes a strong man to make a tender chicken" was translated into Spanish as "it takes an aroused man to make a chicken affectionate."

33

Traditional Approach:

Simple Direct Translation

• Simple, unidirectional translation of survey instrument from a source language into one or more target languages.

• a.k.a., one-way translation

• Now considered an unacceptable approach• Not considered informative because it does not produce objective information about translation quality or specific problems

• A commonly referenced alternative is the Back-Translation procedure

Back-Translation

• a.k.a., double translation

• Described by Brislin (1970) – heavily cited

• Basic procedure calls for a bilingual person to translate a source questionnaire into a target language

• A second bilingual person translates the translated version back into the source language without knowledge of the original instrument.

• The initial and revised versions of the source language version are then compared, discrepancies are identified, and final revisions are made (second round sometimes necessary)

Terms that presented difficulty in translation (Weidmer et al., 1999)

Original EnglishAlternative wording used in final Spanish

version

Back-Translation

Health insurance plan Plan de seguro médico Medical insurance plan

Health provider Profesional de salud Health professional

Rating/rate Calificación/califica Grade/grade

Usually Normalmente Normally

Preventive health steps Medidas de salud preventiva

Preventive health measures

Listen carefully Escucharon atentamente Listen attentively

Health care Altencón médica Medical attention

Prescription medicine Medicamentos recetados Prescribed medicaments

Male or female Niňo o niňa/hombre o mujer

Boy or girl/man or woman

Background Ascendencia Ascendancy

Grade Aňo Year

School Estudios studies36

Alternatives to Back-Translation

• Modified direct translation

• Modified back-translation strategies

• Parallel blind technique

• De-centering strategy

• “Triandis” procedure

• Translation by committee

Current Good Practice for Translation (Harkness, 2008)

• Range of expertise needed: survey design, substantive understanding of content, source/target language competence, knowledge of local fielding situation.

• Team approaches.

• Translation teams should include (1) those who translate, (2) those who review translations, (3) and those who take final decisions on versions (adjudicators).

• Translators should be skilled practitioners who have received training and should normally translate out of the source language into their strongest language.

• Is better to use several translators

• Where possible, each translator should make a draft translation.The alternative is to have each translator do a section.

39

Questionnaire Pretesting Phase

1. Cognitive interviews/structured probes

2. Comparative response scale calibration

3. Comparative behavior coding

4. Compare alternative data collection modes

5. Use of comparative vignettes

40

Behavior Codes Used to Identify

Comprehension Problems

Clarification (unspecified): uncertainty about question, unclear if problem is related to construct or context.

Clarification (construct): request for repeat or clarification of question, or statement indicating uncertainty about question meaning

Clarification (context): uncertainty about question meaning within the context of the question as stated

Clarification (time frame): uncertainty regarding question time frame.

Clarification (rewording): rephrases question before answering.

41

Data Collection Phase

1. Use multiple indicators

2. Use both emic and etic questions

3. Respondent/interviewer matching

42

Data Analysis Phase

Common Goals of Cross-Cultural Analyses:

• Verify interpretive equivalence of measures

• Conduct substantive analyses using equivalent measures and procedures

43

Classification of Analysis Strategies by

Sophistication of Statistical Technique

Elementary and easy-to-use techniques

Less elementary but still easy-to-use techniques

Advanced techniques requiring some expertise and effort

44

Elementary and Easy-to-Use Techniques

Comparison of means (or some other methods of central tendency) of different items across countries

Comparison of correlations between the items under investigation with measures of the underlying dimension or with a benchmark items that can be assumed to represent the dimension

Comparison of correlations between the items under investigation with external variables which are assumed to influence them (or are influenced by them)

45

Less Elementary But Still Easy-to-Use

Techniques

Traditional Item Analysis

Exploratory Factor Analysis

Correspondence Analysis

Multidimensional Scaling

Applying Statistical Controls

46

Advanced Techniques Requiring Some

Expertise and Effort

Multilevel (Hierarchical Linear) Modeling

Multiple group confirmatory factor analysis

Structural equation models

Item-Response models

Multi-Trait Multi-Method analyses

47

Multiple Group Confirmatory Factor

Analysis (CFA)

Can be used to compare factor structure of sets of survey questions across multiple population groups

Several advantages over exploratory factor analysis:

CFA enables overall assessment of model fit, and assessment of the cross-group equivalence of individual items.

48

Structural Equation Models

While confirmatory factor analysis focuses on the measurement model, structural equation modeling considers also the structural model

Can evaluate similarities and differences in theoretical models across multiple cultural groups

49

Understanding Probe

ComprehensionDifficulties

Answering

Probe

Memories

Probe

Certainty

Probe

Question Threat Probe

Embarrassment

Probe

MemoryDifficulties

Social

DesirabilityConcerns

Understanding Probe

ComprehensionDifficulties

Answering

Probe

Memories

Probe

Certainty

Probe

Question Threat Probe

Embarrassment

Probe

MemoryDifficulties

Social

DesirabilityConcerns

Understanding Probe

ComprehensionDifficulties

Answering

Probe

Memories

Probe

Certainty

Probe

Question Threat Probe

Embarrassment

Probe

MemoryDifficulties

Social

DesirabilityConcerns

WHITE AFRICAN AMERICAN LATINO

50

Multilevel (Hierarchical Linear)

Modeling

Permits analyses of individuals nested within cultural groups

Effects on dependent variables of measures assessed at multiple levels of analysis (i.e., respondent, culture, question) can be examined

Recent book by van de Vijver et al.

51

HLM Regression Model Estimates of Fixed Effects for Comprehension and Mapping Difficulties

Comprehension Difficulties Mapping Difficulties

Equation 1 Equation 2 Equation 3 Equation 4 Fixed Effect Coeff. SE Coeff. SE Coeff. SE Coeff. SE

Comprehension Difficulty 0.53** 0.09 Mapping Difficulty 0.52** 0.09 Question characteristics Question Length 0.0004 0.02 -0.0003 0.02 0.10** 0.03 0.10** 0.03 Question Length Squared 0.0001 0.0003 0.0001 0.0003 -0.002** 0.001 -0.002** 0.001 Reading Level 0.45** 0.06 0.43** 0.06 0.18** 0.06 0.16** 0.06 Reading Level Squared -0.02** 0.004 -0.02** 0.004 -0.01 0.004 -0.005 0.004 Abstraction Level Ref=(Concrete)

Most Abstract 0.71** 0.12 0.71** 0.12 0.03 0.15 -0.01 0.14 Somewhat Abstract 0.11 0.08 0.11 0.08 -0.01 0.08 -0.02 0.08 Response Format Ref=(Numerical Open-ended)

Yes/No -1.26** 0.10 -1.11** 0.10 -3.89** 0.22 -3.83** 0.22 Verbal labels -0.99** 0.10 -0.87** 0.10 -1.94** 0.12 -1.89** 0.13 Qualified Judgment 0.28** 0.08 0.27** 0.08 0.20* 0.09 0.19* 0.09 Respondent Characteristics Age 0.01+ 0.01 0.01 0.01 0.02** 0.004 0.02** 0.004 Education -0.05 0.04 -0.05 0.04 -0.03 0.03 -0.03 0.03 Race/Ethnicity (Ref=White) African American 0.31** 0.12 0.31** 0.12 0.01 0.11 0.001 0.11 Mexican American 0.50** 0.12 0.50** 0.12 0.07 0.12 0.04 0.11 Puerto Rican American 0.40** 0.14 0.41** 0.14 0.05 0.11 0.03 0.11 Gender (Male) 0.05 0.09 0.05 0.09 0.13 0.08 0.12 0.08 Intercept -4.28** 0.35 -4.30** 0.35 -3.72** 0.31 -3.69** 0.31

**p<.01 *p<.05 +p<.10 52

Preliminary Conclusions

1. Culture matters.

2. Should assume variability until proven otherwise.

3. Consider how measurement variability might influence results.

4. There are many tools now available to address this problem prospectively.

5. But, no magic solutions.

Thank You.

timj@uic.edu

53