Cultural Biases in Surveys

Sarah MillerMarch 12, 2013

Our QuestionsJapan - Are they always going to say “yes”?Brazil - Are Brazilians just happy?France - Is anything good enough for the French?Germany - Are the Germans just not fun?

Why are your survey responses for Germany & France so different from Brazil & Japan? Is your

survey adapted for culture?

How are they responding currently?

Japan (-)Brazil (+)France (negative)Germany (negative)

Maybe they have one or more of these biases

Acquiescence Response Style (ARS)Where is it a problem?

Agree/Disagree questions

Middle Response Style (MRS)Extreme Response Style (ERS)Response style bias: is there a tendency to

respond to survey questions in a particular way, regardless of content?

ARSAcquiescence Response Style (ARS)

Where is it Prevalent?Cultures where being polite is importantCan we pinpoint the issue?Agree/Disagree questions are tough

Some cultures interpret these questions as a double negatives (agreeing with something negative is doubly negative)

They may not fully disagree or agree

MRSMiddle Response Style (MRS)

Where are MRS prevalent? Cultures where extremes are discouraged: Asian CulturesCan we pinpoint the issue?

Confucious: No one answer is absoluteCollectivism: Avoid making others look bad

ERSExtreme Response StylesWhere are ERS prevalent?

In cultures that value individualism, are masculine, and have a low tolerance for ambiguity

Can we pinpoint the issue?

Individualistic, masculine cultures put a value on assertiveness and decisiveness

Geert wants to remind us…

Power Distance (PDI)Individualism (IDV)Uncertainty Avoidance (UA)Masculinity (MAS)Long Term Orientation (LTO)

Response biases are related to all of these orientations

Brazil: High Acquiescence Highest acquiescence cultures…and may have a bit of SDRWhy?

High UA & Low IDVCollectivism (low IDV)

Expected behaviors More collectivist than Japan!

Aversion to confrontation Politeness

UA - 71IDV - 38

Japan is a Middle Response CultureWhy?

Collectivism Personal opinions are not

strongly voiced Value harmony

Dialectical Thinkers No one, absolute truth

Long Term Orientation LTO societies believe that

truth depends on the situation

Germans are well balanced, but can be a

little negativeWhy?

DRS or “disacquiescence” is associated with low power distance and high individualism. Germany is be a prime example of this type of culture.

High masculinity and UA are also associated negatively to acquiescence.

PDI: 35 IDV: 67 MAS: 66 UA: 65

The French are Extremists!

The French are the 3rd most extreme culture in Europe.

Why?Rules (UA)Societies higher in UA have many rules and have little tolerance for ambiguity.Direct Communication (IDV) Individualistic societies are

characterized by self-expression, independence, and personal opinions.

Individual contributions expected, speaking up appreciated – disagreement is normal.

Individualistic societies have absolute truths

UA: 86 IDV: 71

"You Can't Judge a Book by its Cover", Arun Joshi, Sagar Tamang, Himanshu Vashishtha, Research World, January 2009 International Marketing Research By C. Samuel Craig, Susan P. Douglas

SolutionsLinkert 5 point scales are not the best solution across these response styles & cultures

How to balance acquiescent culturesWhat’s the Solution?

Give various levels of agreement and disagreementReverse order points of scaleEmploy negatively worded statements vs positive

Or don’t use obvious highs/lows or negatives/positives

Use visual stimuli/pictures vs words or numbersTest for acquiescence using a behavioral variable Joshi-Tamang-Vashishtha Scale

Acquiescence is closely related to Social Desireability – lower class people will give responses that reflects the values of the interviewer in face to face interviews

Solutions for MRS CulturesWhat’s the Solution?Kill the middle response!Use a multi-item scale

Solutions for the Extreme Response Styles

What’s the solution?

Survey DevelopmentSurvey TypeItem GenerationSurvey Languages

Survey TypeNot all surveys are transferableScale development is important

Measurement scales need to be investigated

Item GenerationDoes the survey rely on commonalities or distinctions?

Emic or Etic? Applicable universal concepts, values, truths

Frame of reference - not just language - should be conceptually equivalent

Suggestion

Qualitative data (interviews) can be helpful, especially if taking surveys from multiple countries at once

The Role of LanguageEnglish language by default elicits middle responses

Opposites in one language may not be opposite in another language

Double negatives!

Intensity levels or words could be misinterpretedDirect translation isn’t always the key

Scale anchors may not translate well

Harzing, 2006

The Role of LanguageLanguage impacts surveys!

Cultural accommodation Symbolic effect

Concepts and terms Interpretation – culture specific connotations

Researcher needs cultural knowledge as well as language knowledge

Use clear, simple, and familiar wording

Use natives in country where surveying Careful translation + pilot testing with native speakers

Other SolutionsMixture of Positive & Negative Statements

Mitigates acquiescence and disaquiescence

Linkert scales, use more than 5! Reduces response and language bias Scale anchors that have mutually exclusive constructs rather

than agreement level also reduce biases*

Do not compare loyalty scores across countries, instead compare loyalty scores across companies in the same country!

Less complex than statistical adjustments and probably more accurate

NOT Good SolutionsSome organizations try to eliminate the influence of

response bias by calculating a cross-cultural adjustment factor for each country or region in which they do business.

The adjustment factor is calculated as the ratio of the average score for a comparison country over the average score for a base country, which is typically the organization’s largest, most important market.

In practice, however, this method is ill-advised unless the adjustment procedure truly isolates opinion differences from differences due to cultural response bias

RememberThe researcher needs to take culture into consideration in the overall design, language, and item generation of the survey.Benchmarks should be within the same countryMore points on the Linkert scaleCross-country comparison helps to identify

survey issuesNative/local speakers can help with design