can surveys results be compared across countries? · survey methodology and cross-national...

Can surveys resultsbe compared acrosscountries?

What we know andwhat we don’t know

Ineke Stoop

Yes

Lots andlots

Ineke Stoop

No

Lots and not a greatdeal

Ineke Stoop

BaNoCoss 2011

Overview1. Background2. European Social Survey3. What we aim for and how4. Why and how we fail5. What we do to try to make up6. Lessons for national surveys

BACKGROUND

BaNoCoss 2011

What about me?

BaNoCoss 2011

The Netherlands Institute for Social Research• Governmental, independent,

policy related social researchagency

• Data users, not datacollectors

• 70 substantive researchers(total 100)

• Small data services unit

Data fromStatistics Netherlands

Surveys, registersEurostatInternational dataData archive(co-)sponsor of surveys

BaNoCoss 2011

Survey methodology and cross-national comparison

• You have to know about survey methodology whenyou commission data collection (and analysesurvey data)

• You have to know about other countries when youstudy social problems in your own country

• You learn a lot about survey methodology whenyou are involved in cross-national surveys

BaNoCoss 2011

European Social Survey

BaNoCoss 2011

ESAC

BaNoCoss 2011

Nonresponse

EUROPEAN SOCIALSURVEY

European SocialSurvey

Can surveys results be compared acrosscountries?

What we know and what we don’t know

www.europeansocialsurvey.orgBaNoCoss 2011

http://www.europeansocialsurvey.org/


Similar:Norway, Sweden, Finland



ESS

Attitudes, values and beliefsBiennial, start in 2002Small sample size: effective size 1500Central co-ordinationNational implementationMultiple funders



Aims ESS

To monitor and interpretpublic attitudes and valueswithin Europe and toinvestigate how theyinteract with Europe’schanging institutions

• Provide data on beliefs,attitudes and values forscientific and policymaking purposes

• Measure attitude changein a changing Europe

To advance and consolidateimproved methods of cross-national surveymeasurement in Europeand beyond

• Collect data according tohighest standards

• Generate methodologicalresearch

• Develop and disseminatenew best practices

• Develop and improvesocial survey researchinfrastructure in Europe



Content

Direction and change in people’s underlyingattitudes, values and beliefsBackground dataBehavioural patternsCore module + rotating modules

• 1 hour• face-to-face

Supplementary questionnaireEvent data bank + context data



Rotating modules

Call for proposalsQuestions from scientific communityTested measures with proven explanatory value innational settingsAppropriateness to academic and policy needsSuitability for multicultural and multinationalmeasurement



Content

Socio-demo-graphic items• household composition• education• housing• occupation• social structure

Attitudes and behaviour patterns• religious affiliation and identity• ethnic and national identity• political and social trust• party affiliation and voting

behaviour• media consumption• value orientations• social exclusion

R1 ImmigrationCitizenship, involvement anddemocracy

R2 Family, work and well-beingOpinions on health and care-seekingEconomic Morality

R3 Personal & Social Well-beingThe Timing of Life

R4 AgeismWelfare attitudes

R5 Family, work and well-being intimes of economic recessionTrust in police and courts

R6 Personal & Social Well-beingUnderstanding, evaluationdemocracy



Country R1 R2 R3 R4AustriaBelgiumBulgariaCroatiaCyprusCzechDenmarkEstoniaFinlandFranceGermanyGreeceHungaryIcelandIrelandIsraelItalyLatviaLithuaniaLuxembourgNetherlandsNorwayPolandPortugalRomaniaRussiaSlovakiaSloveniaSpainSwedenSwitzerlandTurkeyUKUkraineTotal 22 26 25 30

Participation R1-4



Organisational structure

Specialistadvisorygroups

Question moduledesign teams

Methods

Sampling

Translation

ScientificAdvisoryBoard

Funders’forum

Nationalcoordinatorsand surveyinstitutesCentral

Co-ordinatingTeam



Time table roundsSeptember-December year 2: Quality reportsSeptember year 2: first data releaseFebruary-August year 2: Edits, checks and harmonisation31 January year 2: Data delivery to archiveJanuary year 2: national coding and editingSeptember-December year 1: FieldworkMay-September year 1: Translation, pretest, national

preparationJanuary-May year 1: Sample design, national fieldwork

design, development source questionnaireYear 0: selection fieldwork organisationsYear –1: rotating modules


WHAT WE AIM FORAND HOW


High quality

Strictly random sampling• No substitution, no proxy

Population: all inhabitants aged 15+High target response rates• No substitution• 70% target response rate, max. 3% noncontact

Face-to-face fieldworkPilots, pretesting and MTMM experimentsExtensive monitoringStrict translation proceduresContext and events



Optimal comparability

SpecificationsContracting checklistSampling designFieldwork monitoringTranslation proceduresGuidelines response enhancementGuidelines interviewer trainingUni-modeResponse calculation



Special attention to

Questionnaire structure and development• extensive pre-testing, ensuring functional

equivalence, cognitive testing, maximisingreliability and validity

Strict probability sampling• differing sampling frames, pre-specified minimal

effective sample sizeFieldwork procedures and monitoringArchiving and disseminationMethodological experiments



Access ESS

Round 1, 2, 3 and 4 available• Fully documented• Free access• Tables and graphics online, or download data• Training package for young researchers

Everything availableAll deviations documented



ESS top data user countries (>40)(Jan 2011) (N=36,479)

0 500 1000 1500 2000 2500 3000

MexicoKorea, Republic of

JapanChina

CyprusCroatia

LuxembourgIceland

LatviaPhilippines

LithuaniaAustraliaBulgariaSlovakia

TurkeyCanada

RomaniaRussian Federation

UkraineCzech Republic

IsraelGreeceEstonia

HungarySwedenIrelandFinland

PortugalAustria

SwitzerlandItaly

DenmarkNetherlands

FrancePolandNorway

SpainUnited States

SloveniaUnited Kingdom

BelgiumGermany



WHY AND HOW WE FAIL TOOPTIMIZE QUALITY AND BE

COMPARABLE1. Funding situations2. National situations3. Are identical tool, instruments and procedures identical?4. Plain errors



Where it can go wrong

Survey organisationsNoncoverage and exclusionSampling frames and designsMeasurement and translationNonresponse



Survey organisations

National funding, national knowledgeNational traditionsStatistical offices, universities, market researchorganisations, non-profit organisationsInterviewers• Training, remuneration• Experience f2f, random sampling

Monitoring• Experience• CAPI/PAPI



Organisation and budget

CCT




CCT

NC

FWRC




CCT

NC

FWRC

NCNC

NCNC

NCNC

NCNC

NCNC

NCNC

NCNC

FWFW

FWFW

FWFW

FWFW

FWFW

FWFW

FWFW



Coverage

Random sample, but some groups are excluded

Elderly in institutions, military, student halls, hostelsNot in population register

Non-native speakers (in sample)Opt-out



Sampling

Effective sample sizeOptimal, available sample frameSimple random, systematic, clustered, stratifiedsamplingAddress, household, individual (final stages)Design effects



Country Frame Stages UnitsGreece Building blocks 2 AddressesUkraine Census, streets, random walk 3 AddressesGermany Local residents’ registration 2 AddressesSlovakia Municipality, telephone register, random walk 3 AddressesIsrael Population register 2 AddressesNetherlands Postal delivery points 1 AddressesSwitzerland Postal delivery points 1 AddressesUnited Kingdom Postal delivery points 2 AddressesCroatia Postal delivery points 3 AddressesLatvia State Land Services 1 AddressesRussia Areas, electoral districts, apartment listing 3 HouseholdsFrance Census + random route 2 HouseholdsTurkey Census blocks 3 HouseholdsRomania Census, voting district, random walk 2 HouseholdsBulgaria Electoral sections 2 HouseholdsCyprus Electricity Authority 2 HouseholdsPortugal Municipality + random walk 2 HouseholdsCzech Republic Registr budov (households) 2 HouseholdsSpain Census sections 2 IndividualsDenmark Population register 1 IndividualsEstonia Population register 1 IndividualsFinland Population register 1 IndividualsNorway Population register 1 IndividualsPoland Population register 1 IndividualsSweden Population register 1 IndividualsBelgium Population register 2 IndividualsHungary Population register 2 IndividualsSlovenia Population register 2 Individuals



Nonresponse

Effect sampling frame“Survey climate”Survey fatigueAt home behavior

Contact forms• Call records• Reasons for refusal• Neighbourhood/dwelling data• Differences in implementation



Response rates ESS (selection all rounds)

0

10

20

30

40

50

60

70

80

CH FR ES UK DE BE NO DK NL PT SE HU SI FI PL

2002200420062008



Response rates Poland

05

1015202530354045505560657075

PL

20022004200620082010*



Response rates Switzerland

05

1015202530354045505560657075

CH

20022004200620082010*



Response rates Germany

05

1015202530354045505560657075

DE

20022004200620082010*



Response rates Netherlands

05

1015202530354045505560657075

NL

20022004200620082010*



Response rates Norway

05

1015202530354045505560657075

NO

20022004200620082010*



Response rates Finland

05

1015202530354045505560657075

FI

20022004200620082010*



Response rates Sweden

05

1015202530354045505560657075

SE

20022004200620082010*



Response rates Denmark

05

1015202530354045505560657075

DK

20022004200620082010*



Nonresponse bias

Different response ratesDifferent nonresponse composition (noncontact/bias)Different reasons for nonresponseNo weights yet to adjust for nonresponseSome groups cooperate less• Background characteristics

Big citiesEthnic minorities

• Core variablesInterest in politicsTrust in governmentVoluntary work



Questionnaire design

Theory behind questionsClear conceptsRelevant in every countryFeasible in every country

Length



Translation

Questionnaires in multiple languages• Identical or different questions• Concepts or questions

ESS procedure• Ensure that various concepts were actually represented as precisely as possible• Reliability and validity tests (SQP)• Translation from source language into 1 target language for two-country pilot• Two-nation pilot• Analysis pilot• Comments national coordinators• Final source questionnaire and annotations• Translation and documentation• Small pilots in every participating country



No back-translation in ESS

Back translation• It’s raining cats and dogs• Have you had your tea yet?• Geniessen in vollen Zügen• Dear undertakers

Perfect back translation does not guarantee a goodquestion in the target language• Good help in finding obvious errors (wealthy/healthy)



TRAPD

Translation• More than 1 translator• Parallel or split• What is a good questionnaire translator• Training of translators

Review• Discuss and compare

Adjudicate• Decide on 1 version

PretestDocumentation• Also of review process



MTMM

Questions in different formulations asked withinsingle surveyMulti-trait-multi measurementAnalysis correlation patternsScalability, structural invariance

Batteries?Agree-disagree?Answer categories related to question



Survey mode ESS

Face-to-face• Crucial role of interviewer• Let someone in house• Usually higher response rates• Long fieldwork period• Expensive• Large variation in interview duration• Interviewer effects



Survey modes (not in ESS)Telephone• Crucial role of interviewer• Interviewer can have an effect on answers• Lower response rate• No landline phone, ex-directory• Short interviews

Mail• Less socially desirable• Lower response rates• Who answers the questionnaire?

Internet• Less socially desirable• Lower response rates• Who answers the questionnaire?



Access

Round 1, 2, 3 and 4 availableFully documentedFree accessTables and graphics online, or download dataTraining package for young researchers

Everything availableAll deviations documented



Norway, Sweden, Finland

Fieldwork by NSIInterviewers employeesSample from population registerCAPI surveyFairly high response rateF2F very expensive and increasingly rare



WHAT WE DO TO TRY TOMAKE UP FOR LESS THAN

OPTIMAL QUALITY ANDCOMPARABILITY



Survey organisations

ChecklistsField directors meetingFeedback from round to round

Change of organisations

Site visits (rarely)Long chain of command



Coverage and sampling

Coverage?

Expert sampling panel• Calculate design effects• Improve sample design



Background variables

Quality enhancement groups and improvedquestions on• Education• Occupation• Income• Religion• Marital status



Translation

Cognitive interviewsAdvance translationVerificationCheck formal characteristics



Nonresponse

Improved guidelinesInproved contact formsImprove neighbourhood dataExperiments• Nonresponse surveys

AnalysisCorrect for bias



Nonresponse bias: auxiliary data

Background variables• Aggregate national/low-level, individual• Statistics, databases, sampling frames, interviewer observations

Paradata• Call records, reasons for refusal, proxy refusal, interviewer assessments

of future cooperationCore data• Aggregate (voter turn out), individual information on core data from

nonrespondents



Auxiliary variables and nonresponseadjustment in ESS

Auxiliary variablesin ESS

Populationaggregates

Statistics/survey dataon sex, age,education

Cooperative andreluctant

respondents

Reasons for refusal,Number of contacts,Interviewer judgment

Observational data

Interviewerobservation for

sample and contacts

Doorstep questionsrefusals

Very shortquestionnaire f2f

Follow-up surveyrespondents andnonrespondents

Short/very shortquestionnairemail/telephone



Survey mode

F2F• Expensive• Obsolete?• Not possible in the future

Multi-mode experiments and studies• Country/person• Response effect• Selection effect• Mode effect• Comparability?



In general

LearnAnalyzeFeed backImproveDocument

And how to keep our continuity?….. And our sanity?



LESSONS FOR NATIONALSURVEYS



What about national surveysOne legislation?One sampling frame?One survey organisation?One survey culture?One interviewer culture?One language?One mode?One educational system?One response propensity?One level of answering effort?



Can surveys results be compared acrossindividuals?


can surveys results be compared across countries? · survey methodology and cross-national...

Documents