Surveys as a tool to improve aid in fragile states

The Asia Foundation’s Afghanistan experience

A fragile state environment

Highly dynamic environment; uncertain &unstable

Strained state-society relations; distrust & suspicion

Multiple interests and actors at play; high costs of navigation

Insufficient and unreliable information

Weak local capabilities

Weak physical and transportation infrastructure

Some ways that surveys can help

Capture trends in a dynamic environment, if done over time

Monitor state-society relations over time What matters? Public opinion vs. expert opinion Help capture key issues that are normally sensitive and deeply political Specific issues unique to the state can be studied in depth

Can be a public good and used by multiple actors, where coordination risks and fund flows are high Better inform program design Better inform spending (targeting and resourcing) Test assumptions on key issues As a common baseline to track progress

Build local capacity to provide reliable information

Varying objectives of TAF surveys

Objective Survey1) Inform host government policy-makers and local policy debates

Survey of Public Perceptions of the Mindanao Peace Process (2008)

Local Economic Governance Index (EGI) in 5 countries

Voter surveys in pre-election periods (Bangladesh, Cambodia, Indonesia, Pakistan, among others)

2) Articulate/reflect key concerns of conflict-affected populations

Democracy & Conflict in Southern Thailand

Survey of the Afghan People (2006-11)

3) Inform program design Public perceptions of police (Timor-Leste, Bangladesh, Sri Lanka, Indonesia)

Leaders of Influence surveys (Maldives, Bangladesh)

4) Inform donor strategies or international policy

Survey of the Afghan People (2006-11)

Survey of the Sri Lankan People (2010)

5) Measure impact of programs Community policing pilot program in Sri Lanka, Timor-Leste (2009-11)

Case in point: Survey of the Afghan People

Nationwide quantitative survey conducted annually using a

structured questionnaire among more than 6200 Afghans

6 surveys conducted since baseline in 2006

Findings at http://www.asiafoundation.org/publications/pdf/989

Survey decision environment

Pre survey

During survey

Post survey

Some Pre-survey Challenges

Objective (s) of the survey Partnerships For what and with whom

‘Localisation’ (Afghanisation) of the survey Ownership and accountability Objectivity Capacity

Some Challenges during Survey

In general, minimal secondary information available for design In Afghanistan, last census in 1979, partial only No electoral rolls

Questionnaire development Compatibility to other surveys within & outside the context Collaborative inputs Curiosity questions vs. actionable findings

Scale of the fieldwork Weak to nonexistent local capacity Challenges with female interviewers

…Some Challenges during Survey

Weak infrastructure and harsh weather conditions Mountainous terrain & lack of (good) roads Long and harsh winters

Ethno-linguistic sensitivities/closed society Conservatism Insularity

Respondent selection and gender balance Balanced gender coverage at each sample point Travel restrictions for female interviewers

…Some Challenges during Survey

Quality control Added oversight and supervision of field team and data cleaning Backchecks and accompanied interviews

Managing sampling and error Increasing insecurity and inaccessibility Greater replacement of sampling points Error margin can increase

Some Post-Survey and Other Challenges

Analysis and reporting considerations Overstating the findings Interpreting every finding Emphasizing magnitude vs. direction Looking beyond topline findings to in-depth analysis, when to move

along? When to integrate with comparisons to other worldwide surveys

Dissemination plans and the ethics of research Selective data dissemination, suppression of findings Duplicative efforts, sharing of information

…Some Post-Survey and Other Challenges

Risk management Antigovernment elements, local strongmen, competitors Host Government Donors

Way forward: one-off exercise vs. longitudinal – time series Changing objective over time of conducting large scale surveys of

Afghans Disciplined to team, timing, and, quality control costs Choice between parachute survey firm and in-house management

in early years

Pictures from the field

Team training…

The terrain…

Interviews…

Thank You!

Objectives of the Afghan Survey

Use strong social science research to provide credible, policy-

relevant information about public opinion on recent changes and

trends in government performance, public policy, politics and

political processes, and development progress

Build social research capacity in the country

Dealing with outdated population data

Rural sampling -- Combination of PPS and SRS

Urban sampling -- use of maps/locality lists in the absence of

database such as electoral rolls

Innovative field movements for selection of starting points and

households

Building the field team

Extensive training of interviewers & supervisors Questionnaire Sampling procedure- field movements, selection of households Kish grid

‘Train the trainer’ sessions

Mock interviews Female interviewer recruitment in each district of selected sample

point

Innovations

Team of women interviewers recruited locally ‘Mahrams’ for women interviewers

Men and women interviewers in each sampling area

Modifications in Kish selections Women interviewers obtained details of female HH members; men

obtained details of males