Participatory Impact Assessment

Framework for integrating Local Perception and conventional methods in the analysis of

impacts

Dawit, Abebe, Andrew Catley, John Burns and other FIC Team

29th ALNAP Annual Meeting

11 - 12 March 2014 Addis Ababa

Objectives

• The concept of participatory impact assessment

• Generic framework for designing PIA

Origins of participatory impact assessment

• Development theory • Practical considerations • Process versus impact indicators

Origins: Development theory

• Community participation The involvement of communities in the description and analysis of problems, and identification of solutions It leads to joint design and implementation of action. It often requires profound attitudinal change among professionals; professionals act as facilitators, co-learners and mentors Ultimately, it relates to issues of power, control, respect and Ownership.

• Local people have their own perceptions of change, their own ways of measuring change and their own analysis of project attribution

• These perceptions not only matter, but within a participatory way of working they are crucial to impact assessment and refining future interventions

• However poor or vulnerable, people possess specialist knowledge and skills (human capital). Rather than seeing people as ignorant, uneducated or irrational, participatory approaches highlight indigenous knowledge in areas such as farming methods and livestock husbandry.

Origins: Development theory

Origins: Practical issues

• The logistical and operational constraints to conventional research

• “Where there is no data” – the limited or complete lack of baseline data in relief or development projects

• Action-orientated rather than data driven: the principle of “optimal ignorance”

Origins: Process vs. Impact

Two main types of indicators are measured in aid projects:

• Process indicators

• measure things being done e.g. quantity of medicines delivered

• Impact indicators • measure the outcome of things being done

e.g. changes in the prevalence of disease • Assessment of aid projects dominated by

process rather than impact – the “truck and chuck” mentality

Policy input

Information use: by project / Donor

Analysis: Jointly with the community

Data collection: Participatory methods

Indicators: Community identified

Information use: by project / Donor

Analysis: by project /external

Data collection methods: Conventional, by project / external

Indicators: Process by project /external

Overview / definition of PIA

Project Planning

Implementation

Monitoring

Evaluation

PIA

Community Participation

Gen

eric

Fra

mew

ork

for

desi

gnin

g PI

A

Question

Scale

Method

Indicator

Attribution

Triangulation

• Define research questions Policy (Formulation / influence) Project (Planning / Implementation)

• Spatial scale of analysis • Temporal scale of analysis

• Community perceived / identified

• Sampling • Participatory methods and tools

• Project and non-project factors positively or negatively affecting result

• Monitoring and other secondary data

• Feedback to the community and other stakeholders

Validation

Stage 1. Identify the key questions

The most important, and often the most difficult stage of a PIA is deciding which questions should be answered. Who defines the questions?

If you’ve already worked with communities to identify their impact indicators at the beginning of a project, the questions will focus on the measurement of these indicators and assessment of project attribution. If you’re using a retrospective approach, discuss the impact assessment with community partners / representatives and jointly define the questions with them.

Defining the questions in an impact assessment is like defining the objectives of a project – unless you know, specifically, what you’re trying to achieve, you’re unlikely to achieve it

Example: Provision of sheep and goats to female-headed households

Assume that a project provides sheep goats to poor female headed-households. For a PIA, such a project may have only three questions which need to be answered:

1. How has the project impacted, if at all, on the livelihoods of the women involved in the project

2. How has the project impacted, if at all, on the nutritional status of the women’s children?

3. How might the project be changed to improve impact in the future?

Stage 2. Define the boundaries of the project in space and time

Define spatial unit of analysis: geographical boundaries of the project (mapping)

Aims to ensure that everyone understand the limits of the area in which impact is supposed to take place.

• Participatory Mapping • “Interview the map”: When the group finished the diagram • Cross-check community map with other groups (villages) and

secondary data (such as project document, etc.)

Define temporal boundaries of the project (timelines) Sometimes called the “temporal boundary”. When did

the project start, how long has it been running? Aims to ensure that everyone is clear about the time

period which is being assessed.

Stage 3. Identify community-defined indicators of project impact

• As far as possible, the PIA should use impact indicators which are identified by the community or beneficiaries of the project.

• Local people have their own way of describing change, and their own priorities for improving their lives

• Participatory impact assessment (PIA) works with communities to identify locally-defined impact indicators

2. How can we identify community impact indicators? 3. Specific versus general indicators 4. Number of indicators 5. Quantitative and Qualitative Indicators

1. Who identifies the impact indicators?

Stage 4: Methods

1. Sampling • Time and money • Questions and • Methods

Stage 4: Methods

2. Decide which participatory methods you will use, and test the methods Mapping Time-lines “Before and after” proportional piling* “Before and after” scoring* Simple ranking* Matrix scoring*

All methods involves the use of semi-structured

interviews as part of the method * = method which produces numerical data

Stage 5: Measure changes in the impact indicators which occurred during the project

If possible, use community-defined indicators of impact “Before and after” scoring methods are useful. For each indicator,

What was the situation (score) at the start of the project? What is the situation (score) now? If there is a difference, why? Probe these reasons

Example Method: ‘Before’ and ‘After’ Scoring (Example: Community sources of income – Garissa, Kenya)

Example Methods: Matrix Scoring

• List indicators in Y-axis and items to be compared against the indicator on the X-axis

• Provide a counter, and ask one volunteer to distribute to the items according to importance

• Repeat for all indicators

• Get a consensus from the rest of the group on the distribution and record the count after each step

Stage 6: Assess project attribution

This stage aims to understand all the project and non-project factors which contributed to the changes in the impact indicators – these factors should be listed. Also, it aims to understand the relative importance of these project and non-project factors – methods include simple ranking or scoring methods, and causal diagrams with scoring of causes.

Example of assessing Attribution Before After Change factors

Stage 7: Triangulate

This is a crucial stage of the assessment It uses “secondary data” to cross-check the results of the participatory methods. The key source of secondary data is the project’s monitoring of process indicators (implementation of activities) Other secondary data may be government reports, or studies by other agencies

Stage 7: Triangulate

Stage 8: Validating the finding by the community

This final stage involves presentation of the PIA findings back to the community. If a CBO or local group is involved in the project, they should receive a copy of the results (and the final report). This stage is a final opportunity for local people to verify that the results are correct. A “feedback” workshop can also be an opportunity to plan further work to improve the project.

Kendall’s W Friedman test Chi sq. test

Qua

ntita

tive

Ana

lysi

s

Matrix scoring of veterinary service providers in southern Ethiopia (source: Admassu et al., 2005)

Example: Comparison of source of income ‘before’ and ‘after’ (Destocking intervention in Ethiopia)

Scoring income utilization – household use of income derived from a commercial de-stocking

project (n=114) (source: Abebe et al., 2008)

Before and after scoring of food basket contributions (n=145) (source: Burns and Suji, 2007)

Application of PIA

PIA is now used worldwide by Government research institutes, NGOs, and others

• Impact of humanitarian and development projects

• Understanding of resilience to climate variability and extreme events in coupled social and ecological systems

Catley, A., Abebe, D., Admassu, B., Bekele, G., Abera, B., Eshete, G., Rufael, T., Haile, T. (2007). Impact of drought-related livestock vaccination in pastoralist areas of Ethiopia. Disasters

Abebe, D., Cullis, A., Catley, A., Aklilu, Y., Mekonnen, G. and Ghebrechirstos, Y. (2007). Livelihoods impact and benefit-cost estimation of a commercial de-stocking relief intervention in Moyale district, southern Ethiopia. Disasters

Impact Assessments of Livelihoods-based Drought Interventions in Moyale and Dire Woredas, Ethiopia This is a comprehensive report describing assessments of Save the Children US commercial de-stocking in Moyale district, and CARE's veterinary, livestock feed and de-stocking in Dire district.

Examples

Participatory impact assessment helps to bridge two gaps: • The limited understanding of the livelihoods

impact of relief and development interventions – implications for organizational learning and national-level policy reform

• Methodologically, a gap between the ad hoc use of participatory methods and systematic research with statistical analysis

• Correct use of the approach requires specialized

training and experience in participatory approaches and methods

Conclusions

PIA can be useful because: • It focus on changes which local people

prioritize and can be measure using their own indicators, methods and resources

• It is based on indigenous knowledge and skills which is based on continuous monitoring and assessment of changes

• It uses participatory methods that often describe trends or compare variables across spatial and temporal scales, rather than seek to make precise measurements

Conclusions

Thank you

Participatory Impact AssessmentSlide Number 2Slide Number 3Slide Number 4Slide Number 5Slide Number 6Slide Number 7Slide Number 8Slide Number 9Slide Number 10Slide Number 11Slide Number 12Slide Number 13Slide Number 14Slide Number 15Slide Number 16Slide Number 17Example Methods: Matrix ScoringSlide Number 19Example of assessing AttributionSlide Number 21Stage 7: TriangulateSlide Number 23Slide Number 24Slide Number 25Example: Comparison of source of income ‘before’ and ‘after’ (Destocking intervention in Ethiopia)Scoring income utilization – household use of income derived from a commercial de-stocking project (n=114) (source: Abebe et al., 2008) Slide Number 28Slide Number 29Slide Number 30Slide Number 31Slide Number 32Thank you