Research Methodology: An Overview of Impact Evaluation

BY: IBITOLA OMOTAYO23RD SEPTEMBER 2016

Knowledge Check 1. What is Impact Evaluation? 2. Is there a need for Impact Evaluation? If yes, why? If no, why not? 3. What are the methods for conducting Impact Evaluations?

ObjectivesTo understand the:key concepts of Impact Evaluation.aspects of Impact Evaluation.3Ws of Impact Evaluation: why, what, and who methods for conducting Impact Evaluation.considerations for ensuring quality in Impact Evaluation

Impact Evaluation: what is it? “IMPACT” refers to the positive and negative, intended and unintended, direct

and indirect, primary and secondary effects produced by an intervention. (OECD)

•Impact evaluations measure the change in a development outcome that is

attributable to a defined intervention; (USAID)

•Impact evaluation is a systematic and empirical assessment of the effects brought

about by an intervention. (AusAID)

Aspects of Impact Evaluation

Impact Evaluation

Formative(ex-ante)

Summative

Formative Aspect of Impact Evaluation

Seeks to inform decisions with regards to making changes to a programme or

policy.

While many formative evaluations (i.e. baseline evaluation) focus on processes,

impact evaluations can be used formatively if an intervention is ongoing. For

example, the findings of an impact evaluation can be used to improve

implementation of a programme for the next intake of participants.

Summative Aspect of Impact Evaluation

Seeks to inform decisions about whether to continue, discontinue, replicate or

scale up an intervention.

Ideally, summative impact evaluation not only produces findings about ‘what

works’ but also provides information about what is needed to make the

intervention work for different groups in different settings, which can then be

used to inform decisions.

Impact Evaluation: Why should it be done?

•To decide whether to fund an intervention– “ex-ante evaluation” is conducted before

an intervention is implemented to estimate its likely impacts and inform funding

decisions.

•To decide whether or not to continue or expand an intervention.

•To learn how to replicate scale up a pilot.

•To learn how to successfully adapt a successful intervention to suit another context.

Impact Evaluation: Why should it be done?•To show accountability:

reassure funders, including donors and taxpayers (upward accountability),

that money is being wisely invested.

inform intended beneficiaries and communities (downward accountability)

about whether or not, and in what ways, a program is benefiting the

community

What Questions Does Impact Evaluation Seek to Answer?High-level questions directed at the overall purpose of the intervention and should address-:

1. Overall impact

2. Nature of impacts and their distribution

3. Influence of other factors on the impacts

4. Mode of operation

5. Alignment of intended impacts to needs***These are not the specific questions that are to be asked during interviews

Overall impact

•Did it work? Did the intervention produce the intended impacts in the short,

medium and long term?

•For whom, in what ways and in what circumstances did the intervention work?

•What unintended impacts (positive and negative) did the intervention

produce?

Nature of impacts and their distribution

•Are the impacts likely to be sustainable? •Did these impacts reach all intended beneficiaries?

impact

Influence of other factors on the impacts

•How did [the intervention] work in conjunction with other interventions, programs or services to achieve outcomes?

•What helped or hindered [the intervention] to achieve these impacts?

Mode Of Operation

•How did the intervention contribute to intended impacts?•What were the particular features of the intervention that made a difference?•What variations were there in implementation?•What has been the quality of implementation in different sites?•To what extent are differences in impact explained by variations in implementation?

Alignment of intended impact to needs•To what extent did the impacts match the needs of the intended beneficiaries?

Who Should Conduct Impact Evaluation?

Impact evaluation can be undertaken by:

an external evaluator or evaluation team (e.g Preston –ENGINE)

an internal but separate unit of the implementing organization (e.g. M&E dept

within implementing organizations)

those involved in an intervention (including community members);

a combined team of internal and external evaluators.

Choosing Methods for Impact Evaluation?

Evaluation design

& Estimation methods

purpose of the

evaluation

availability of resources

nature of the

intervention

Evaluation designs

http://www.socialresearchmethods.net

Evaluation designsExperimental

• Independent measures

• Repeated Measures• Matched Pairs

Quasi- experimental

• Time series• Separate sample

pre-test/post-test• Separate sample

pre-test/post-test control group etc….

Non-experimental

• single-variable research

• correlational and quasi-experimental research

• qualitative research

www.simplypsychology.org/experimental-designs.html

Estimation methods The are usually based on the evaluation designs. Different designs require different estimation methods to measure the counterfactual.

The four categories of estimation methods are:◦ Comparison of means◦ Multi-variate regression◦ Instrumental variables methods◦ Double difference or difference-in-difference.

*** Counterfactual: a comparison between what actually happened and what would have happened in the absence of the intervention

Considerations for quality Impact Evaluation•UTILITY

•ACCURACY

•PROPRIETY (ETHICS)

•PRACTICALITY

•ACCOUNTABILITY

Considerations for quality Impact Evaluation

•UTILITY: the results of the evaluation must be useful to those who require the information.

•ACCURACY: ensuring that findings are reported fairly, comprehensively and clearly.

•PROPRIETY (ETHICS):ethical issues of confidentiality and anonymity, as well as potential harmful effects of being involved in the evaluation must be adequately addressed.

Considerations for quality Impact Evaluation?

•PRACTICALITY: taking into account the available resources (time, money, expertise and existing data) and when the results are needed to inform decisions.

•ACCOUNTABILITY: refers to presenting clear evidence and criteria on which conclusions have been drawn, and acknowledging their limitations. Transparency about data sources is important.

QUESTIONS & COMMENTS

More Information

www.betterevaluation.orginteractive textbook at http://www.worldbank.org/pdthttps://www.cdc.gov/std/Program/pupestd/Types%20of%20Evaluation.pdfhttp://www.interaction.org/impact-evaluation-notes.https://dfat.gov.au/aid/how-we-measure-performance/ode/Documents/impact-evaluation-discussion-paper.pdf

Post-test 1. What is impact evaluation? 2. Is there a need for impact evaluation? If yes, why? If no, why not? 3. What are the methods for conducting Impact evaluations?