Impact evaluation-approaches and methodologies

John Maynard Keynes

It’s better to be approximately right than precisely wrong.

Impact evaluation - as the component of a project cycle that seeks to validate to what extent changes occurring to target groups can be attributed to project interventions - is a topical issue in the development sector.

Impact evaluation is a necessary part of project planning and monitoring and for improving implementation strategies regardless of type of organization

For evaluators the challenge now is how to adapt the basic approaches of M&E (qualitative and quantitative) to develop an innovative and cost effective M&E model

For evaluators of market development projects developing models is even more of a challenge due to the complexity of the market itself (supply and demand side)

1. Market intervention indicators-Relating to outputs, measuring the immediate results of project activities e.g

sales performance of products, increased stock volumes, size and expanse of retail networks

2. Market development indicators-Relating to outcomes, measuring results over space and time including

access to information on products location, price and benefits, impressions of specified products to drive demand, supply side development and adherence to business and marketing plans, suppliers capabilities in product stocking and marketing. Others may includes adherence to product quality, assessment of the users knowledge on how to use technology, their purchasing power and demand.

3. Market sustainability indicators-Relating to impacts, measuring results when the project intervention is

withdrawn

Market development evaluators need to go beyond measuring market intervention indicators as the success of a market development project depends on sustained product adoption

Market development indicators are difficult to track because they are qualitative

The MEDA experience has shown that the adoption of MSC as an evaluation technique can be an effective tool in assessing market development projects and specifically market development indicators

What is Most

Significant Change?

• Most Significant Change (MSC) is a participatory, qualitative and complimentary approach to impact evaluation.

• MSC can be likened to newspaper development. Where stories are collected and prioritised. In this case, however, groups of beneficiaries are asked the MSC question and a panel picks the “best” MSC story. These stories are then compiled into a “catalogues/newspaper” and disseminated to decision makers. There can be many levels of sifting stories depending on the size or goal of MSC enquiry.

• The MSC approach was originally developed in 1994 by Rick Davies through his work on a rural development project in Bangladesh with the Christian Commission for Development Bangladesh (CCDB) and has been adopted by a number of organizations in different countries

• MEDA has applied MSC to measure market development indicators to be specific issues around sustainable adoption of SITT’s in the value chain.

Steps of MSC

(Davies et al 2005)

The MEDA Project - Prosperity Through

Innovation

Partner Organisation;

Irrigation Trainers

Smallholder Farmers

Water TechnologySuppliers

and Retailers

MEDA

Provision of Training on Irrigation

Business Transactions on water technologies

Supply side strengthening through capacity building on marketing and cross cutting issues

BUSINESS SUPPORT SERVICES Organization interventions or

Member Services

Market Facilitation Market Development

Lack of Baseline Survey from partners Market Development program; requiring in-

depth understanding of the benefits to program beneficiaries

To supplement training needs assessment tool and client satisfaction survey

To validate sustainable adoption of technologies with the view of accelerating demand. Are clients buying into the technology as appropriate, aim of the project

To check anticipated project impacts/ monitor unintended impacts

Cost

Stories were compiled and at the MEDA office the number of stories on a particular domain of change were weighed with those considered MSC. In both scenarios increase in income was highest placed impact.

The application of MSC enabled the MEDA Zambia team to establish that the major impact of the intervention i.e getting the SSIT’s out to the farmers was an increase in gross income for farmers.

The validation of sustainable adoption for most farmers pointed to benefits in terms of finance accumulated versus livelihood indicators that project monitoring activities were focused on mostly e.g health, education, housing

In project planning terms meaning that the main information that other farmers should be furnished with to accelerate demand should be the potential for them to increase their business income.

It is not a stand alone technique; Hit the nail on the head-get to the issue directly Its never too late too use it Gives a richer description of the context in which a

positive change is occurring, beyond the data from the quantitative case studies

Quality is added to the data collected as the stories are collected in the beneficiaries voice(verbatim)

It is more flexible as it is not based on predefined indicators, you can measure unpredicted outcomes

MSC can be adapted to measure specific qualitative aspects of a multi faceted project

Not all MSC domains will vary from indicators but can still give a closer look at outcomes

During story telling other demand side players learnt from their peers on best methods to employ the adopted technology

The stories can be furnished with quantitative data for depth

If the MSC is conducted in a target group with participants in the same areas they serve as a verification board to measure the truth in a story.

Market development indicators are often a challenge to monitor due to the complexity of the market and the qualitative nature they take.

MSC as an evaluation approach can be applied to capture specific qualitative aspects of change for a multi faceted intervention.

For MEDA its application enabled the project validate sustainable adoption of SITT’s as it related to income increase.

1. There is proof farmers have not only bought the SITT, but have spotted benefits of it and realized its need (accelerating demand)

2. It was not complicated to apply and a larger number of farmers were reached to gather this specific vital information

MEDA is sure what message should be fed to farmers to accelerate demand as the supply chain is being developed even if it was not what was prioritised initially.