The role of monitoring and evaluation in improving public policies – challenges and achievements.

International Conference on the Institutionalization of public policies Evaluation

Rabat, 5 October 2015

Fabio Veras Soares (IPC-IG)

M&E: what for?

Monitoring and evaluation are long acknowledged as essential elements of the policy making process.

Monitoring and evaluation activities are meant to inform… governments, policy makers, practitioners and society at large

whether programmes are working… as planned, over-performing, or.. under-performing.

M&E: connected but not the same

Monitoring and evaluation should be interconnected but they are essentially two different process.

It is possible to have a good monitoring system without having an evaluation and vice-versa.

However, a good monitoring system will never be able to respond to the key evaluation question: What would have happened to the target population in the absence of the intervention that has been monitored? Would they have been better off or worse off or just the same?

The possibility to attribute observed changes in outputs and outcomes to a specific intervention is at the heart of the evaluation problem.

M&E: monitoring role

a good monitoring process provides information with a periodicity that enable quick action to rectify implementation hurdles.

Through the record of the inputs, process and outputs of a programme, a monitoring system provides constant feedback on the extent to which the intervention is achieving its targets.

Besides identifying potential problems at an early stage, it should also be able to suggest possible solutions.

The provision of a reliable flow of accurate information during implementation allows managers to keep track of progress and adjust activities accordingly.

The monitoring system also allows to assess how accessible the intervention has been to the target population.

Finally, it gauges the efficiency with which the different components of the intervention are being implemented and suggests improvements, providing internal and external accountability of the resources used and the results obtained.

M&E: the importance of monitoring for the evaluation process

Monitoring systems per se are not able to answer the evaluation question, but…

They offer the inputs – indicators and qualitative information– that will support the evaluation process…

• by enhancing our understanding of the reasons for a good, bad or normal performance of a programme.

• by proving key information on costs that will allow the measurement of the cost-effectiveness of the programmes and interventions.

M&E: the importance of monitoring for the evaluation process

The Evaluation component of an intervention analyses and interprets the data collected to understand the causal relationship between inputs invested into the programme, its components and the overall impact on outputs and outcomes.

In order to do so in a rigorous way, particularly, when performing an impact evaluation, it is important to be able to measure a counterfactual.

Impact evaluation: the counterfactual

The counterfactual is the potential output/outcome indicator in the absence of the programme.

The difference between the observed output/outcome and the counterfactual is the estimate of the impact of the programme.

As one never observes the same population under both states – affected by programme and not affected by the programme – it is necessary to build a control or comparison group as similar as possible to the target/treated group in order to proxy this counterfactual and estimate the impact of the intervention.

Impact Evaluation: counterfactual

The requirements to build a credible counterfactual can be quite cumbersome (and sometimes expensive vis-à-vis the overall cost of the intervention). Thus one should be able to prioritize what should be evaluated and how using some clear criteria.

Ideally, the comparison group should be identified at the design phase of the intervention. Data collection includes both target/treated population as well as the comparison/control group.

As data is also collected for population groups not immediately benefiting from the interventions, most likely the monitoring and information system of the programme will not suffice for this process. The evaluation component, in general, needs to have its own data collection process. Early definition of the evaluation procedure potential can allow the incorporation of the comparison group in the MIS system, improving the quality of the evaluation and, in some cases, reducing the costs of the evaluation.

Evaluation starts with the design of an intervention

Treating the M&E process as an ex-post phase of an intervention increases its costs and jeopardizes its credibility as second-best approaches and methodologies need to be used.

However, the rule has not been to implement the M&E component from scratch and the reason behind it is that many governments and policymakers still see monitoring and evaluation as a source of potential criticism and either do not actively search to make it current practice or do not take full advantage of it – overlooking its findings.

Investment case for M&E and political will To show that the benefits of a proper M&E system outweighs its

costs (including political costs) is one of the challenges faced by practitioners worldwide.

During the International Conference on National Evaluation Capacity in 2013 (São Paulo, Brazil) practitioners from 52 countries identified among the challenges to implement and strengthening the evaluation components of public policy issues related to the engagement of parliamentarians and the development of legislation and institutional arrangements that could incentivize the implementation and use of evaluation results in the improvement of public policies.

Important successful cases…

The CONEVAL experience in Mexico is a clear example of the type of institutions that has the ability to foster evaluation of public policies, ensure accountability and put forward suggestions on how to improve their efficiency.

A similar role is developed by the ONDH whose institutional mission in Morroco is to assess and evaluate in a continuous way the impact of human development programmes and to put forward suggestions to improve these programmes.

In Tunisia the CRES has been supporting the government in the reform of their Social Protection System through rigorous research and evaluation components.

Strong M&E for the achievement of the SDGs.

The SDG is due to set an important role for the evaluation community both at the international and national level. "Evaluation" has been integrated in the new Agenda within the "follow-up and review" section.

This section suggests that the follow-up and review processes of implementation of the post-2015 Development Agenda will be informed by country-led evaluations and data and calls for strengthening national evaluation capacity.

In this conference we will have the opportunity to learn more about the work of these institutions and to reflect how they can contribute to the strengthening of national and international evaluation capacities and the improvement of public policies.

Merci beaucoup

شكرا

Gracias

Thank you

Obrigado