Importance of Monitoring and Evaluation

Lecture Overview

Monitoring and Evaluation

How to Build an M&E System

MONITORING & EVALUATIONWhat is M, what is E, why and how to monitor

What is Monitoring

Ongoing process that generates information to inform decision about the program while it is being implemented.

Routine collection and analysis of information to track progress against set plans and check compliance to established standards

Helps identify trends & patterns, adapts strategies, and inform decisions

Key words:

• Continuous – ongoing, frequent in nature

• Collecting and analyzing information – to measure progress towards goals

• Comparing results – assessing the performance of a program/project

Why is Monitoring Important?

Evidence of how much has been or has NOT been achieved

• Quantitative: numbers, percentage

• Qualitative: narrative or observation

Examination of trends

Highlight problems

Early warning signs

Corrective actions

Evaluate effectiveness of management action

Determine achievement of results

What is Evaluation

Evaluation is an assessment of an intervention to determine its relevance, efficiency, effectiveness, impact, and sustainability. The intention is to provide information that is credible and useful, enabling incorporation of lessons learned into decision making processes.

Key Words:

• Assessment – of the value of an event of action

• Relevance

• Efficiency

• Effectiveness

• Impact

• Sustainability

• Lessons learned

What is Evaluation?

Evaluation

Program Evaluation

Impact Evaluation

What is M and what is E?

MonitoringMeasures progress towards goals, but doesn’t tell us the extent to which results achieved or the impact

Continuous, frequent

Has to take place during the intervention

Evaluation

Measures whether progress towards goal is caused by the intervention - causality

Infrequent, time bound

Can evaluate ongoing or completed intervention

Evaluation

Program Evaluation

Impact Evaluation

Monitoring and Evaluation

Monitoring

Components of Program Evaluation

What are the characteristics of the target population? What are the risks and opportunities? What programs are most suitable?

What is the logical chain connecting our program to the desired results?

Is the program being rolled out as planned? Is their high uptake among clients? What do they think of it?

What was the impact and the magnitude of the program?

Given the magnitude of impact and cost, how efficient is the program?

Are your questions connected to decision-making?

Needs assessment

Program theory assessment

Monitoring and process evaluation

Impact evaluation

Cost effectiveness

Evaluation

Programme Evaluation

Impact Evaluation

Program Evaluation

Who is this Evaluation For?

Academics Donors

• Their Constituents Politicians / policymakers Technocrats Implementers Proponents, Skeptics Beneficiaries

How can Impact Evaluation Help Us?

Answers the following questions

• What works best, why and when?

• How can we scale up what works?

Surprisingly little hard evidence on what works

Can do more with given budget with better evidence

If people knew money was going to programmes that worked, could help increase pot for anti-poverty programmes

Programs and their Evaluations: Where do we Start?

Intervention

Start with a problem Verify that the problem

actually exists Generate a theory of why

the problem exists Design the program Think about whether the

solution is cost effective

Program Evaluation

Start with a question Verify the question hasn’t

been answered State a hypothesis

Design the evaluation Determine whether the

value of the answer is worth the cost of the evaluation

Endline Evaluation

Life Cycle of a Program

Baseline Evaluation

Change or improvement

Distributing reading materials and training volunteers

• Reading materials delivered• Volunteers trained • Target children are reached • Classes are run, volunteers show up• Attendance in classes• Entire district is covered

• Refresher training of teachers• Tracking the target children, convincing parents to send their child • Incentives to volunteer to run classes daily and efficiently (motivation)• Efforts made for children to attend regularly • Improve coverage

Theory of Change/ Needs

Assessment

Designing the

program to

implement

Background

preparation,

logistics, roll out of program

Monitoring implementation• Process

evaluation• Progress

towards target

Planning for

continuous

improvement

Reporting findings - impact, process

evaluation findings

Using the findings to improve program

model and delivery

Program Theory – a Snap Shot

Impacts

Outcomes

Outputs

Activities

Inputs

Results

Implementation

HOW TO BUILD AN M&E SYSTEM

With a Focus on measuring both implementation and results?

Methods of Monitoring

First hand information Citizens reporting Surveys Formal reports by project/programme staff

• Project status report

• Project schedule chart

• Project financial status report

• Informal Reports

• Graphic presentations

Monitoring: Questions

Is the intervention implemented as designed? Does the program perform?

Is intervention money, staff and other inputs available and put to use as planned? Are inputs used effectively?

Are the services being delivered as planned? Is the intervention reaching the right population and target

numbers? Is the target population satisfied with services? Are they utilizing

the services? What is the intensity of the treatment?

Implementation

Plans and targets

Inputs Outputs Outcomes

Implementing Monitoring

Develop a monitoring plan

• How should implementation be carried out? What is going to be changed?

• Are the staff’s incentives aligned with project? Can they be incentivized to follow the implementation protocol?

• How will you train staff? How will they interact with beneficiaries or other stakeholders?

• What supplies or tools can you give your staff to make following the implementation design easier?

• What can you do to monitor? (Field visits, tracking forms, administrative data, etc.)

• Intensity of monitoring (frequency, resources required,…)?

Ten Steps to a Results-based Monitoring and Evaluation System

1 2 3 4 5 6 7 8 9 10

Conducting a readiness and needs

assessment

Selecting key

indicators to monitor outcomes

Planning for improvement

selecting realistic targets

Using evaluation information

Using findings

Agreeing on outcomes to monitor and

evaluate

Gathering baseline data on

indicators

Monitoring for results

Reporting findings

Sustaining the M&E

system within the

organization

1 2 3 4 5 6 7 8 9 10

Conducting a needs and readiness assessment

What are the current systems that exist? What is the need for the monitoring and evaluation? Who will benefit from this system? At what levels will the data be used? Do we have organization willingness and capacity to establish

the M&E system? Who has the skills to design and build the M&E system? Who

will manage? What are the barriers to implementing M&E system on the

ground (resource-crunch)? How will you fight these barriers? Will there be pilot programs that can be evaluated within the

M&E system?

- DO WE GO AHEAD?

1 2 3 4 5 6 7 8 9 10

Agreeing on outcomes (to monitor and

evaluate)

What are we trying to achieve? What is the vision that our M&E

system will help us achieve?

Are there national or sectoral goals (commitment to achieving

the MDGs)?

Political/donor driven interest in goals?

In other words, what are our Outcomes: Improving coverage,

learning outcomes… broader than focusing on merely inputs

and activities

1 2 3 4 5 6 7 8 9 10

Selecting key indicators to

monitor outcomes

Identify WHAT needs to get measured so that we know we

have achieved our results?

Avoid broad based results, but assess based on feasibility,

time, cost, relevance

Indicator development is a core activity in building an M&E

system and drives all subsequent data collection, analysis,

and reporting

Arriving at indicators will take come time

Identify plans for data collection, analysis, reportingPILOT! PILOT!

PILOT!

1 2 3 4 5 6 7 8 9 10

Gathering baseline data on indicators

Where are we today?

What is the performance of indicators today?

Sources of baseline information: Primary or Secondary data

Date types: Qualitative or Quantitative

Data collection instruments

1 2 3 4 5 6 7 8 9 10

Planning for improvement

selecting realistic targets

Targets – quantifiable levels of the indicators Sequential, feasible and measurable targets If we reach our sequential set of targets, then we will reach our

outcomes! Time bound – Universal enrolment by 2015 (outcome – better

economic opportunities), Every child immunized by 2013 (outcome - reduction in infant mortality) etc.

Funding and resources available to be taken into account

Target 1

Target 2

Target 3

Outcomes

1 2 3 4 5 6 7 8 9 10

Monitoring for implementation and

results

Impacts

Outcomes

Outputs

Activities

Inputs

Results

Implementation

Results monitoring

Implementation monitoring

Provision of materials; training of volunteers; usage of material; number of volunteers teaching

Change in percentage children who cannot read; Change in teacher attendance

1 2 3 4 5 6 7 8 9 10

Evaluation(?), Using Evaluation Information

Monitoring does not information on attribution and causality. Information through Evaluation can be useful to

Helps determine are the right things being done

Helps select competing strategies by comparing results –

are there better ways of doing things?

Helps build consensus on scale-up

Investigate why something did not work – scope for in-

depth analysis

Evaluate the costs relative to benefits and help allocate

limited resources

1 2 3 4 5 6 7 8 9 10

Reporting findings

Reporting Findings: What findings are reported to whom, in what

format, and at what intervals. A good M&E system should provide

an early warning system to detect problems or inconsistencies, as

well as being a vehicle for demonstrating the value of an

intervention – so do not hide poor results.

Using Results: recognize both internal and external uses of your

results

Sustaining the M&E System: Some ways of doing this are

generating demand, assigning responsibilities, increasing

capacity, gather ing trustworthy data.

Sustaining M&E System

Using Results

THANK YOU