VCC Extractive Industries Executive TrainingMatthew HarrisSehrish BariShira MitchellJune 18, 2014

Designing and Implementing Effective Monitoring and Evaluation Systems

Objectives of Session:

1) To introduce the concepts of an M&E system

2) To illustrate its application to a large-scale development project (Millennium Villages Project)

3) To discuss its relevance to Extractive Industries

Program Cycle

What is M&E?

Basic M&E Concepts

Purpose of M&E

Developing an M&E Plan

1) To introduce the concepts of an M&E system

Program CycleNeeds Assessment

Planning/Designing the Program

Implementation

Monitoring and Evaluating

Scaling up and Sustainability

Logical Framework• Logical Framework Approach (LFA): A general approach to program planning,

monitoring and evaluation that encourages consideration of relationships between resources, activities, and desired changes/results.

• Components of Logical Framework:– Inputs (Resources)– Processes (Activities)– Outputs (Results)– Outcomes (Objectives)– Impacts (Goal)

Simplified Logic Model• Logic Model depicts the logical framework components as a linear process

– Causal thinking (IF -> THEN)– Input -> Activity -> Output -> Outcome -> Impact

• Example: What would a simplified logic model look like for an educational program that can lead to reduction of HIV/AIDS transmission?– Goal: Reduction of HIV/AIDS transmission– Objective: Increasing knowledge in community about HIV/AIDS

Input

•Funding for the educational course resources (staff, materials, space, etc)

Activity

• Implementation of the educational course

Output

•At-risk community members attend the course

Outcome

•HIV/AIDS knowledge among community members increases

Impact

•Reduction in HIV/AIDS transmission

7

What Do We Mean by M&E?• Set of procedures & analytical tools to examine

• how programs are conducted (inputs & activities)• their level of performance (outputs)• whether they achieved what they were intended to

achieve (outcomes & impact)

8

Monitoring vs Evaluation

Monitoring: What are we doing?Tracking inputs and outputs to assess whether programs are

performing according to plans; collecting routine data to measure progress toward achieving goals and objectives

(e.g., people trained, materials distributed)

Evaluation: What have we achieved?Assessment of impact of the program on population,

community or industry (e.g., use of environmentally sustainable approaches, economic growth)

9

Monitoring• Is a continuous systematic process of collecting,

analyzing and using information to track the efficiency of achieving program goals and objectives

• Provides regular feedback that measures change over time in any of the program components such as costs, personnel and program implementation

• An unexpected change in monitoring data may trigger the need for a more formal evaluation of activities

10

Illustration of Program Monitoring

Programstart

Programend

TIME

Programindicator

Key Monitoring Questions

• Is the program being implemented as planned?• Are things moving in the right direction?• Which program activities were more (or less)

important/effective in reaching the desired immediate outputs?

12

Evaluation• Is a systematic process limited in time of collecting, analyzing

and using information to assess the effectiveness, relevance and impact of achieving your program’s goals.

• Requires study design; some evaluation methods require a control or comparison group; often measurement over time.

• Often involves measuring changes in knowledge, attitudes, behaviors, skills, community norms, utilization of services or approaches, & long term indicators of “success” at population, community or industry level

• Provides feedback that helps programs analyze the consequences, outcomes and results of its actions

13

• Did the program achieve its objectives?• Did the target population benefit from the program?• At what cost? • Can improved social, environmental, agricultural, economic

etc outcomes be attributed to program efforts? • Which program activities were more (or less)

important/effective?• What would have happened in the absence of the program? • How can we know or measure this (the counterfactual)?

Key Evaluation Questions

Monitoring vs. Evaluation Monitoring

• Repeat measures • Consistent measures

structured around key indicators

• More “bare bones” • Quantitative

Evaluation

• Tailored and nuanced• Structured around key

indicators and evaluation questions

• Can give a more in depth understanding

• Can include qualitative measures

How would we know if we achieved success? What would it look like?

Indicator: A quantitative or qualitative variable (something that

changes) that provides a simple and reliable measurement of one

aspect of performance, achievement or change in a program or

project.

Indicator of a country’s wealth = GDP

Indicators

What Role do Indicators Play in M&E?• Reduce large amount of data down to its simplest form

• Measure program or project progress towards targets and desired outcomes

• Measure trends over time

• Provide a yardstick whereby organizations, facilities etc. can compare themselves to others doing similar work

• Provide evidence for achievement (or lack of) of results and activities

• Process Indicators vs. Outcome Indicators vs. Impact Indicators

Program Components as They Relate to M&E Indicators

Processes/Activities Outputs Impact

Assessments Input/Output Monitoring

Process Evaluation

Outcome Monitoring

Outcome Evaluation

InputsPlanning Outcomes

Impact Evaluation

Outcome Indicators

Impact Indicators

Process Indicators

Indicator Characteristics

• Specific

• Measurable

• Achievable

• Relevant

• Time-bound

Collecting data for indicators: • Existing records (archival data)• Verbal data – individual and group interviews• Surveys - written questionnaires• Observations; reports from trained observers• Participation logs• Mechanical measures or apparatus (e.g. water purity; HIV

tests; etc.)• Standardized tests• Site visits

(Kusek & Rist, 2004)

Millennium Development Goals• Goal 1: Eradicate extreme poverty and hunger• Goal 2: Achieve universal primary education• Goal 3: Promote gender equality and empower

women• Goal 4: Reduce child mortality• Goal 5: Improve maternal health• Goal 6: Combat HIV/AIDS, malaria, and other

diseases• Goal 7: Ensure environmental sustainability• Goal 8: Develop global partnership for development

The Millennium Development Goals:Select Indicators

21

Goals and Targets(from the Millennium

Declaration)Indicators for monitoring progress

Goal 2: Achieve universal primary educationTarget 2.A: Ensure that, by 2015, children everywhere, boys and girls alike, will be able to complete a full course of primary schooling

1. Net enrollment ratio in primary education2. Proportion of pupils starting grade 1 who

reach last grade of primary 3. Literacy rate of 15-24 year-olds, women and

men

Goal 3: Promote gender equality and empower womenTarget 3.A: Eliminate gender disparity in primary and secondary education, preferably by 2005, and in all levels of education no later than 2015

1. Ratios of girls to boys in primary, secondary and tertiary education

2. Share of women in wage employment in the non-agricultural sector

3. Proportion of seats held by women in national parliament

Why Monitor & Evaluate?

Purpose of M&E

Assessing Impact

Reporting/ Accountability/ Transparency

Policy & Advocacy

• To make decisions about project management and service delivery

• To ensure effective and efficient use of resources and provide accountability to donors/funders and community

• To assess whether the project has achieved its objectives - has the desired effects

• To learn from our activities, and provide information to design future projects

Developing an M&E Plan

M&E Action Plan – DefinitionComprehensive document that describes all:

– Program objectives, interventions developed to achieve these objectives, & procedures to be implemented to determine whether or not the objectives are met

– Expected results of the program and how they relate to goals and objectives

– Data needs, how they will be collected & analyzed– Information use, including resources needed to do so– How the program will be accountable to stakeholders

M&E Plan Suggested Outline1.) Introduction2.) Description of the overall program- including problem statement and framework(s)3.) Indicators- including definitions (presented in indicator matrix and/or indicator reference sheets: very detailed!)4.) Data sources and reporting systems (including management/roles and responsibilities)

5.) Plans for demonstrating program outcome/impact6.) Plans for dissemination and use of information7.) Analysis of data quality constraints & potential solutions8.) Implementation plan (aka- M&E action plan or road map-should include budget and timeline)

Elements of an M&E Plan: Introduction (1)

• Purpose of the plan• Description of how it was developed

– Stakeholders involved– Consensus process

Elements of an M&E Plan: Program Description (2)• Problem Statement

– What is the nature of the development issue/public investment program being addressed?

• Conceptual Framework• Goal and Objectives

– What is the ultimate outcome of the program (goal)– What are the shorter-term aims (objectives)

• Program Description:– Interventions– Geographic scope– Target population, community, industry, etc.– Duration

• Logical Framework (Logic Model)

Elements of an M&E Plan: Indicators (3)• Selection of indicators based on:

– Conceptual and logic frameworks– Strategic information needed for decision-making at appropriate level

(country/state/local)– Donor or funder, government, stakeholder requirements– Existing data – Funding (available and dedicated to M&E)

• Presented in two ways:– Indicator Matrix- a table presenting indicators including information on data

source, frequency and who is responsible– Indicator Reference Sheets- detailed sheets describing each indicator, how

to measure it, underlying assumptions & interpretation considerations (may be included as appendices)

Elements of an M&E Plan:Data Sources and Reporting Systems (4)

• Sources of data for indicators• Diagram of data collection, processing, analysis, and reporting

system• Data collection tools

– Records or registers– Survey instruments– Commodity management forms (materials distributed)– Others?

• Management– Roles and responsibilities of each group/member of the M&E

system• Data Flow

Elements of an M&E Plan: Plan for demonstrating program outcome/impact (5)

• A methodology for measuring program outcome or impact (the evaluation, to be explained later in presentation)

• Protocols for special studies– E.g. Nutrition assessment of children under five in

development project

Elements of an M&E Plan: Dissemination and Use Plan (6)

• Clearly defined users• Databases for information storage• Dissemination methods

– Reports (schedule and audience)– Media outlets (interviews, press releases, etc.)– Speaking events– Community feedback– Others?

Elements of an M&E Plan: Data Quality Assurance (7)

• Describe known constraints to data quality and/or system performance

• What will be done to address these constraints?

Elements of an M&E Plan: Implementation Plan (8)

• Assessment of feasibility to implement plan• A detailed work plan for the M&E Plan to

include:– Each M&E activity (including update of M&E

Plan)– Timing of each activity– Party responsible for each activity– Budget necessary for each activity

M&E Costs to ConsiderHow much will it cost? Benchmark 5-10% of total program budget (rough estimate) Need to budget:• Costs of information systems (costs of data collection,

processing, and analyzing)• Costs of information dissemination and use• Costs of the data quality control system• Costs of coordination and capacity building• Human Resources for M&E (coordinator, data collectors,

analysts)

Developing & Implementing an M&E Plan: Dos and Don’ts

Do:• Start early• Involve stakeholders at all stages in the process• Assess current capacity and use what is already available• Avoid duplication of data collection and reporting

Do not:• Collect information that will not be used• Underestimate the importance of stakeholder buy-in and

ownership at every juncture

Monitoring and evaluation is important but ineffective if the information is not used.

Setting up systems to facilitate M&E use is just as important as the M&E systems designed to collect

the data.

Introduction to the Millennium Villages Project

MVP M&E structure

MVP M&E Components

2) To illustrate its application to a large-scale development project (Millennium Villages Project)

Country

District

Community

EducationHealth

AgricultureBusiness

development

Infrastructure

Design & Implementation of a Multi-Sectoral Project

• MVP is divided into sectors, each with a strategy:– Agriculture– Water & Sanitation– Energy– Environment– Technology & Innovation– Gender Equality– Maternal & Child Health– Education– Business Entrepreneurship

Example 1: AGRICULTURE 1. Increasing sustainable agricultural crop production;2. Improving food and nutrition security;3. Farm diversification for income generation; and4. Underpinning sustainability by restoring and conserving the natural resource base.

Example 2: HEALTH• Building and operating clinics and dispensaries

to address:– Child malnutrition– Contraception access– Malaria prevention and treatment– Emergency obstetrics services– HIV testing– Free primary health care

Example 3: WATER & SANITATIONProtected spring Borehole

A Millennium Village

(Artist’s Rendering)

MVP Chain of Stakeholders

Village Community Members --- Local Government --- Site Teams --- MDG Regional Centers --- National Governments --- The Earth Institute at Columbia University --- Millennium Promise (NGO) --- United Nations --- Local Implementing Partners --- External Donors

Overview of the MVP M&E Structure

M+E Platform • Documents and evaluates the

implementation experience of the MV systems and major programs and interventions

Process Evaluation

• Tracks costs of all MV, Community and Government expenditures

Economic Costing

• Monitors MDG-related Indicator progress.

Outcome Monitoring

Surveys • Evaluates MDG Progress

Outcome Monitoring &

Surveys

+Outcome MonitoringContinual and systematic process of

collecting and analyzing data to measure the performance of interventions towards achievement of outcomes

Alert managers to problems; ▲accountability

Promote timely performance-based decision-making

+OM in the context of LogFrameInputs: resources mobilized to support

activities.Activities / Processes: actions or work to

convert inputs into specific outputsOutputs: converts inputs into tangible

outputsOutcomes: use of outputs by targeted

populations (intermediate objective)Impacts: final object of the program (long-term

goal)

Impl

emen

tati

on

Leve

l (m

ore

cont

rol)

Resu

lts

Leve

l (l

ess

cont

rol)

MVIS: Millennium Villages Information System

• Web-based data entry and visualization tool– accessible in low-connectivity settings

• Clearinghouse for aggregated data across multiple data collection streams in all MVP sites – multi-sector

– multi-source

– multi-site

• Tool to assist decision-makers at all organizational levels.

51

+Links with Information Systems

MVISRIFA

Schools• Periodic

Clinics• Periodic

CommCare Verbal

Autopsy

Vital Events

Agri-Business

High Priority Aggregate Indicators

Survey Overview• To measure the adequacy (outcome evaluation) of the

intervention effects and progress towards MDG-related outcomes, assessment rounds have been conducted in the Millennium Research Village at baseline (2005–2007), after three years of project exposure (2008–2010), and after 5 years of project exposure (2010–2012). The next major evaluation survey round will be in 2015.• Smaller surveys are currently being conducted (2013-14) for monitoring

purposes.

• Comprehensive survey based MDG assessment Produce core MDGs and MDG-related indicators, proxies, and related indicators that are operationally useful.

Survey Overview, Cont’d• Demographic census

– Establish population profile, survey sampling frame• Household survey

– Head of Household / Most Knowledgeable Member– Agriculture, Economics: (income, assets, consumption, expenditure), Education and

Infrastructure: water, sanitation, energy, transport, communication• Adult survey

– All 15-49 yo adults , within research households– Health: health seeking, maternal-child health, social capital, HIV, functional literacy,

mental health– Starting in Year 3, fielded Reproduction and Pregnancy Tool, expanded sample of

additional 15-49 yo females700 HH in MV1 (research village)• Anthropometric monitoring

– Children under 5• Biological specimens

– malaria, anemia, stool and urine• Nutrition survey

– Food security, food frequency

Household Surveys: Baseline – Year 5

Y0 Y3 Y5

Scope All survey modules All survey modules All survey modules

CoverageMV 300 HHs for 14 sites

MV 300 HHs for 12 sites

MV 300-400 HHs for 10 sites

Timeline 2005-2008 2008-2010 2010-2012

OutputsMDG indicators report

Cleaned micro-level dataMDG indicators report

Cleaned micro-level dataMDG indicators reportY5 Dashboard

HH Surveys: Process & Functions

Survey Planning and Instruments

Data Collection and Processing

Data / Indicator Analysis

Data Archiving and Dissemination

• Pre-testing / refinement of survey tools

• Training of survey managers and field teams

• Supervision of data collection, entry, & cleaning

• Reporting and process documentation

• Simultaneous data entry

• Micro-level data cleaning

• Quality assurance checks (error report)

• Cleaned micro-level data exportation

• Standardization & de-identify micro-level data

• Perform 2nd level de-identification of micro-level data

• Prepare metadata records

• Generate survey data catalog

• Document process• Manage micro-level data

requests

• Generate MDG indicators

• Analysis & validation of indicators

• Validate results with sector / site teams

• Generate MDG indicators reports, and dashboard / progress chart

• Prepare / generate external reference indicators

• Conduct statistical analysis for publications

• Develop budgets & work plans

• Manage IRB protocols• Redesign/update

questionnaires, training and enumeration protocols.

• Translate survey instruments

• Sampling design, power calculations, weights, etc.

• Procure survey equipment and supplies

• Update data entry template, logic, & cleaning scripts

From paper to datasets - CSPro: Used on-site for data entry and data cleaning

CSPro

CSPro

STATA

QA: Data Entry/Cleaning –data entry work flow

Millennium Villages Data Archive

Process Evaluation

How was the overall project designed, deployed, maintained and utilized? (Institutional narrative.)

Context thickening: Reality vs. the model!

What were the major lessons learned (i.e. successes and failures, barriers and facilitators) over the duration of the project?

What does the Process Evaluation Attempt to Answer?

What is a process evaluation?

• Complementary evaluation technique to the conventional impact or outcome evaluation which asks: ‘Did the intervention work?’

• Process evaluation focuses on program operations, with a view to understanding how outcomes were achieved.

• Thus, both impact and process data complete the full picture of what is needed for a thorough PROGRAM EVALUATION.

How does this fit with quantitative data?

Process Evaluation Methodology

• Clinic workers• Community Health

Workers• Traditional

healers/birth attendants

Health

• Extension Agents• Farmers

Agriculture• Cooperative members• Agro-dealers

Business Enterprise

• Borehole Committees• Government extension

workers

Infrastructure• Teachers• Parent & Teacher

Committees• Students

Education

• Men, Women • Opinion Leaders• Chiefs• Religious leaders

Community

•Pre-site preparation

•Critical document review – quarterly/ annual reports

•Review of MVP :manual” and “handbooks” to ask what is the MVP model?

•Interviews with

•Interviews with NY Earth Institute and Millennium Promise staff

Location: New York

•Site team interactions

•Kry informant Interviews with Team Leader, Science Co-ordinator, Sector Heads.

•Focus Group discussion with entire Site Team

Location: Case Study

Site

•Community Focus Groups.

• Discussions with Men, Women, Youth

Location: Case Study

Site

•Local government interviews

Location: Local / District

Government. Offices in Case

Study Site •National government interviews

Location: National

Government Headquarters for Case Study Site

• MDG Center Interviews

Location: Regional

MDG Center•Final semi-structured interviews with MP and Earth Institute staff

Location: New York

Methods employed with each level of stakeholder in MVP PE

Coding Dimensions: 7 Stages of project implementation • To what extent were stakeholders involved in designing and planning the interventions?• What problems were encountered and how were they resolved?• If objectives, plans, timetables and targets were revised, why was this necessary?

Design, planning and targeting

• What are the strengths and weaknesses in the current MVP organizational structure? • What adaptations have been made, and why?

Organizational structure and personnel

• How were initial cost projections formulated? How have they been re-formulated?• What are the greatest concerns related to budgeting and financing of the MVP?• How has the MVP been able to mobilize resources most efficiently?

Budgeting and Finance

• What have been the major facilitators and barriers encountered with logistics and resource procurement for the stakeholder? How have these barriers been overcome? What recommendations can be made?

Logistics and procurement

• How have the current mechanisms for monitoring and support worked? • What aspects of the existing protocols have been useful? Unuseful?• What kinds of resources are required to improve the current Monitoring and Support system?

Monitoring and Support

• What are the current strategies for ensuring effective transitioning & sustainability?• What are the major threats to the sustainabilty of the MVP?• How has the MDG Center been most effective in helping to transition and sustain the MVP?

Transitioning and Sustainability

• What are examples of successfull synergies between sectors?• Have synergies strengthened the sustainability of the project?• Which factors are most important to consider when determining the sequence of MVP interventions planning and delivery?

Sequencing and Synergies

Data Management: Thematic Content Analysis

http://www.millenniumvillages.org/index.htm

Qualitative methods enrich quantitative analysis in three ways:

─ Qualitative findings will inform the transfer of successful practices and their implementation into policies

─ the MV Field Guide Manual for new sites.

Operations Policy & scale upResearch─ Feedback on what is working,

what isn’t and underlying root causes …

─ … allowing timely improvement/adjustments to work plans at all levels

─ Mixed methods research will enrich publication efforts

Contributions of Process Evaluations

Economic Costing

MV Core

External donors

Government

Community

$60

$20

$30

$10

$30

$20

$40

$20

Infrastructure

Agriculture

Health

Education

Stakeholder Sector

$10 Management

Total: $120/person/year ($160 in today’s value)

Costing Model

Costing evaluation aim and objectives:– Aim: To determine the total yearly cost of all cluster-

level activities within the MV that contribute to achieving the MDGs

– Objectives:• Generate activity timeline, based on expenditure• Assess relative contributions by stakeholders• Assess relative contributions across sectors• Identify main expenditure areas within sectors• Analyze changes over time, including changes since baseline• Compare costs in MV cluster to costs in CV

Costing evaluation is a combination of:• Financial tools

– Expenditure records, budgets, annual reports, quarterly reports

• Economic tools– Key informant interviews, focus groups, cost

imputation, cost estimation

Why conduct the costing evaluation?

– Test the concept that the MDGs can be achieved at $120/person/year

– Providing an indicator for the level of local ownership • Have communities and gov’t bought in?• Implications for sustainability and scalability

– Provides an accurate measure of intervention “dose”• Expenditures tell us when an activity actually happened, and to what extent

– To inform planning and scale-up• Example: Projecting the cost of universal primary health care in rural sub-Saharan Africa

– Can eventually pair costing data with impact data for cost-effectiveness studies

                Methods: Typology and Expenditure Chart

Baseline 2006 2007 2008 2009$0

$20

$40

$60

$80

$100

$120

$140

$160

MVPComm.DonorGov't

Baseline 2006 2007 2008 2009$0

$20

$40

$60

$80

$100

$120

$140

$160

Mgmnt.Infra.Ag.HealthEdu.

Annual Costs by Stakeholder Annual Costs By Sector

Sample Results (from Bonsaaso, Ghana)

Non-amortized average annual per capita MDG-related expenditures, by stakeholder

Base

line

Ave.

, 200

6 - 2

009

Base

line

Ave.

, 200

6 - 2

008

Base

line

Ave.

, 200

6 - 2

011

Base

line

Ave.

, 200

6 - 2

009

Base

line

Ave.

, 200

6 - 2

010

Base

line

Ave.

, 200

5 - 2

010

Base

line

Ave.

, 200

6 - 2

009

Base

line

Ave.

, 200

6 - 2

010

Base

line

Ave.

, 200

6 - 2

009

Base

line

Ave.

, 200

6 - 2

009

Base

line

Ave.

, 200

5 - 2

009

Ave.

Bas

elin

eAv

e. Y

early

MVP

Mod

el

Bonsaaso Pampaida Potou Tiby Dertu Koraro Mayange Mbola Mwandama

Ruhiira Sauri 11 Sites

$0

$20

$40

$60

$80

$100

$120

$140

$160

MVPComm.DonorGov't

Notes:Government baseline contribution to Pampaida estimated at $0.47/person (not visible). Infrastructure costs amortized for baseline years only; baseline costing data collected for gov’t only for GB, NP, MT, UR, KS.Results are shown in current USD.

What has been done so far?

Final MVP Impact Evaluation

Discussion/Exercise

3) To discuss relevance of M&E concepts to Extractive Industries

GROUP EXERCISE & DISCUSSION

Contacts:Sehrish Bari – sbari@ei.columbia.eduMatthew Harris – mharris@ei.columbia.edu

THANK YOU!