monitoring and evaluation capacity building for program ... · monitoring and evaluation capacity...

Monitoring and Evaluation Capacity Building for Program Improvement

October 28-31, 2003

Monitoring and

Evaluation

Teen Club Community Partners Training Programme

Did you know …?Did you know …?

Managers commonly state that measuring Managers commonly state that measuring performance is too complex, time intensive, performance is too complex, time intensive,

and costly. and costly.

Many organizations consider M&E to be a Many organizations consider M&E to be a

requirement of the funding agencies that requirement of the funding agencies that support them and thus see M&E as an support them and thus see M&E as an external necessity. external necessity.

Few managers consider M&E to be the Few managers consider M&E to be the strategic system assessing organizational strategic system assessing organizational capacity, judging economic effectiveness or capacity, judging economic effectiveness or

predicting future programmatic sustainability. predicting future programmatic sustainability.

Principles and

purposes of monitoring, evaluation and

reporting

What Is Monitoring &

Evaluation?� Monitoring:

� Tracks information relevant to program planning, outputs, outcomes and impacts.

� Tracks costs & program functioning.

� Provides basis for program evaluation

� Evaluation:

� Is a rigorous, scientifically based collection of information about program activities, characteristics, and outcomes to determine the merit or worth of a specific program.

� Is used to improve programs and inform decisions about future resource allocations.

Purposes of Monitoring and

Evaluation

� Ensuring planned results are achieved

� Improving and supporting management

� Generating shared understanding

� Building the capacity of those involved

� Motivating stakeholders

� Ensuring accountability

� Fostering public and political support

Evaluative Thinking

Evaluation is a process and way of Evaluation is a process and way of thinkingthinking

Evaluation is not just about providing Evaluation is not just about providing information for someone else at the end of information for someone else at the end of

a projecta project

M & E should be an integral part of M & E should be an integral part of managementmanagement

M & E provide key information for M & E provide key information for managementmanagement

M & E can be creative, fun and rewardingM & E can be creative, fun and rewarding


October 28-31, 2003

Why M&E is Important

M&E provides managers with a M&E provides managers with a

tool to gain timely information on tool to gain timely information on the progress of project activities the progress of project activities

allowing them to compare what allowing them to compare what

was planned to what is actually was planned to what is actually occurring.occurring.

M&E promotes organizational M&E promotes organizational learning and encourages learning and encourages

adaptive management.adaptive management.

Why M&E is Important Cont.

M&E provides organizations with M&E provides organizations with

a tool to evaluate how a tool to evaluate how EFFECTIVELY they promote EFFECTIVELY they promote

change.change.

Essential Elements for a Successful Essential Elements for a Successful M & E Program M & E Program

Building an M&E system begins with Building an M&E system begins with

development of a Monitoring development of a Monitoring Evaluation and Reporting PlanEvaluation and Reporting Plan

An M&E plan documents data An M&E plan documents data

collection, evaluation and reporting collection, evaluation and reporting procedures to be undertaken by the procedures to be undertaken by the

organization. organization.

The M&E plan identifies:

� The mission and results sought

� Who needs information and data

from the organization, why, and

when.

� What data (management

information and indicators) will be collected by the organization.

� The source, method, frequency and schedule of data collection.

The M&E plan identifies (cont.):

Plans, schedules, and tools for evaluating Plans, schedules, and tools for evaluating information.information.

The team or individuals responsible for The team or individuals responsible for

data collection.data collection.

Plans, schedules, and tools for Plans, schedules, and tools for communicating information.communicating information.

How data will be analyzed, compared, How data will be analyzed, compared,

reviewed and presented, and the reviewed and presented, and the associated tools to be used associated tools to be used

Value of M&E for You Value of M&E for You

Facilitates project designFacilitates project design

Provides ongoing data and is thus a Provides ongoing data and is thus a

management toolmanagement tool

Provides evaluation data for reporting Provides evaluation data for reporting

purposespurposes

Can generate data for operational Can generate data for operational

researchresearch


October 28-31, 2003

The Project CycleThe Project CycleOrganisation

Mission

Financing and contracting

Final Evaluation

Gov/Donor Goals/Policies

Beneficiary Needs

Plan

Act

Monitor

Evaluate

Implementation

Monitoring and Evaluation

Scoping

Formulation / Design

M&E Strategy

Mobilisation and planning

M&E Plan

Results-Based M&E

The Development

Objective� Today organizations must present the

results of their programs as

contributing to a larger development objective.

� A development objective is the overall and long term effect of an intervention (it is your highest level of impact

anticipated). Reduction in incidence of HIV or improved food security are examples

Results Based Management (RBM)Results Based Management (RBM)An approach to project design and An approach to project design and management that emphasizes management that emphasizes

achieving results rather than simply achieving results rather than simply carrying out activitiescarrying out activities

Common sense but many projects Common sense but many projects having fallen into the trap of losing having fallen into the trap of losing sight of what they are really trying to sight of what they are really trying to

achieveachieve

RBM implies an effective project RBM implies an effective project monitoring and evaluation systemmonitoring and evaluation system

Setting Goals and

Objectives

DefinitionsDefinitions

Goal: a general statement that Goal: a general statement that

describes the hopeddescribes the hoped--for result of a for result of a programmeprogramme. Goals are achieved over . Goals are achieved over

the long term (5the long term (5--10 yeas) and often 10 yeas) and often

through the combined efforts of through the combined efforts of multiple multiple programmesprogrammes (e.g. reduction (e.g. reduction

of HIV/AIDS incidence)of HIV/AIDS incidence)


October 28-31, 2003

DefinitionsDefinitions

Objective: the specific, Objective: the specific, operationalizedoperationalized

statement detailing the desired statement detailing the desired accomplishment of the accomplishment of the programmeprogramme. .

Objectives address questions of what Objectives address questions of what and when but not why. and when but not why.

They should be stated in terms of They should be stated in terms of results to be achieved, not processes results to be achieved, not processes

or activities to be performed. or activities to be performed.

SMART MethodSMART MethodSpecific: Specific: Identifies concrete events or actionsIdentifies concrete events or actions

that will take place.that will take place.

Measurable: Quantifies the amount of resources,Measurable: Quantifies the amount of resources,

activity or change.activity or change.

Appropriate: Logically relates to the overallAppropriate: Logically relates to the overall

problem statement and desiredproblem statement and desired

effects of the program.effects of the program.

Realistic: Provides realistic dimension that Realistic: Provides realistic dimension that

can be achieved with availablecan be achieved with available

resources resources

TimeTime--based: Specifies a time within which the based: Specifies a time within which the

objective will be achieved.objective will be achieved.

Sample ObjectivesSample Objectives

Process Objective: Process Objective:

Provide VCT clients with HIV test Provide VCT clients with HIV test

results.results.

Outcome Objective: Outcome Objective:

Assist VCT clients with Assist VCT clients with

developing personalized riskdeveloping personalized risk--

reduction and treatment reduction and treatment strategies.strategies.

Examples of SMART ObjectivesExamples of SMART Objectives

Process Objective: Process Objective:

By the end of the first program year, By the end of the first program year, 98% of clients tested for HIV will 98% of clients tested for HIV will receive test results. receive test results.

Outcome Objective:Outcome Objective:

By the beginning of the second By the beginning of the second

program year, 65% of clients receiving program year, 65% of clients receiving HIV test results will have developed HIV test results will have developed and adhered to personalized riskand adhered to personalized risk--

reduction/treatment strategies.reduction/treatment strategies.

Examples of Objectives Improve use of TB register data (as measured by decrease Improve use of TB register data (as measured by decrease

in the transferin the transfer--out rate, interruption rate & missing data, as out rate, interruption rate & missing data, as well as improved validity of data) in 8 of 9 provinces by well as improved validity of data) in 8 of 9 provinces by

March 31, 2004.March 31, 2004.________________________________________________________________________________________________________

Pilot and evaluate a WindowPilot and evaluate a Window--based Electronic TB Register (ETR) in based Electronic TB Register (ETR) in 1 province by 3/31/03.1 province by 3/31/03.

Decrease the transferDecrease the transfer--out rate to < 7% in 8 provinces within 12 out rate to < 7% in 8 provinces within 12

months of implementation of ETR (baseline 2000, 11.1%).months of implementation of ETR (baseline 2000, 11.1%).

Decrease the interruption rate to <10% in districts within 12 months Decrease the interruption rate to <10% in districts within 12 months of implementation of ETR (baseline 2000, 15.1%).of implementation of ETR (baseline 2000, 15.1%).

Decrease missing data to less than 10% within 12 months.Decrease missing data to less than 10% within 12 months.

Conduct TA visits in 8/9 provinces and support implementation of Conduct TA visits in 8/9 provinces and support implementation of

the ETR particularly regarding validation and use of data.the ETR particularly regarding validation and use of data.

ExerciseExercise

Mafikeng OVC Trust is developing a Mafikeng OVC Trust is developing a

satellite Teen Club satellite Teen Club programmeprogramme in in order to increase access to care and order to increase access to care and

support for HIVsupport for HIV--positive adolescents positive adolescents

in Northwest Province.in Northwest Province.

Write specific SMART objectives for Write specific SMART objectives for

the the programmeprogramme


October 28-31, 2003

Logic Model

Logic Model Definition

A logic model describes the main elements of a A logic model describes the main elements of a

program and how they work together to prevent program and how they work together to prevent

HIV in a specific population.HIV in a specific population.

This model is often displayed in a flow chart, map This model is often displayed in a flow chart, map

or table to portray the sequence of steps leading or table to portray the sequence of steps leading

to program outcomes. to program outcomes.

Levels of Results, Inputs Levels of Results, Inputs

& Processes& Processes

Inputs & ProcessesInputs & Processes

OutputsOutputs

OutcomesOutcomes

Impact Impact

Levels of Results - Outputs

� Outputs are information, products, or results produced by undertaking activities

� Outputs relate to completion of activities.

� Outputs reflect what you hoped to produce

from a particular input. For example: You decide the process you want to use is to train people, thus people trained is the result at

the input/process level

� Outputs usually reflect a result achieved in a

relatively short-time period (0-2 years).

Levels of Results - Outcomes

Outcomes are broad changes in Outcomes are broad changes in development conditions, answering the “so development conditions, answering the “so

what” question (So… we trained 100 people what” question (So… we trained 100 people and increased their knowledge but did or did and increased their knowledge but did or did they not change their behavior?). they not change their behavior?).

Outcomes often reflect behavior or economic Outcomes often reflect behavior or economic change and show how our activities change and show how our activities

contribute toward development objectives. contribute toward development objectives.

Outcomes usually reflect a result achieved Outcomes usually reflect a result achieved an intermediate time period (2an intermediate time period (2--5 years) 5 years)

Levels of Results - Impacts

Impacts are the overall and long term Impacts are the overall and long term effects of an intervention. Impacts are effects of an intervention. Impacts are

the ultimate result attributable to a the ultimate result attributable to a development intervention over an development intervention over an extended period. Improved food extended period. Improved food

security, increased standards of living security, increased standards of living etcetc

Impacts usually reflect a result Impacts usually reflect a result achieved over a longer time period (5achieved over a longer time period (5--10+ years)10+ years)


October 28-31, 2003

A Good Logic Model

� Includes a problem statement, inputs, activities, outputs, outcomes, and impacts

� Reflects agreement among major stakeholders about intended implementation

and outcomes (planned logic model)

� Illustrates clear, sequential, and logical linkages between each part of the logic

model

� Contains a problem statement that identifies

underlying causes

VCT Program Logic Model

Counseling and Testing

Personnel*

Funding from gov’t, GAP, & other donors

VCT MIS**

HIV test kits

Referral system for

prevention & Tx services**

ACTIVITIES OUTPUTS OUTCOMES IMPACTSINPUTS

VCT protocols,guidelines, and

training documents**

Problem Statement: HIV infection rates continue to rise, underscoring the importance for people to know their serostatus:

develop personalized risk-reduction strategies and access care and treatment services.

ExerciseExercise

Write down the inputs available Write down the inputs available

for implementation of the satellite for implementation of the satellite

Teen Club Teen Club programmeprogramme for for MafikengMafikeng


Train counseling & testing personnel and site managers


Personnel*

Provide pre-test counseling, HIV testing, and post-test counseling to all clients tested

Provide pregnant HIV+ women with PMTCT services


VCT MIS**

HIV test kits

Referral system for


Screen HIV+ clients for (or refer to

screening for) OIs and TB specifically


Refer HIV+ clients to Tx services



Exercise Exercise

Define the activities the Mafikeng Define the activities the Mafikeng

OVC Trust should undertake to OVC Trust should undertake to

implement the Mafikeng satellite implement the Mafikeng satellite Teen Club Teen Club programmeprogramme..




Personnel*




VCT MIS**

HIV test kits

Referral system for




# clients received pre-test counseling,# clients received HIV tests,# clients received results & post-test counseling***

# pregnant HIV+ women receive PMTCT services

# HIV+ clients referred to or

provided OI/TB screening

# personnel certified in VCT



# HIV+ clients referred to Tx, care & support including

ARVs




October 28-31, 2003

ExerciseExercise

Determine the outputs (i.e. shortDetermine the outputs (i.e. short--

term results) you expect from the term results) you expect from the

new Mafikeng satellite Teen Club new Mafikeng satellite Teen Club programmeprogramme



Risk behaviors decreased


Personnel*




VCT MIS**

HIV test kits

Referral system for








Quality of VCT increased

Access to VCT increased

Clients (HIV+ and -) develop & adhere to

personalized HIV risk-reduction &

treatment strategy

Increase in care, prevention, and

treatment services for HIV+, HIV-, and

discordant couples





ARVs

Health outcomes of HIV + improve



ExerciseExercise

Determine the outcomes (i.e. Determine the outcomes (i.e.

midmid--term results) you expect from term results) you expect from

the new Mafikeng satellite Teen the new Mafikeng satellite Teen Club Club programmeprogramme..



Risk behaviors decreased


Personnel*




VCT MIS**

HIV test kits

Referral system for








Quality of VCT increased

Access to VCT increased

Clients (HIV+ and -) develop & adhere to

personalized HIV risk-reduction &

treatment strategy

Increase in care, prevention, and

treatment services for HIV+, HIV-, and

discordant couples

HIV transmission rates decreased

HIV incidence decreased

HIV morbidity & mortality decreased





ARVs

Health outcomes of HIV + improve



ExerciseExercise

Determine the impact (i.e. longDetermine the impact (i.e. long--

term results) you expect from the term results) you expect from the

new Mafikeng satellite Teen Club new Mafikeng satellite Teen Club programmeprogramme.. Indicators


October 28-31, 2003

What is an indicator?

� Indicators identify what you will measure to know if conditions have or have not

changed.

� An indicator is specific information that

provides evidence as to the achievement of results and activities

� Indicators can be quantitative measures or

qualitative (narrative related) observations

� Indicators are indicative – it enables you to

reduce a large amount of data down to its simplest form

Why are Indicators Important?

� Indicators serve as tools to examine trends and highlight problems.

� Indicators provide an early warning signal to predict future conditions.

� When compared with targets, indicators can signal the need for corrective management action, can evaluate the

effectiveness of various management actions, and provide evidence as to whether objectives are being achieved.

Characteristics of Good Indicators

Measurable

Practical (data can be collected on a timely basis and at reasonable cost).

Reliable (can be measured repeatedly

with precision by different people).

Relevant - Attributable to YOUR

ORGANZATION (the extent to which a result is caused by YOUR activities).

Management Useful (the information provided by the measure is critical to decision-making).

Characteristics of Good Indicators

Continued

Direct (the indicator closely tracks the result it is intended to measure).

Sensitive (serves as an early warning of changing conditions).

Responsive (can be changed by management control).

Objective (the measure is operationally precise).

Exercise Exercise

Make a list of possible indicators Make a list of possible indicators

associated with the outcomes you associated with the outcomes you

have determined for the new have determined for the new Mafikeng satellite Teen Club Mafikeng satellite Teen Club

programmeprogramme

Selecting the “Best” IndicatorsSelecting the “Best” Indicators

� Based on your analysis, narrow the list to the final indicators that will be used

in the monitoring system.

� They should be the optimal set that

meets management needs at a reasonable cost.

� Limit the number of indicators used to

track each objective or result to a few (two or three).


October 28-31, 2003

Data Collection,

Analysis and Dissemination

Data Collection and StorageData Collection and Storage

The data collection methods The data collection methods

including standardized data including standardized data collection tools, explanatory notes collection tools, explanatory notes

and the means for data and the means for data

aggregation (e.g. cumulative, aggregation (e.g. cumulative, monthly report forms) should be monthly report forms) should be

detailed in the M&E plan. detailed in the M&E plan.

Data Analysis and ReportingData Analysis and Reporting

Information provided by staff at all Information provided by staff at all

levels should be input in levels should be input in appropriate databases or entered appropriate databases or entered

in a filing system in a filing system

Data disseminationData dissemination

Data collected through M&E activities will Data collected through M&E activities will be used to highlight successful strategies, be used to highlight successful strategies,

model practices that can be replicated, model practices that can be replicated, and inform others about lessons learnt. and inform others about lessons learnt.

Information is also used to determine if Information is also used to determine if planned planned programmeprogramme objectives are objectives are achieved or not, if target populations have achieved or not, if target populations have

been reached and the desired quality of been reached and the desired quality of services provided has been reached.services provided has been reached.

Quality feedback loopQuality feedback loop

�� Failure to reach preset goals Failure to reach preset goals

prompts investigation of possible prompts investigation of possible causes.causes.

Quality element of BIPAI M&EQuality element of BIPAI M&E

Feedback on aggregated dataFeedback on aggregated data

according to defined schedule andaccording to defined schedule and

preset criteria: preset criteria:

e.g. 3 month adherence data on alle.g. 3 month adherence data on all

patients recently initiated on AVRs:patients recently initiated on AVRs:

failure to attain a level of 80% offailure to attain a level of 80% of

patients at or above 95%patients at or above 95%

adherence is flaggedadherence is flagged


October 28-31, 2003

Quality element of Quality element of

BIPAI M&EBIPAI M&E

Immediate feedback on a per Immediate feedback on a per

patient basis according to preset patient basis according to preset

criteria: criteria:

e.g. patient is flagged if not e.g. patient is flagged if not

reaching desired efficacy on reaching desired efficacy on ARVs assessed as assessed by ARVs assessed as assessed by

viral loadviral load

Evaluation

Key Evaluation QuestionsKey Evaluation Questions

� Relevance - Was/is the project a good idea given the situation to improve? Was

the logic of the project correct? Why or Why Not?

� Effectiveness - Have the planned results been achieved? Why or Why Not

� Efficiency - Have resources been used in

the best possible way? Why or Why Not?

Key evaluation questions (cont.)Key evaluation questions (cont.)

Sustainability Sustainability -- Will there be continued Will there be continued positive impacts as a result of the project positive impacts as a result of the project

once it has finished? Why or Why Not?once it has finished? Why or Why Not?

Impact Impact -- To what extent has the project To what extent has the project contributed towards its longer term goals? contributed towards its longer term goals?

Why or Why Not? Have there been any Why or Why Not? Have there been any unanticipated positive or negative unanticipated positive or negative consequences of the project? Why did consequences of the project? Why did

they arise?they arise?

ExerciseExercise

Formulate the key evaluation Formulate the key evaluation

questions you would like to be questions you would like to be

answered at the midanswered at the mid--point or point or completion of the new Mafikeng completion of the new Mafikeng

satellite Teen Club satellite Teen Club programmeprogramme..

M&E Data Use

ProgramImprovement

ProgramImprovement

Reporting/Accountability

Reporting/Accountability

Share

Data withPartners

Share

Data withPartners

monitoring and evaluation capacity building for program ... · monitoring and evaluation capacity...

Documents