monitoring & evaluation lfa presentation
DESCRIPTION
TRANSCRIPT
Programme for Internal UNIFEM meeting Migration team for finalizing the Project Document of Phase III
26-27th August 2010
UNIFEM SARO officeDelhi, India
1
Planning, Monitoring and Evaluation in
UNIFEM
Session 1: 9.30-10.30Presentation on LFA from UNIFEM perspective
Session 2: 2.00-3.00Presentation on Key aspects of Monitoring and Evaluation
2
Agenda
To learn relevant planning techniques1. Stakeholder Identification2. Causal Analysis
Making a Problem Tree Making an Objective Tree
3. Project Selection4. Project Logic – Making a Logical Framework
To apply the skills to develop a Monitoring and Evaluation (ME) plan
3
Objectives
Session 1Planning techniques to develop a Logical Framework
4
5
1. Stakeholders Identification
UNIFEM would like to help to improve migrant women’s situation in Asia.
What are the problems of migrant women in Asia?
6
Stakeholder Identification Exercise
List on cards all individuals, groups and organizations related to migrant women in Asia.
7
Stakeholder Identification Exercise
8
For instance
Children
Women in rural areas in Thailand
UNIFEM
Community leader
For instance:
◦ Who are beneficiaries?◦ Any negatively affected groups?◦ Who are decision-makers?◦ Who are funding agencies?◦ Who are implementing?◦ Who will oppose? ◦ Who will support?
9
Categorizing Groups
Different groups have different needs Project may effect different groups in
different ways Whose problems should be solved? Women first!
10
Stakeholder Identification helps
11
2. Causal Analysis – Making a Problem Tree
Root causes is to keep looking. Ask “why”, get a direct cause. Keep asking “why”, get a cause that you
can act on.
12
Purpose of Causal Analysis- Making a Problem Tree
What are the problems in countries/areas you are working?
List the problems.
13
Causal Analysis Exercise – Making a Problem Tree
Indicate existing problems (not theoretical or potential)
One problem per card Describe in a sentence Try to avoid expression such as “No
(solution or resource) is available). Describe the condition resulting from the lack or absence of resources.
14
Principles of writing problem cards
15
For instance
There is no school.Children cannot receive
quality education.
AVOID PREFER
Decide the core problem Keep ask “why” to identify the causes
16
Causal Analysis Exercise – Making a Problem Tree
Causal Analysis (Problem Tree) Exercise: Causes to the Core Problem
17
Causes
Core Problem
Core Problem
Example: The Core Problem to Effects
18
Children drop out school.
Children are engaged in child labor.
Many children become
street children.
Children are exploited.
Many street children are involved in illegal acts.
Core Problem
Negative Effects
Causes Core Problem Negative Effect If we ask “why”, we know what the
problem is. If we know what the problem is, we
should know how to solve!
19
Summary of Causal Analysis (Problem Tree)
Causal Analysis (Objective Tree) Exercise: Solutions to Core Objective
20
Solutions
Core Problem
Core Objective
Example: Core Objective to Positive Ends
21
Children complete school.
Children can basic skills.
Children live in a protected environment.
.Children can
get an adequately paid job.
Children stay away from illegal acts.
Core Objective
Positive Ends
Solutions Core Objective Positive Ends
If we can act on causes, we can solve the core problem.
22
Summary of Causal Analysis (Objective Tree)
23
3. Project Selection
1. Use the Objective Tree2. Categorize related topics into a group3. Name the Group4. Refer the selection criteria5. Compare and examine the different Groups6. Select one group to be developed into a
project.
24
Project Selection Exercise
Target group Needs of people Priority Technical aspect Social risk factors Environment aspects Inputs Economic aspects Probability of achieving goals Relationship with other projects
25
Project Selection Criteria
Example
26
Core Objective
1. School infrastructure
Children complete school.
2. Training
3. Income generation
1. School infrastructure 2. Training 3. Income generation
27
Example: Project Options
Example: Project Selection Criteria
1. School infrastructure
2. Training 3. Income generation
Size of target group
+++ ++ +
Needs of people
+++ +++ +++
Priority + +++ +
28
Example: Project Selection Criteria
1. School infrastructure
2. Training 3. Income generation
Technical aspects (new technology)
+ + ++
Social risk factors
+++ + +++
Environment aspects
+++ + ++
29
Example: Project Selection Criteria
1. School infrastructure
2. Training 3.Income generation
Inputs + + ++
Economic aspects +++ + +++
Probability of achieving goals
+++ +++ +
Relationship with other projects
Duplicated with ODA projects
New approach
Duplicated with NGO project
30
1. Training2. School infrastructure3. Income generation
31
Project Selection Priority
Compare all approaches using the selection criteria
Discuss advantage and disadvantage of different approaches
Decide an approach
32
Summary of Project Selection
33
4. Project Logic- Making a Logical
Framework
34
Now you have
Objective TreeBranch: Project
Approach
35
Now we are going to
Branch:Project Approach
Pyramid:Project Logic
36
Project Logic is like Pyramid
Strategic Result
Key Result
Key Result
Key Result
Key Result
37
Remember your Problem Tree
Positive Ends
Core Objective
Solutions
38
Change to Project Design Pyramid
Positive Ends
Core Objectives
Solutions
Project Goal
Project Outcome
Outputs
Activities
Inputs
39
Project Design Pyramid
Project Goal
Project Outcome
Outputs
Activities
Inputs
Module 2-1 40
LOGIC MODELS
Outcomes
Impacts
Outputs
Activities
Inputs
Resources used to implement activities (financial, materials, human)
Inputs are what is needed for the Activities
Question: In Real Life, what may be the inputs needed to build a bridge?
Answer: Laborers, cement, steel beams, etc.
41
Inputs
Activities daily efforts needed for the outputs (and then objectives and goals).
Question: In Real Life, what may be the activities involved in building a bridge?
Answer: Identify plan, lay foundation, build structure, etc.
42
Activities
Outputs are the products needed to achieve the objective.
They are the deliverables.
Question: In Real Life, what may be the product you want in building a bridge?
Answer: completed bridge connecting two different towns.
43
Outputs
Outcomes are the major CHANGES needed to achieve the goal.
Question: In Real Life, what may be the primary outcome you want to achieve with the bridge?
Answer: Increase trade and between two towns.
44
Outcomes (Objective)
Simple, clear statement of the IMPACT or RESULT to achieve with by the project.
Change we hope to bring in the community.
Can be bigger than the project itself.
Question: In Real Life, what may be the Overall Reason you want to build a bridge between 2 communities?
Answer: Improve economy in two towns.
45
Goal
Narrative Summary
Project Goal
Project Outcome
Outputs
Activities
46
Project Logic Tool: Logical Framework
At this point, you should have your own “project logic”
47
Summary of project logic
48
What is the second column in the Logical Framework?
INDICATORS
Indicators
Project Goal
Project Outcome
Outputs
Activities
49
50
Indicators are like the dashboard of a car.
51
Indicator Dashboard
SpeedPetrol
Odometer 12400km
OilHow fast are we going?
How much did we do?
Are people happy with our work?
How many resources did we use to get there?
Not Enough Enough
Example: Training Project
Project Goal Ensure a protected environment for children.
Project Outcome Children complete school.
Outputs Teachers’ skills are updated.Education managers have appropriate planning and management skills.
Activities Conduct participatory teaching methods training for elementary school teachers.Conduct school management training for education managers in elementary schools.
52
Example: Training Project
Narrative Summary Indicators
Project Goal Ensure a protected environment for children.
Number of children stay with their care-givers.
Project Outcome Children complete school. Number of children completed school.
Outputs Teachers’ skills are updated.Education managers have appropriate planning and management skills.
# of teachers who receive a certificate.# of managers who updated management skills.
Activities Conduct participatory teaching methods training for elementary school teachers.Conduct school management training for education managers in elementary schools.
# of training conducted.
53
54
Standard Indicator Levels
Project Goal
Project Outcome
Outputs
Activities
Impact
Outcome
Output
Process
Input Input
Measures ultimate change desired in people or a condition (not always feasible or necessary).
Measures key changes necessary for goal
Measures the immediate product of an activity
Measure the activities (process), i.e. attendance at the activities
Measures the resources used for the activities.
Impact Indicator Outcome Indicator Output Indicator Process Indicator Input Indicator
55
What are the indicators for making a bridge?
Specific (Who & What?)Measurable (How much?)Area specific (Where?) - Achievable
Relevant – RealisticTime-Bound (When?)
56
SMART Indicators
Economic wellbeing of households restored and construction of new houses.
Compound Indicator!!!
57
What’s wrong with this Indicator?
Satisfaction in this workshop.
SMART: # participants in the Planning workshop (date) who report they would recommend the workshop to others.
58
Example: How can we make this Indicator SMARTer?
Quantitative - Can be directly counted and expressed as a number◦ % of……◦ # of ……◦ Frequency of……◦ Ration of ……◦ Amount of …….◦ Timeliness of ……..
Qualitative – Involve perception ( can be expressed quantitatively or as narrative)◦ Level of congruence with◦ Satisfaction with…..◦ Knowledge of…..◦ Ability to…. ◦ Appropriateness of ….◦ Importance of……
59
Indicator Examples
Objective Tree Branch (Project Selection) Pyramid (Project Logic)
Project Logic (Inputs, Activities, Outputs, Outcome, and Goal)
SMART Indicators for each Level.
60
Summary of Logical framework
Expected Results Statements Indicators Means of Measurement / Verification
Assumptions/Risks
Goal – Link to SP. The programme is expected to contribute towards the achievement of this result, collectively with other development partners.
Outcome 1 – Link to SP. The programme is expected to contribute towards the achievement of this result, collectively with other development partners.
Link to relevant SP indicators.
Output 1.1 – The programme will be directly held accountable for this result.
X – not needed
Activity 1.1.1 X – Not needed X
Activity 1.1.2 X X
Activity 1.1.3 X X
Module 2-1 61
UNIFEM LOGFRAME
62
Questions?
Session 2:
Apply the Logical Framework to develop a Monitoring and Evaluation (ME) Plan
63
To apply the skills to develop a Monitoring and Evaluation (ME) plan
1. Monitoring – What to monitor?2. Evaluation – What to evaluate?
64
Objectives of Session 2
65
COMPLEMENTARY ROLES OF MONITORING & EVALUATION
MONITORING
Clarifies program objectives
Links activities & resources to objectives
Translates objectives into performance indicators & sets targets
Routinely collects data, compares actual results with targets
Reports on progress & identifies problems
EVALUATION
Analyzes why intended results were or were not achieved
Assesses specific causal contribution of activities to results
Examines implementation process
Explores unintended results Provides lessons, highlights
significant accomplishment or program potential & offers recommendations for improvement
Rist & Kusek, 2004Ten Steps to a Results-Based M&E System
Module 2-1 66
LOGIC MODELS
Outcomes
Impacts
Outputs
Activities
Inputs
Monitoring
Evaluation
1. Monitoring – What to monitor?
67
Expected results
Indicators
Sources/Means of verification
Baseline Collection methods
Responsibilities
Goal
Outcome
Outputs
68
UNIFEM Monitoring Framework
A description (qualitative or quantitative) of the situation prior to the intervention against which progress can be assessed or comparison made
Used as a benchmark for assessing programme induced outcomes or impacts
Often the first data collected for an indicator is the baseline
Baseline data are gathered before or shortly after project implementation begins
69
Baseline
The starting point Data and information Related to results (outputs, outcome and
impact)
70
Baseline
2. Evaluation – What to evaluate?
71
Making a decision Enable managers and partners to make
decision and plan strategically Programme improvement, knowledge
generation and accountability
72
Why evaluation?
What you want to learn about the programme
What you need to know to make the programme decisions you need to make.
73
How to decide an evaluation type
Purpose of evaluation Audience Information to decide Sources How to collect the information When Available resources
74
Points to condider
Corporate & Decentralized EvaluationsCorporate & Decentralized Evaluations
Evaluation Sub-Types Evaluation Sub-Types Evaluation Modalities Evaluation Modalities
Scope Timing Unit of Analysis
Management Conduct
Output
Outcome
Impact
Impact
During: Evaluability assessment; Formative; Mid-term
After: Final; Summativ
e
Sometime after: Ex-
post
Institutional
evaluation
Policy, strategy, plan evaluationProject programme
Cluster thematic
Meta evaluation
Individual evaluation
Joint evaluation
External evaluation
Self evaluation
Peer evaluation
Terms Definition When used Questions
Output evaluation
Goods and services (results of inputs)
Useful when preceded by formative evaluation.
Deliveries in a timely manner?
Outcome evaluation
Changes (behavioral change) by intervention
Immediately following the end of an activity (short term influence)
Achieve results? Unintended results? Make a difference?
Impact evaluation
Broad, long-term impacts or effects
After a programme has concluded
Effect of programme outcome? How much better off?
76
By Scope
UNIFEM programm
e cycle
Terms Definition When to use
Questions
Formulation Ex-ante Predict the likelihood
Formulation Will the activities meet the needs?
Before or during implementation
Evaluability assessment
Meaningfully evaluated
Before any evaluation takes place
Gender equality identified? Objectives realistic?
77
By Timing
UNIFEM programme cycle
Terms Definition When to use
Questions
Evaluation phase
Formative evaluation
Early insights
During implementation
What is happening?
Mid-term evaluation
Half-way During implementation
Is it working well?
Closing phase
Final evaluation
End implementation
Immediately the end of an activity or intervention cycle
Achieving?
Closing phase
Summative and ex-post evaluation
Some time after the programme
Concluded or a year after
Making difference?
78
By Timing
Formulation
Appraisal
Approval
Implementation and Monitoring
Evaluation
Closing
79
Timing in UNIFEM Programme Cycle Ex-ante
Summative
and ex-post
Formative, mid-term,
outcome, impact
Before and during:
evaluability assessment
Terms Definition How to use
External evaluation External specialists Objective analysis
Self evaluation Periodic reviews to assess the achievements and constraints
By responsible for implementation, often used in smaller projects or as the first step
Peer evaluation Composed of external evaluators and programme staff
Formative, combine internal understanding with external expertise.
80
By Conduct
Mandatory evaluation Investment
A final evaluation during programme life cycle
Programme with budget over USD 1 million
A mid-term and final evaluation during programme life cycle
Programmes with budgets over USD 3 million
Mandatory evaluation
Need a quality Logical Framework Decide which level you monitor and
evaluate Use UNIFEM Monitoring Framework Think about baseline from the beginning Decide what to evaluate and which level of
the Logical Framework Evaluation is to make a decision about the
programme.
82
Summary of develop a ME plan
83
Questions?
84
Thanks!