1 jela tvrdonova, 2014. strategic approach to rural development common approach to evaluation:...
TRANSCRIPT
Common monitoring and evaluation framework for
evaluation of rural development programmes
1
Jela Tvrdonova, 2014
Strategic approach to rural development Common approach to evaluation: legal
background and CMEF Monitoring and evaluation – what we are
talking about? What is evaluated? Intervention logic in evaluation Indicators Evaluation questions CMEF guidance
2
Content of the presentation
Focus on limited number of objectives Competitiveness – Environment – Quality of life
Axes connected with strategic objectivesStrong and dynamic agri-food sectorAgriculture and forestry with high added
valueEmployment and growth in rural areasImprovement of the governance in rural
areas and mobilisation of the endogenous potential
Strategic approach to rural development
Common approach to monitoring and evaluation Exact definition of
objectives in Strategic Guidelines for Rural Development, Council Regulation 1698/2005 (art. 77-87), EC Regulation 1974/2006 (art. 60-62), Health check,
National strategic plans and EU strategic monitoring
Single framework for all program interventions
CMEF: Common monitoring and evaluation framework – Handbook, Annexes, Guidance notes
Baseline indicators at the program start
Suitable combination of input, output, result and impact indicators,
Common evaluation questions for all RD programs
Guidance for ex ante, mid-term, ex post and ongoing evaluation
Monitoring – measurement of immediate outputs and results at measure and axis level (different from strategic monitoring)
Evaluation - measurement of long-term effects/impacts of intervention, within the program development context (rural areas)
5
Monitoring and evaluation – what we are talking about?
Accountability, comparability and lessons learned for next
interventions
On-going process which monitors the gradual implementation of the program at the level of financial inputs, physical outputs and results
Subject of annual reporting Instruments – common and programme
specific indicators linked to outputs and results,
Monitoring
Since 2010 and every other two years Focus of the EC on the progress of national
strategic plans and its objectives implementation
Assesment of the contribution of national plans to the EU Rural Development Strategy
Strategic monitoring
On-going evaluation is based on CMEF and country´s own methodology to carry on the evaluation during the program implementation
Process based on annual reporting on result/impact indicators including periodical exercises:◦ Ex – ante: evaluation of relevance of the planned
intervention and optimizing of the RDP budget,
◦ Mid-term: for the assessing the progress related to the specific and overall/program objectives,
◦ Ex post : for the assessment of impacts and
acievements towards objectives.
Evaluation
WILL PROGRAMME WORK AS IT SHOULD?
IS PROGRAMME WORKING AS IT SHOULD?
DID PROGRAMME WORK AS EXPECTED?
Good programme is the basis for the high quality of monitoring and evaluation
Normally the monitoring and evaluation is linked to programme intervention logic
But also specific topics, such as:◦ Technical assistance◦ Administrative arrangements◦ Delivery mechanism
9
Monitoring and evaluation of programme
Intervention which logically responds to the most important needs of the targeted area
4/5 June 2009Challenges of the CMEF &
Ongoing Evaluation 10
What is an intervention logic?
Hierarchy of objectives: • overall, • specific, • operational
Hierarchy of expected effects:
• impacts on the territory, • results on the supported beneficiaries • immediate outputs at project level
4/5 June 2009Challenges of the CMEF &
Ongoing Evaluation 11
Intervention logic of EU rural development programmes
RDP Intervention logic
Context , its descriptionSWOT and needs assessment
Relevance
Efficiency
Overall objectives EU/MS Programme level
Specific objectives EU/MS Axis level
Operational objectives EU/MSMeasure level
Impacts
Results
Outputs
Effe
ctiv
enes
s
Measures, projects and their management and implementation
Inputs
Coh
eren
ce
EU policy objectives
Complementarity
Source: EENRD 2014
tools to measure the programme effects and achievements towards expected objectives by measures or the whole programme
should be Specific, Measurable, Achievable, Relevant for the programme, and Timely (SMART)
Indicators should be filled with quantitative statistical data; however in some cases, indicators might also be linked to qualitative assessments or logical assumptions
13
Role of indicators
Baseline indicators: they relate to general socio-economic context of the programme area (context-related baseline indicators) and to the state of the economic, social or environmental situation in direct relation with the wider objectives of the programme (objectives-related baseline indicators)
Financial execution (input) indicators: they refer to the budget or other resources allocated to the programmes
Output indicators: measure activities directly realized within programmes
14
Types of indicators
Result indicators: measure the direct and immediate effects of the intervention within the group of programme beneficiaries and provide information on changes that have taken place
Impact indicators: refer to the benefits of the programme both at the level of the intervention but also more generally in the programme area. They are linked to the wider objectives of the programme
15
Types of indicators
Indicators are linked to process of programme implementation
Inputs
Activities
Result
Ouputs
Impact
A common set of baseline, output, result, and impact indicators for the RDPs (Art. 62 Reg. 1974/2006) “shall form the Common Monitoring and Evaluation Framework (CMEF)’’ (Annex VIII lists the common indicators)
17
Common indicators
Since common indicators may not fully capture all effects of programme activities, it is necessary to define additional programme specific indicators (see Guidance notes A & K), which relate to programme specific issues.
They are developed if:◦ Programme contains programme specific objectives and common indicators
are not sufficient to capture achievements◦ There is the need to evaluate specific issues such as delivery mechanism etc.
18
Programe specific indicators
Define the focus of evaluations
Demonstrate the progress, impact, achievements, effectiveness, efficiency and relevance of rural development policy
Common Monitoring and Evaluation System distinguishes EQs:
a) Common Evaluation Questions for Rural Development (CEQ-RD)
b) Programme-Specific Evaluation Questions (PSEQs)
19
Purpose of Evaluation Questions
Source: EENRD 2014
20
Purpose of Evaluation Questions
Demonstrate the contribution of programme interventions
Evaluation of matters relevant at the EU policy
level
Encourage the assessment of programme results and impacts
Enhance comparability across RDPs
COMMON EQS FOR RURAL
DEVELOPMENT
Evaluation of specific RDP related topics
PROGRAMME SPECIFIC EQS
Evaluation of matters relevant to the programme-
specific policy
Source: EENRD 2014
21
Clear and straightforward evaluation framework
Evaluation questions and judgment criteria
Policy objectives
Indicators
Triangular consistency between objectives, questions and indicators which allows for:
• Less EQ and more targeted towards policy
• Clearer formulation of EQ and harmonization of all terms used
• More evidence to answer EQ
• Better planning the evaluation
Source: EENRD 2014
1. Common Evaluation Questions Linked to RD policy objectives Causal-effect questions (To
what extent..?)
22
How are the CEQs developed?
2. Common judgment criteria
Set the foundations to assess the success of intervention
Formulate explicit judgments on the basis common terms
Facilitate the identification of data, information and analysis needed3. Common RD indicators
Additional information when necessary
1. Development of CEQs linked to the
RD policy objectives
2. Development of common
judgment criteria
3. Identification of relevant common rural development
indicators
Source: EENRD 2014
HandbookAnnex 1: Guidance notes - A: Choice and use of indicators,
B: On-going evaluation , C: Mid-term evaluationAnnex 2: Guidance notes – D: Hierarchy of objectives, E –
Measure FichesAnnex 3: Guidance notes – F – K, Common indicators
FichesAnnex 4 Guidance J – O Other guidances
23
CMEF – guidance
Original set of 150 common evaluation questions of CMEF has been simplified and reduced to the essential demand for knowledge from the European perspective.
24
Set of common evaluation questions (ex post guidelines)
3 groups of CEQs
Programme-related:Community strategic priorities
Health Check objectives7 impact indicators
TA and NRNEfficiency of RDP resources
Leader approach-related: Community strategic priorities:Employment, diversification
and governanceLeader approachLAG contribution to LDS and RDP
Measure-related:Contribution of the measure
to its axis objectivesOther measure effects
and contribution to other axes objectives
Source: EENRD 2014
M&E framework
25
Evaluation Question Indicators
Evaluation questions – cross-cutting
Evaluation questions Axis/sector specific
Output indicators
Impact indicators
Result indicators Axis based baselines, objective related, contextual
Baselines and their quantification,
Target levels
Horizontal baselines, objective related, contextual
RDP objectives
Overall strategic objectives
Axis specific objectives
Measure/operational objectives
Baseline values of
output indicators
Targets for impact indicators
Targets for results indicatiors
Targets for output indicators
Common and program spec. Eval. questions
CMEF provides guidance for MAs in setting up indicators at each level of intervention and baseline indicators
Annex 3 of the CMEF Handbook provides detailed description of all common indicators (Guidance notes F – K):
◦ F: COMMON INDICATOR LIST (overview of all common indicators)
◦ G. BASELINE INDICATOR FICHES (detailed description of indicators)
◦ H. OUTPUT INDICATOR FICHES◦ I. RESULT INDICATOR FICHES◦ J. IMPACT INDICATOR FICHES
26
CMEF guidance on indicators
Each indicator fiche contains the following elements: Type of indicator Related measures Measure Codes Definition of the indicator Subdivision Unit of measurement Level of collection Responsible actor for collection Collection method/good practice Sources Registration frequency
27
Indicator Fiches
Guidance on practical use of indicators in monitoring and evaluation of RDPs
Guidance note E provides the following on use of indicators within each particular measure:
◦ Measure Code◦ Rationale of the measure◦ Target group◦ Target area◦Common indicators ◦Link rationale of the measure and
indicators◦Evaluation questions
28
Measure Fiches
29
Ex post evaluation guidelines
Mainly Managing Authorities
Introduces the process of the ex post evaluation, steps to be conducted and role of evaluation stakeholders
Explains also specificities with respect to NRN
Part I
Mainly Evaluators
Discusses and explains intervention logic, evaluation questions, indicators, methods and data
Part II
Toolbox
Provides additional practical tools for ex post evaluation preparing, implementation and reporting.
Part III
For everyone
Introduces ex post evaluation and its role in policy cycle
Explains the scope and focus of ex post evaluation, legal requirements and common evaluation elements
Introduction
Source: EENRD 2014
RDP Intervention logic
SWOT and needs
assessment
Efficiency
Overall objectives
EU/MS
Specific objectives
EU/MS
Operational objectives
EU/MS
Impacts
Results
Outputs
Measures and their
implementation
Evalu
ati
on
quest
ions
Ind
icato
rs
Methods
Data
EU policy objectives
Evaluation. What it is about?
Basis of evaluation
Attribution of impacts
Collection of evidence
Focus of evaluation
Measurementtools
Source: EENRD 2014
