The Hungarian system of ex post and on-going evaluation focusing on Structural Funds

Kinga Kenyeres, Evaluation Division 6-7 May, 2010

National Development Agency

The present situation of evaluation – Evaluation

• Evaluation is a centralized activity in the National Development Agency

• Separated financial source for evaluation (Implementation OP)

• Capacity building, cost planning based on The Evaluation Plan,

• Strong cooperation with Managing Authorities• In the focus of the activity:

– Project management• ToR• Legal affairs (public procurement procedures)• Project managing of outputs (Inception Report,

Semi-final Report, Final Report)• Communication about evaluational results

– Messages towards policies – Development Policy Academy - Conferences

every 3 months. Goal: extending evaluation culture, sharing experiences

The present situation of evaluation – Evaluation Diviso

• Evaluation is a centralized activity in National Development Agency

• Separated financial source for evaluation (Implementation OP)

• Capacity building, cost planning based on The Evaluation Plan,

• Strong cooperation with Managing Authorities• In the focus of the activity:

– projectmanagement• ToR• Legal affairs (public procurement procedures)• Project managing of outputs (Inception Report,

Semi-final Report, Final Report)• Communication about evaluational results

– Messages towards policies – Development Policy Academy – Conference

every 3 month. Goal: extending evaluation culture, sharing experiences

Mission

Evaluation on using of the Cohesion Fund and the Structural Funds

Provide feedback on evaluation results towards Units of execution and planning,

Active role in using the EU funds more effectively

Goals

1. The Evaluation activity• Provides information for decision-making• Provides suitable methods for

evaluational topics• Perfoms high quality• Is cost-effective2. Transforms the results of the

evaluations into messages and forwards them to Stakeholders

Decision-making support

• Policy planning, level of program coordination– Takes into account the results, impacts of

former measures (Ist National Development Plan, 2004-2006),

– Horizontal disciplines– The ex ante evaluation of OP-modification– Support to ActionPlanning

• Supporting execution– Fine tuning of measures– Practical recommendations in order to improve

processes and institutional activities „Service” approach

Rising demand for on-going evaluations

Valid Methodology• Ex post evaluation

– Econometrical methods based on databases and primer data collection – impact assessment

– Document analysis, interviews – helps to understand the results of the impact assessment, takes them into a context

• On-going evaluations– Monitoring database analyses– Process- and organizational analyses

• Ex ante evaluation– Support to the planning procedure– The goal is impoving the methods of

former impact assessment

Professional credibility, objectivity

High quality, cost efficiency

Evaluation Framework1.roundSelecting evaluators (Autumn 2008 – spring 2009)• Public procurement procedure for 4 lots

within the framework of a 4-year-contract• Technical, financial eligibility on the basis of the

strict qualification of experts• Comittee for decision including colleagues from

Managing Authorities

Evaluation Framework - 4 LOTs

1.Environment (includes: strategic environmental assessment)

2.Infrastructure (transport, energy) 3.Economic and social development

(education, health, employment, R+D and innovation)

4.Operative evaluations (includes: system of institution development)

High quality, cost efficiency

2. roundProcurement of Evaluations through

the Evaluation Framework (Spring 2009 – end of 2013)

• In 4 LOTs: 6 evaluator consortiums compete with each other for one specific evaluation task

• Call for proposal of evaluators based on well-defined TOR and defined number of expert-days

• The price is the only criterion for selection.

High quality, cost efficiency

High Professional Responsibility of Evaluation Unit

Communication of evaluational results 1. Built in the evaluation process

• Cooperation with Managing Authorities• Professional discussions in professional level and in management

level – ToR– Evaluation– Project Managing Committee

2. Following the evaluation process

• Publish evaluational final reports, open opportunity to comment on NDA homepage

• Conferences/ workshops

• Inform European Commission and other member states: Executice Summary in English

• Inform Monitoring Committees

The goal is to communicate clear messages

Reaching the goals

Evaluation Plan– 3-year plan –review every year

– Capacity building and cost plan based on Evaluation Plan.

– Evaluation Plan is based on the OP execution chapter and the Interactive Executing Handbook of NDA.

– Eval Plan has been accepted by the Implementation OP Monitoring Committee

– Inputs• Legal requirements • Information demand of the organization• Requirements of other institutional stakeholders• OP Monitoring Committe opinion• European Commission opinion• Currently: requirements of the forthcoming Hungarian EU

presidency– 2010 - shift from ex post evaluations to on-going ones

Rising rate of on-going evaluations, because– Rising interest in facts of implementation and

coordination– The findings of ex post evaluations cannot be used

easily during implementation Pros:

– Increasing overall demand for evaluation– Stakeholders are more and more concerned in

having proper quality evaluations Cons:

– The very sophisticated methodology of ex post evaluations cannot be used

– We have more experience in ex post evaluations

Evaluations of the Division

The balance between ex post and on going evaluations is a MUST!

Evaluations in the Evaluation Division

The Division launched 11 evaluations in 2009 (9 ex posts and 2 on-goings) and 12 in 2010 (8 on-goings and 4 ex posts)

Párhuzamosan futó értékelések száma

02468

10121416

Párhuzamosanfutó értékelésekszáma

Evaluations in the Evaluation Division

2010 értékelései téma területek szerint

1

1

1

1

1

1

1

2

3

4

4

5

6

6

9

0 1 2 3 4 5 6 7 8 9 10

Energy

Horizontal - equal opportunity

Horizontal - sustainability

Horizontal - handicapped people

Health

Social inclusion

Partnership

Monitoring support

Institutional

Infrastructure

Environmental issues

Labour policy

Public administration

Combined

Economy

2010 értékelései téma területek szerint

Our on-going evaluations:• Evaluation on the financing system of the

Intermediate Bodies of the NSRF - finished

• Evaluation of the action planning process

• Evaluation of the selection procedures of the NSRF

• OP mid-term reviews– Strategy-focused mid-term evaluation of the

implementation – Provides advice to optimize further

implementation

Evaluations in the Evaluation Division

Thank you for your attention!