Experiences in Finland on self-evaluation of Local Action Groups

Seminar on Monitoring and Evaluation of the LEADER Approach

Brussels, 25 September 2005

Päivi Pylkkänen, University of Helsinki,

Ruralia Institute for Rural Research and Training, Mikkeli paivi.pylkkanen@helsinki.fi

Overview Definitions and characteristics

Findings on the Finnish LAGs’ self-evaluations from the perspective of MTE

Measures taken to improve the usefulness of the LAGs’ self-evaluations for the LEADER+ MTE

Conclusions and recommendations

Definitions and characteristics - SE vs. PE

LAG self-evaluation LEADER Programme evaluation

Evaluative activity carried out at

the level of a Local Action Group, not

necessarily throughout by the LAG

Principally guided by a LAG’s own

needs and interests

Recommended, yet a voluntary

activity for a LAG

Usually more geared towards

learning and corrective actions than

accountability (formative)

Rather standardised activity carried

out at the level of the LEADER

Programme

Guided by the EU regulations and

Guidelines on M&E, together with

national programme level interests (top-

down needs and interests)

Usually more geared towards

accountability than learning (i,e

summative); However, contains also

learning objectives (policy level)

The challenge from the Commission Guidelines for the Evaluation of LEADER+

(2002): Mid-term evaluations of the LEADER+ programme should, to the extent

possible, draw on LAG self-evaluations

Findings on the Finnish LAGs´self-evaluations

Self-evaluation recommended for all LAGs in the Finnish

LEADER+ Programme (2000-2006) Training on methods for (self-)evaluation provided by the MoA

and the national network unit training sessions as from 2002,

also trainings for individual LAGs by evaluation specialists Occurrence of planned and documented systematic self-

evaluation grown from 25% of LAGs in 2003 to over 50 % by

2005, while all LAGs conduct some kind of evaluative self-

reflection Varied focuses of the LAGs’ self-evaluation activities cover e.g.:

LAG’s internal board working processes

Client satisfaction and feedback on the LAGs’ services

Stakeholders’ assessment on the LAGs’ impact

Findings on the Finnish LAGs´self-evaluations (cont.)

Only about one fourth of the LAGs’ self-evaluations sufficiently

documented from the point of view of programme level mid-

term evaluation Following from the different key focuses of the self-evaluations

and the programme evaluation, the former proved to be of

limited usefulness for the MTE, particularly so in the 2003 MTE The above findings called for measures to improve the

usefulness of the LAGs’ evaluative activities for the mid-term

evaluation (for the 2005 MTE update and future evaluations)

Measures taken to improve the usefulness of the LAGs’ self-evaluations for the MTE

Further evaluation capacity building through national training workshops, material, and tailored training sessions for individual LAGs

Independently from diverse self-evaluations, reviewed guidelines for the LAGs’ annual reporting as from the year 2004 to serve as a foundation for the MTE update 2005:

Aiming at more reflective, self-evaluative annual reporting by each LAG

Merging mere standard recording and reporting with evaluative flavour

Focus on selected common topics reflecting the overall MTE evaluation questions, e.g. - Realisation of the LEADER –method, particularly bottom-up, networking..

- Analysis of the LAG’s innovative results and processes

- The long-term impacts of completed LAG projects (case studies!)

…as perceived, argued and supported by relevant evidence by LAGs themselves

Conclusions and recommendations In the Finnish experience the LAGs’ self-evaluations

and the programme evaluations can be mutually complementary:

Some data best collected and analysed at the level of LAGs,

some at the level of the programme LAG self-evaluations and reporting form a set of data for the

programme level evaluations for further analysis, including

their critical review Programme evaluations provide data for the LAGs self-

reflection (For example in Finland, results from

comprehensive questionnaires for funded projects and the

stakeholders of the programme, broken down by each LAG,

proved useful and interesting for LAGs as self-evaluation

material)

Conclusions and recommendations (cont.)

To be more useful for programme evaluations, LAGs’ self-evaluations should be:

documented (both on methods and results) focused on (selected) common evaluation questions,

particularly those related to the LEADER –method and value added of the LAG/the LEADER method in its area of operation

systematic in terms of data used as evidence (the burden of evidence on reported result or impact claims rest with the LAGs)

Mutual transparency and dialogue on evaluation plans crucial for optimal use of resources

Beware: The top-down interests of the programme evaluations should not jeopardise the LAGs’ genuine self-evaluations geared towards LAGs own learning needs;resources needed for both interests!