1

VI/8865/99- Rev.

Evaluation

of rural development programmes 2000-2006

supported from

the European Agricultural Guidance and Guarantee Fund

Guidelines

Directorate General for Agriculture

1999

2

1. INTRODUCTION ....................................................................4

2. SCOPE OF EVALUATION ......................................6

2.1. Types and purposes of evaluation................................................................... 6

2.2. Regulatory scope............................................................................................ 6

2.3. Temporal coverage......................................................................................... 6

2.4. Geographical coverage................................................................................... 7

3. SETTING UP THE EVALUATION:

GENERAL CONCEPTS.................................................8

3.1. Designing the evaluation system .................................................................... 8

3.2. The intervention logic .................................................................................... 9

3.3. Examining the impacts of the programme .................................................... 11

4. EVALUATION QUESTIONS,

CRITERIA AND INDICATORS ...................12

4.1. Common evaluation questions...................................................................... 12

4.2. Questions for individual programmes........................................................... 13

4.3. Using appropriate criteria and indicators ...................................................... 13

5. EVALUATION METHODOLOGY ..........19

5.1. Use of recognised methods........................................................................... 19

5.2. Data collection ............................................................................................. 19

5.3. Data analysis ................................................................................................ 20

6. REPORTING ON THE

EVALUATION ........................................................................22

6.1. Regulatory requirements .............................................................................. 22

6.2. The recommended common structure for the reports .................................... 22

7. ADMINISTRATIVE MATTERS ..................24

7.1. Responsibility for evaluation........................................................................ 24

3

7.2. Terms of reference ....................................................................................... 24

7.3. The independent evaluator ........................................................................... 25

7.4. Follow up to the evaluation report................................................................ 25

7.5. Co-financing ................................................................................................ 25

7.6. Assessment of the quality of evaluation reports ............................................ 26

7.7. Timetable for evaluation .............................................................................. 27

Annex I - Common evaluation questions 28

Annex II - The recommended common structure for the reports 32

Annex III - Ex-Ante evaluation 36

Annex IV - Glossary of evaluation terms 39

Annex V - Reference documents 42

4

1. INTRODUCTION

The present document offers information about evaluation in the context of ruraldevelopment. It is aimed at national or regional authorities in charge of programming

and evaluation of rural development in the period 2000-2006 supported from theEuropean Agricultural Guidance and Guarantee Fund (EAGGF).

Rural development evaluation must provide information on the implementation andimpacts of the co-financed programmes. The aims are, on the one hand, to increase the

accountability and transparency with regard to the legal and budget authorities and thepublic and, on the other hand, to improve the implementation of the programmes by

contributing to informed planning and decisions concerning needs, delivery mechanismsand resource allocation.

More specifically, evaluation helps to assess key aspects of the assistance (i.e., therelevance, effectiveness, efficiency, utility, sustainability of the assisted actions,

depending on the stage of programme implementation) in relation to the generalobjectives of the rural development policy at Community level, notably regarding the

Common Agricultural Policy, as well as in relation to the specific needs and prioritiesincorporated into each rural development programming document.

The overall requirements concerning evaluation are set out in Article 43(1) and above all

Article 49 of the Council regulation1 on support for rural development from the EuropeanAgricultural Guidance and Guarantee Fund (hereafter referred to as the Council

regulation). They specify that the evaluation of measures covered by rural developmentprogramming shall be carried out on the basis of the principles laid down in Articles 40

to 43 of the Council regulation2 laying down general provisions on the Structural Funds.

They also allow the EAGGF Guarantee section to co-finance evaluations.

These general rules are substantiated in Articles 42-45 of the Commission regulationlaying down the detailed rules for the application of Council regulation on support of

rural development from the EAGGF3 (hereafter referred to as the implementing

regulation), which specifies how evaluations must be carried out at the ex ante, mid term

and ex post stages of the program. These articles describe the procedures,responsibilities and deadlines for evaluations as well as their contents. They also require

the elaboration of common evaluation questions and of a recommendation for thestructure of evaluation reports. Furthermore they contain a requirement concerning the

use of recognised practice both for the evaluation itself and for the assessment of thequality of evaluation reports. Article 40 of the implementing regulation describes the

conditions under which the EAGGF can co-finance evaluations in the Member States.The present guidelines provide further details on the above requirements of the

implementing regulation and offer brief complementary information about evaluation

1 Council regulation (EC) No 1257/1999 of 17.05.1999 on support of rural development from the European

Agricultural Guidance and Guarantee Fund (EAGGF) and amending and repealing certain Regulations, OJ L160, 26.06.1999, p.80

2 Council Regulation (EC) No 1260/1999of 21.06.1999 laying down general provisions on the structural funds, O.JL 161, 26.06.1999, p. 1

3 Commission regulation (EC) No 1750/1999 of 23.07.1999 laying down the detailed rules for the application ofCouncil regulation (EC) No 1257/1999 on support of rural development from the European AgriculturalGuidance and Guarantee Fund (EAGGF), OJ L 214, 13.08.1999, p. 31

5

concepts and methodology relevant for rural development evaluation. [Finally, Article 6

of the Commission Regulation laying down transitional rules for rural development4

(hereafter referred to as the transitional rules) provide the opportunity of paying ex ante

evaluations carried out prior to the submission of the plan if paid after 31 July 1999].

4 Commission regulation (EC) No 2603/1999 of 9.12.1999 laying down rules for the transition to the rural

development support provided for by Council Regulation (EC) No. 1257/1999, OJ L 316, 10.12.1999, p. 26.

6

2. SCOPE OF EVALUATION

2.1. Types and purposes of evaluation

The implementing regulation prescribes mandatory evaluation at the ex ante, mid

term and ex post stage of the programming cycle. The major purposes of theseevaluations are described in Box 2.1.

The implementing regulation also provides the option of an update of the mid termevaluation, which can facilitate the well organised and efficient closure of the

programming period, assist the follow-up to it where appropriate and contribute toearly reporting.

Box 2.1

Ex ante evaluationThe ex ante evaluation helps to prepare the rural development plan and facilitates itsimplementation. In particular it helps in clarifying the objectives, their relevance to the needs

and in assuring consistency between the proposed strategy and the selected targets with the

existing situation in the region or sectors concerned. A proper ex ante evaluation facilitates theCommission’s task of appraising the rural development plan.

Mid-term evaluation

The mid-term evaluation helps to reorientate the programme if necessary and improve the

implementation. It can also provide important information to authorities with general

responsibilities in relation to the programme.

Ex post evaluationThe ex post evaluation recapitulates and judges an intervention when it is over. It is central for

the accountability and the transparency of the interventions with regard to the legal and budgetaryauthorities and the public. It can also give guidance about a possible follow-up to the

programme, e.g., in the form of best practices.

2.2. Regulatory scope

The requirement to evaluate concerns rural development assistance co-financed

from the EAGGF. The present guide applies in the first instance to the assistancefinanced by the Guarantee section of EAGGF. If, in Objective 2 areas, certain

measures (excluding the four measures mentioned in Article 35(1) of the Councilregulation) have been subject to integrated programming together with those of the

Structural Funds, other procedures may apply for reporting on the evaluation ingeneral. However, the common evaluation questions (at present summarised in anindicative manner Annex I) should normally be answered where this type of action

is a significant part of the programme. The same applies, mutatis mutandis, toassistance from the Guidance section of EAGGF in Objective 1 areas.

2.3. Temporal coverage

The evaluation concerns the new rural development programming period, which in

principle covers the period from 2000 to 2006. However, where necessary, accountmay be taken of actions previously initiated, for example, follow-up to actions

undertaken under the previous objectives 5a, 5b or mainstreaming within theprogramme of previous LEADER activities.

7

More specifically, the mid-term evaluation considers the period 2000-2003, while

the ex post evaluation examines the entire implementation period.

2.4. Geographical coverage

The evaluation must cover the geographic area of the rural developmentprogramme, which has been drawn up at the most appropriate geographical level,

according to Article 41 of the Council regulation. However, it must also be takeninto account that certain measures may not apply throughout the whole region (cf.,

Point 6(3) of the Annex to the implementing regulation) or may have a different“degree of targeting in terms of geographical, sectorial or other coverage” (cf., Point

9(3) VI(A) of the Annex). Any other need for geographical differentiation (cf.,Articles 15 and 29-31 of the Council regulation), local disparities and other

problems arising in heterogeneous zones should also be taken into account. Section4.3 discusses how certain elements of the evaluation should be geographically

focused in such cases.

Account may also have to be taken of synergy or antagonism with other programs or

measures on a different geographical scale or in an adjacent area.

8

3. SETTING UP THE EVALUATION: GENERAL

CONCEPTS

3.1. Designing the evaluation system

The arrangements for evaluating the rural development programme should in part beprovided by the programme itself and in part by the common evaluation questions

with criteria and indicators. Article 44 of the implementing regulation states in thisrespect that in particular the mid term and ex post evaluations must consider two

types of evaluation questions, i.e., those which are specific for the individual ruraldevelopment programme and the common evaluation questions relevant at the

Community level.

Concerning the first type of question, Point 12 of the Annex to the implementing

regulation specifies that the rural development plan must describe the arrangementsfor the programme specific part of the evaluation. The second type of question, in

principle common to all rural development programmes are discussed in Part 4 andsummarised in an indicative manner in Annex I.

The arrangements for evaluating the rural development programme should bedesigned as an articulated process in order to maximise the quality and usefulness of

its results. The contents of the evaluation will vary according to the stage (ex ante,mid term, ex post) of programme implementation as explained in Box 3.1.

Box 3.1 Functions of evaluation

"The ex ante evaluation shall analyse the disparities, gaps and potential of the current situation

and assess the consistency of the proposed strategy with the situation and targets and have regard

to the issues raised in the common evaluation questions. It shall assess the expected impact of theselected priorities for action and quantify their targets where they lend themselves thereto. It shall

also verify the proposed implementing arrangements and consistency with the Common

Agricultural Policy and other policies" (Article 43(1) of the implementing regulation).

"The mid term evaluation, while covering the evaluation questions, shall in particular examine

the initial achievements, their relevance and consistency with the rural development plan and the

extent to which the targets have been attained. It shall also assess the use made of financialresources and the operation of monitoring and implementation. (Article 44(2) of the implementing

regulation).

"The ex post evaluation, while answering the evaluation questions, shall in particular examine the

utilisation of resources and the effectiveness and efficiency of the assistance and its impacts and

it shall draw conclusions regarding rural development policy including its contributions to the

Common Agricultural Policy" (Article 44(2) of the implementing regulation).

The evaluations at the various stages of the programme implementation must deal with aspects

such as relevance, effectiveness, efficiency, utility and the sustainability of the results (defined inthe glossary, Annex IV). Relevance is mainly related to the ex ante situation, while the utility

and the sustainability of the results are particularly relevant for the ex post situation.

Figure 3.1 shows the interaction between the rural development programme and itsevaluation at different stages of its implementation. The dotted arrows show that

information on results and impacts from the plan or the implemented programme is

9

Figure 3.2 Relations between objectives and impacts

in the programming cycle

Impacts

Outputs

Inputs

Specificobjectives

Results

Needs

Overall

objectives

OperationalObjectives

collected and processed in the evaluation. The full line arrows indicate the time of

evaluation and show how the conclusions and recommendations from the evaluationought to feed back into the programme and its implementation or to the follow-up to

it.

The ex ante evaluation takes place before the implementation of the programme and

examines the proposed strategy. The components that a proper ex ante evaluationought to deal with are listed in Annex III. The mid term evaluation will take place

before 31 December 2003, i.e. almost four years after the beginning of theimplementation and may lead to budgetary reallocation or strategic reorientation,

based on the results and initial impacts of the programme. An update of the midterm evaluation may take place by 31 December 2005. The ex post evaluation takes

place within two years of the end of implementation and should examine theimpacts the programme has produced. Figure 3.1 also indicates that the evaluation

should take advantage of the results of previous evaluation works (upper arrows).Hence, the mid term evaluation should consider the results of previous ex ante

evaluation, and the ex post evaluation should consider the conclusions of the midterm evaluation.

3.2. The intervention logic

Evaluations should focus on the results and impacts of the programme. Figure 3.2shows how, within the programming cycle, results and impacts are related to the

objectives derived from the needsidentified in the region or sector.

Such needs relate to socio-economic or environmental

problems to which the programmeshould respond. The inputs are

financial or administrativeresources. Through programme

activities they produce the outputsand achieve the intended

operational objectives. Thesubsequent results are the most

immediate impacts of the

AchievedResults &

impacts

Mid term

evaluationEx ante

evaluation Ex post

evaluation

Expected

Figure 3.1 Programme & evaluation

Programmeimplementation

Follow-up,

e.g. new programme

10

assistance, in other words the contribution of the operational objectives to the

specific objectives. The mid term evaluation will have to focus on these aspects.Irrigation piping or canals are examples of output, and the availability of water to

new areas, or water of better quality, could be the results. The latter respond to thespecific objectives. The impacts derive from the results. From the above example,

an intermediate impact might be a reorientation in the local cropping system orreduced local soil pollution due to better quality irrigation water. Later on, more

global impacts might be a higher level of agricultural income or a change in marketbalance due to the availability of water for certain crops. Global impacts respond to

the overall objectives of the programme and, in a well-designed programme, theymeet the previously identified needs that led to the implementation of the

programme. There may also be unexpected, possibly negative impacts. Forexample, in the case of irrigation it might be pertinent to look for impacts such as

partial depletion of the water resource or salination.

Intermediate impacts may be obvious soon after the assisted action has produced the

direct outputs while more global ones may become apparent only at a later time andmay reach other or more people than those directly targeted by the assisted action.

Obviously, the evaluation should take this into account and avoid looking forimpacts that have not yet been produced.

The reconstruction of the links between the inputs, outputs, results and impacts ofthe programme is called the ‘intervention logic’. It shows how assistance is

transformed into intermediate and global impacts such as rationalised agriculturalproduction, stabilised agricultural markets, improvement of rural living conditions

or protection of the rural environment. The elaboration of this into a so-calledLogical Diagram of Impacts can help clarify the intervention logic of a programme,

cf., Box 3.2.

Box 3. 2 The Logical Diagramme of Impacts (LDI)

Usually it is straightforward to identify the outputs and the expected global impacts of a

programme, but it may be difficult to identify how the outputs are transformed intoglobal impacts. A Logical Diagramme of Impacts can help clarify the intermediate

impacts arising from the assistance. It identifies causal links and describes what theoutputs contribute to, and from which action the impacts are produced (see further in

Figure 4.1)

Global impact

Output 1

Output 2

Output n

Result 1

Result 2

Result n

Impact 1

Impact 2

11

3.3. Examining the impacts of the programme

Evaluation must go beyond monitoring, reporting and auditing which deal with

inputs, outputs and sometimes with results. Too often evaluations are restricted tothe gathering of data about how the financial inputs were used and about the direct

outputs that have been paid for. Evaluation must be concerned with the impactswhether they are positive, negative, expected or unexpected, and including those

which only become manifest in the long term or which are to the advantage ofpersons other than the direct beneficiaries.

A single intervention can contribute to several impacts as illustrated in the aboveoutline of a Logic Diagramme of Impacts. For example, processing equipment

(Chapter VII of the Council regulation) can contribute to the development of newproducts and outlets as well as to the rationalisation of processing procedures and

cost reduction. Moreover, several outputs can contribute to the same impact.Environmental concerns are often cross-cutting and can be partly dealt with through

a training scheme (Chapter III), by investing in adapted equipment (Chapter I) or bymodifying agricultural practices (Chapter VI). The evaluation should take such

interactions between different actions into account and should examine thecomplementarity and synergy effects they may produce. The evaluation may also

have regard to certain aspects of other Community or national measures that mayinteract with the rural development program, e.g., aspects of LEADER or state aid

measures, cf., also Point 5(3) and 6(2) of the Annex to the implementing regulation.

Evaluation should focus on aspects such as the utility, relevance, coherence,

effectiveness, efficiency and sustainability of the results (see the glossary, Annex IVfor definitions). The pertinence and relative importance of these aspects would vary

according to the stage of implementation of the programme. It is also important todistinguish net and gross effects and the possible contribution of exogenous factors

(e.g., local context or market influence) to the impacts in order to identify moreprecisely what can be attributed to the evaluated assistance. Further aspects such as

deadweight effect that may influence the achievement of the objectives must also beexamined. The influence of the implementing procedure (selection procedure, zonal

planning, administration, …) must also be examined in so far as they influence theeffectiveness and efficiency of the assistance. Section 5.3 discusses how these and

other aspects of the impacts can be examined in the data analysis.

12

4. EVALUATION QUESTIONS, CRITERIA AND

INDICATORS

4.1. Common evaluation questions

Article 42 of the implementing regulation indicates that evaluations shall respondto common evaluation questions defined by the Commission in consultation with

the Member states and, as a general rule, be accompanied by achievement relatedto criteria and indicators. These questions are of interest beyond the beneficiaries

of the programme, i.e., at the sectorial, national and, above all, the Communitylevel. Hence, they help to assure the assessment of the added value of the

programme at Community level.

According to Article 44 of the implementing regulation it must be substantiated if acommon evaluation question is inappropriate in relation to a particular rural

development plan. This relates specifically to the chapter-specific questions, cf.,below. Hence, a particular common question is considered applicable and must be

answered if a measure is being implemented, which would normally produce theresult or impact under consideration.

Notwithstanding this requirement, the evaluation ought also to reflect the balance,including the financial volume, of the measures selected in the programming. This

means that in exceptional and well-justified cases the evaluation may partlyconcentrate on those of the applicable common questions that really reflect the key

issues of the programme. However, such concentration must by no meansjeopardise the scope and quality of the evaluation.

The agri-environmental measures are a compulsory element of any programme andmust hence always be part of the evaluation according to the procedure described in

the present document.

Two types of common evaluation questions have been defined. Firstly, chapter-

specific questions have been elaborated for each of the nine chapters of the Councilregulation. Secondly, a few cross cutting questions that cover all chapters of the

regulation have been defined. They analyse the transversal global impacts of theassistance and the relationship between different kinds of measures. These

transversal impacts include the effects on living conditions, structure of the ruralpopulation, employment income, environment and agricultural markets as

prescribed in Article 44 of the implementing regulation. They also prompt ananalysis of the administrative and implementing arrangements.

Several questions would include a main question that examines an intermediate orglobal impact plus two or more operational sub-questions. The latter, while being

important in their own right, serve to answer the main question.

In order to facilitate the aggregation of results at Community level, the authorities

responsible for programme management must integrate the common questions intheir terms of reference when they are relevant to the programme. They may also

complement the predefined common evaluation questions, either by adding moresub-questions or by adding criteria and indicators that supplement the predefinedones according to the characteristics of the programme, cf., Figure 4.2. Annex I

13

lists both the chapter-specific and the cross cutting common evaluation questions on

an indicative basis.

4.2. Questions for individual programmes

In addition to the common questions defined at Community level, the responsibleauthorities need to formulate specific evaluation questions, cf., Article 44(1) and

Point 12 of the Annex to the implementing regulation. These questions shouldrespond to the situation of the specific programme in terms of its context, for

example the natural conditions or in terms of its specific objectives and eligibilityconditions. Appropriate indicators should, as a general rule, accompany such

questions. The procedure used by Member states to elaborate their specificevaluation questions should in principle be based on the reconstruction of the

intervention logic as described in Section 3.2. Evidently, the programme specificpart of the evaluation should reflect the balance of measures contained in the

programme and should also cover the compulsory agri-environmental measures inan appropriate manner.

4.3. Using appropriate criteria and indicators

Following the requirement in Article 42(2) of the implementing regulation, the

Commission will, as a general rule and at an appropriate time, specify criteria andindicators for the common evaluation questions listed on an indicative basis in

Annex=I.

For a particular evaluation question, the criterion would help to formulate ajudgement on the success of the assistance under examination by linking the

indicator to the expected result or impact. If the question, for example is: “To whatextent has the scheme reduced the cost of processing and marketing of agricultural

products?”, then the criterion would be “the cost have been reduced by X %”. Forthe reason set out below it would be necessary to split such a criterion in its two

parts (a) ‘the cost have been reduced’ and (b) ‘by X %’. The first part (a) iscommonly referred to as criterion in this document while the second part (b) is

referred to as the target level, i.e. the level expected to be reached to fulfil thecriterion and assess that the assistance has been successful. In straightforward cases

the target level corresponds to a quantified objective. In the given example, it couldbe “…by 2%” or alternatively “… by 0.5 € per unit”.

The reason for the above mentioned split up derives from the fact that ruraldevelopment programmes are set up to meet specific regional, local or sectoral

objectives within the framework of the objectives at EU-level in the regulation.Hence, for a given criterion it may be necessary to have different target levels in

different programmes. In the examples above that would mean that the figure of

0.5=€ (or alternatively 2%) might vary between programmes.

The precise target level can be defined in relation to a baseline or a benchmark. Theexample above of “2% cost reduction” relates to the evolution compared to a

baseline, which normally would be the situation for the beneficiaries at the start ofthe programme. Alternatively, the target could have been “below a cost of 16

€/ton”, where the benchmark of 16 €/ton would be derived from norms orknowledge of the best practice in the sector, for example established from the ex

post evaluation of a previous programme.

14

At the EU-level the target level may frequently be specified in a rather general form,

for example “>X%” or “An upward trend in the indicator compared to the evolutionfor non-assisted but otherwise similar firms/holdings”. This latter case illustrates a

combination of the reference to a baseline (“an upward trend”) in comparison of abenchmark (“compared to the evolution for non assisted”). In such cases, the

responsible authority should, as appropriate and depending on the situation asregards of the programme or the local context, define the target more precisely.

Indicators may, depending on the implementing stage of the programme, relate to anoutput, a result or an impact. A single question may include more than one criterion

and more than one indicator. Such programme indicators are quantified from the

monitoring systems or from data collected ad hoc for=the=evaluation.

15

Figure 4.1 E V A L U A T I O N

Inputs Outputs Results Impacts

intermediate globalWater supplied to more holdings

Number of holdings (unit) affected

by irrigation

(% of total number of holdings)

Improvement in farmers' incomes

due to irrigation

- impact on costs

- impact on revenue

Evolution of farm incomeComparison with farmers not

affected by irrigation (NVA/AWU)

Improvement in farmers' incomes

Water available to new areas

New irrigated and irrigableareas(ha)

(% of cultivated land)

Reorientation in the cropping

systemCrops affected by irrigation

(% of crops affected)

Impact on the balance of the

markets in the crops in question

Quality of the available water

Contents of pollutants or minerals in water (mg/m3)

Quantity of water used

Additional quantity of water

taken from water supply network

(incl. through dams) (m3, in % of

available volume)

Soil quality affected

- Pollution(area affected by type of pollution

in ha)

- Salination

(area affected in ha)

Water depletion

Number of days with reduced

flow in river

Lowered water table for surface

or ground water

Impact on environment:- natural resources (water, soil, )

- landscapes

- biodiversity.

Irrigation network

Piping and canals built

(length, throughput)

Dams built

(number/capacity)

Financial

Administrative

Other

16

Objectives:

- Programme

level

- Community

level

Common questions

(Community level)

Sub-questions[if relevant]

(Community level)

Criteria & indicators

(predefined at EU-level)

Additional sub-questionsto the common questions

(national, regional, sectoral)

Figure 4.2

Other information:

- previous evaluations

- annual reports

- research

- other studies

Key evaluation aspectsEffectivenessEfficiency

Relevance

Utility

SustainabilityCoherence

Exogenous factors

Deadweight effect

Leverage effect

Displacement effect

Substitution effect

Programme specific

questions & sub-questions

(national, regional,sectoral)

Criteria & indicators

at programme level

(national, regional,

sectoral)

EU level

Programme level

The indicators discussed above are programme indicators. They concern that

part of a population or of a territory, which the programme touches upon. Assuch they attempt to measure the direct or indirect effects of the programme.

By contrast, context indicators apply to an entire territory, population orcategory, for example a particular agricultural sector within the relevant

region or country. By measuring appropriate economic, social orenvironmental variables, such indicators can put the programme into

perspective. A context indicator may, for example, show that a certainprogramme, albeit successful in relation to its own given objectives, is no

longer justified or, on the contrary, that a certain type of aid is still relevanteven if the relevant programme indicator demonstrates limited progress.

Some context indicators may justify regular monitoring while others should becollected ad hoc for the evaluation.

The statistical information contained in EUROFARM and the FarmAccountancy Data Network may be of particular help in this situation.

EUROSTAT, in cooperation with other services of the Commission, has alsoproduced recent publications containing useful context information.5

In principle, indicators should be provided at the same geographic scale as therural development programme. However, certain measures may not apply

throughout the whole area covered by the programme or may be specificallytargeted according to the areas concerned. The specific regulatory

arrangements in this respect concern particularly the less-favoured areas andareas with environmental restrictions, the agri-environmental measures and

the forestry measures as mentioned in Section 2.4.

Similarly, a further geographical break down would be appropriate for the

purpose of evaluation in the case of heterogeneous zones or when sectoralplans cover a large part of a country. Such a break down can help to extract

specific geographical information and to elucidate the effects of and onregional or local disparities. The need for geographical differentiation applies

not only to the indicators (including data collection and analysis), but also tothe answers, conclusions and recommendations following from them.

Indicators are important in providing factual information, but often they wouldneither completely cover nor explain the causes acting behind the facts. They

are instruments to help answer the evaluation question, but they do notconstitute the complete answer. A wider data collection as well as a

qualitative assessment may well be necessary to provide comprehensive andfully reliable findings.

Figure 4.1 shows how indicators can be used at the different stages of theintervention logic. Various programme indicators should be used according to

the level of evaluation: output indicators when dealing with outputs, resultindicators to assess the results, impact indicators to assess the impacts. The

5 1) European Commission, Agriculture, environment, rural development; Fact and Figures – A Challenge

for Agriculture, 1999

2) in preparation: Indicators of rurality and rural development

18

figure also shows the interrelations between the elements of the intervention

logic, as explained in Section=3.3. A particular output may simultaneously

contribute to several impacts. For instance, piping and canals built cancontribute to provide more water in a given area and simultaneously influence

the quality of water. Also, the quantity of water used in a given area mayinfluence both the level of salination and the depletion of water in this area.

An impact may also derive from several elements in the intervention logic: theimprovement of farmers’ income could be due to the irrigation as such or due

to the reorientation in the cropping system which it may lead to.

Figure 4.2 shows the procedure that should be followed to answer to the

evaluation questions. Criteria and indicators help to answer questions thathave been defined in relation to the objectives at the level of the regulation

and the individual programme. Other sources of information, such as previousevaluations, studies, or research can also contribute to the answers. The figure

also indicates that certain key evaluation aspects should be taken into accountfor answering the questions. Questions may refer to effectiveness, efficiency,

utility, etc… This should be considered when collecting the information. Inorder to assess the net effects and have then more reliable evaluation results, it

is also necessary to take into account factors such as exogenous factors,displacement effect, … These aspects are examined in more details in Section

5.3.

19

5. EVALUATION METHODOLOGY

5.1. Use of recognised methods

The implementing regulation on rural development in Article 42(1) requires

evaluations to be performed in accordance with recognised evaluation practice.This has a bearing on both the collection of data and the subsequent analysis. In

addition, Article 45 specifies that the evaluation reports must explain themethodologies applied, including the implications for the quality of the data and

the findings. Hence, the report must indicate the sampling techniques and the datasources that have been used to collect data. This should enable the reliability of the

evaluation findings to be assessed and facilitate the production of sound and usefulconclusions and recommendations. This issue is closely related to the quality of

evaluation as described in Section 7.6.

5.2. Data collection

The data collection should be performed with a view to the subsequent analysis.

Article 42(3) of the implementing regulation requires the responsible authority toassemble the appropriate resources for evaluations while making use of monitoring

results supplemented where necessary by the gathering of additional information.

Secondary=data

Firstly, this means that optimum use should be made of so called secondary data,i.e., date that exist already, for example from the monitoring system. They may

sometimes be sufficient to answer a question or a part of it. As such data arealready available they can be assembled at a moderate cost. Hence, it is important

to have a well-designed monitoring system. The use of monitoring data wouldespecially be important in the mid term evaluation. Other secondary data may be

obtained from existing surveys, previous evaluations or programming documents.Programme documents and monitoring data can often especially provide financial

information (inputs) and information about the outputs.

National statistics offices, and sources like the Farm Accountancy Data Network,EUROFARM, EUROSTAT may provide information concerning the baseline

situation, the contextual situation or exogenous factors for a certain area or type ofagricultural holding.

Primary=data

Secondly, the additional information referred to in Article 42(3) will often beprimary data, i.e., data collected ad hoc for the evaluation, most often among the

direct or indirect beneficiaries. Representative sampling methods are essential forthe collection of primary data. This means that the units (holdings, persons, etc)

under investigation must represent the programme in a satisfactorily way. Thesample must comply with statistical rules, for example concerning the size of the

sample compared to the variance of the data or the manner in which the units areselected, e.g. using stratified sampling, in order to enhance representativity and

avoid biases. Sampled data can be scaled up with the aid of the monitoringindicators provided that the sample corresponds to the monitoring categories.

Numerous tools such as questionnaires, interviews or case studies exist forextracting the information. Questionnaires can be used to collect quantitative or

20

qualitative information. They can be used for larger samples than, for example,

interviews and are hence useful for ensuring representativity of the samplecompared to the population under investigation. Interviews with programme

managers, operators and beneficiaries can complement management data where thelatter and other secondary data are insufficient. Interviews can for example, help

identify successes and failures. Individual interviews are commonly used, but it isalso possible to organise different types of group interviews. It should be kept in

mind that certain tools, particularly interviews introduce subjective views;alternative points of views should be tested and rationale behind opinions put

forward should be checked.

Case studies produce very comprehensive qualitative information (through thecollection of management data, interviews, …) on a limited number of actions. It

can be used for analysing the implementation of a programme or to provide an in-depth analysis of the impacts. Cases cannot, strictly speaking, comply with the

rules for sampling, but must be carefully selected, based for example on arepresentative typology of actions.

5.3. Data analysis

Once data are collected, the evaluation must analyse them and estimate the effect ofthe programme. Several tools can then be used in order to analyse data, form

conclusions in order to formulate a judgement. Their adequacy will depend on thetype of programme evaluated, its degree of complexity, or the type of information

searched. Volume 3, Part IV of the MEANS collection (see Annex V) recapitulatesrelevant techniques, according to the evaluation situation: when evaluating globally

a programme or in the case of an in depth evaluation, depending on the stage of theprogramming cycle (ex ante, mid-term, ex post) and depending on the phase within

the evaluation (data collection, data analysis, …). For instance, cost-effectivenessanalysis can be used to formulate a judgement concerning specific and well defined

issues within a programme, while a multi-criteria technique is more appropriate inthe case of complex matters or entire programmes.

Data should be analysed to allow comparisons and conclusions so the evaluation

can judge whether the effects the programme has produced (or will produce in thecase of an ex ante evaluation) are sufficient in relation to the objectives. It must be

made possible to examine the counterfactual situation, i.e., assess what would havehappened without the intervention (e.g., if the investments on farms had not be

made, or if a specific training had not been offered).

Enquiries about the added value of the assistance (“Would you have investedwithout the assistance?”) can assess the counterfactual situation. Comparison

groups can also serve to estimate the net effects of the assistance, i.e., what isdirectly attributable to the assistance from the programme. This consists ofexamining a group of beneficiaries with a group of non assisted peers, possibly

before and after the intervention. Comparison groups are also helpful to estimatedeadweight effects. For that purpose, a survey among non-beneficiaries can help

analyse whether certain projects would have been implemented even without theassistance. In some cases, substitution and displacement effects should also be

analysed, e.g., in case of aid for employment. Interviews are helpful to analysesuch effects. Subsequently, the net effect of the programme can be estimated by

subtracting deadweight, substitution and displacement effects from the gross effect.

21

Unintended effects, including negative effects and the influence from significant

exogenous factors such as market prices should also be appreciated where relevant.

Links to other programmes, and / or between actions within a programme should beexamined in order to estimate complementarity and synergy effects. Several

actions, or programmes, can produce the same impacts or can be mutuallyreinforcing. As an example, the improvement of infrastructure for basic services

(e.g., transport or communication networks) in rural areas can alleviate remotenessand can, together with the creation of tourist equipment, have synergy effects in

terms of development of activities and employment. Annex V providesbibliographical references about proper methods for data analysis.

22

6. REPORTING ON THE EVALUATION

6.1. Regulatory requirements

The implementing regulation on rural development demands that: Thestructure of the evaluation reports for the individual programmes must

follow, as far as possible, a recommendation for a common structure for theevaluation reports to be provided by the Commission, c.f., Article 45(3)

A certain minimum level of information must be provided within the

individual evaluation reports: description of the methodologies applied,including the implications for the quality of the data and the findings;

description of the context and contents of the programme; financialinformation; the answers including the utilised indicators to the common

evaluation questions and to the evaluation questions defined at national orregional level; conclusions and recommendations, c.f., Article 45(3)

Evaluations must be performed in accordance with recognised evaluation

practice, c.f., Article 42(1).

The above requirements are taken into account in the recommended common

structure for evaluation reports presented below and in a more complete formin Annex II.

6.2. The recommended common structure for the reports

The report must describe the programme being evaluated, including its context

and purpose, together with the procedure and findings of the evaluation andthe conclusions and recommendations it leads to. The common structure for

doing so would ensure that the individual evaluations embrace the essentialissues while making their results more comparable. Especially it would make

it attainable for the Commission to elaborate the required Community levelsynthesis. Unmotivated deviations from the recommended structure should

hence be avoided. The recommended common structure is explained in moredetail in Annex II.

The recommended common structure for evaluation reports(see complete version of this structure in Annex II)

The recommended structure foreseen in the implementing regulation embrace the

following main elements:

(a) Executive summary

(b) Introduction (context of programme, characteristics of the implementation, purpose of

the evaluation)

(c) Explanation of the methodological approach (design and analysis, data collection and

sources, soundness of data and findings)

(d) Presentation and analysis of the information collected (financial information andoutputs, uptake information on beneficiaries, replies to the common questions, findings on

the programme specific questions)

(e) Conclusions (concerning community level objectives, programme specific goals,

effectiveness, efficiency, utility, sustainability of the results, ..) and recommendations

(f) Annexes

23

The report for the exante evaluation can to

some extent besimplified compared to

the description abovein order, firstly, better

to accommodate theprovision of Article

42(1) of the Councilregulation and 43(2) of

the implementingregulation, which specifies that the ex ante evaluation shall form part the rural

development plan and secondly, comply with the time constraint under whichthe plan must be prepared. The rationale is that, to a large extent, the

improvements flowing from the ex ante evaluation ought to be taken up in therural development plan itself before it is submitted to the Commission.

Hence, the very object of the ex ante evaluation may cease to exist in its initial

form. In order to adapt to this situation Box=6.1 presents a simplified format

for reporting on ex=ante evaluations which takes into account the need for

transparency and for assessing the quality and scope of the evaluator’s work,c.f., Article 40 of the implementing regulation concerning potential co-

financing.

Box 6.1 wgc mpguget mi eoearvrgda du rhe ex ante

ebvpsvrgda Agrhga rhe mpva

The description within the plan must specify:

� The evaluation activities that have taken place, c.f.,

Annex III

� The conclusions and recommendations

� The extent to which the conclusions andrecommendations have led to changes to the rural

development plan.

24

7. ADMINISTRATIVE MATTERS

7.1. Responsibility for evaluation

The authorities preparing the rural development plan are responsible for the ex

ante evaluation (see Annex III) while the authorities responsible for managingthe rural development programmes, in consultation with the Commission, are

in charge of carrying out the mid term and ex post evaluations (Article 43(2)and 44(3)).

The prescribed consultation with the Commission should, above all, help inimproving the evaluations, but may also help in clarifying the conditions

(quality and scope) for co-financing the evaluation activities (Article 40).These consultations would imply a prior information and discussion of the

implementation of the actual evaluation, especially concerning the mainelements of the terms of reference. The authorities responsible for managing

the rural development programming document in the Member States are alsoresponsible for assembling the appropriate resources for evaluations, making

use of monitoring results supplemented where necessary by the gathering ofadditional information (Article 42(3)). An adequate uptake in the terms of

reference of the methodology in the present guide would greatly assist theconsultation process.

7.2. Terms of reference

The terms of reference upon which the call for tender and the contract with the

evaluating body would normally be based, should reflect the frameworkoutlined in the present guide as well as the arrangement for evaluation set out

in the individual rural development programming documents, c.f., Point 12 ofthe Annex to the implementing regulation.

The terms of reference need to address, inter alia, the objective and scope ofthe evaluation, how it is organised and what its use would be. The evaluation

questions (the common evaluation questions as well as those for individualprogrammes) must be precisely defined. Any principal methodological

requirements should be indicated, as well as the initial work plan and timeschedule, budgetary indications and criteria for selection of the evaluator (i.e.,

the required skills in terms of methodology and in terms of experience in thefield of agriculture and rural development).

In addition, the terms of reference have to address the deliverables that theevaluator must provide: preliminary reports, final report, type of support

(paper and electronic versions), hand-over of the detailed data that have beencollected. The evaluator should also be obliged to provide the indicators for

the common questions (where applicable, split up to regions, sector, types ofholding, etc) so that they can be passed to the commission services in a

practical form, i.e., normally on a electronic support medium. Specifically forthe ex ante evaluation, certain modifications may be needed compared to the

outline above in order to meet the specific constraints connected to this stageof evaluation.

25

Problems that might, at a later stage, hinder the publication of the results

should also be eliminated at this stage. As an example, it must be madepossible for the Commission to place the executive summaries of the

evaluation reports on the Internet, c.f., Section 7.4 below about the follow upto the evaluation report.

7.3. The independent evaluator

The evaluations must be carried out by independent evaluators from bodies

without direct involvement in the implementation, management and financingof the programmes. The evaluator should be competent regarding up to date

evaluation practice. Public institutions are not excluded if they fulfil thecriteria of independence and competence. The terms of reference should

define further criteria for the selection of the evaluator.

Independence does by no means imply that the evaluator must avoid co-

operation with the programme managers. For the ex ante evaluation, such aco-operation is essential in order for the evaluation to help ameliorate the rural

development plan before it is finalised. The same evaluator may deal with theevaluation at all stages of the programming cycle. Such an arrangement may

in some cases improve continuity and reduce the costs of evaluation.

7.4. Follow up to the evaluation report

A key purpose of evaluation, above all at the mid term stage, is to improve theimplementation of the rural development programmes or in certain cases to

refocus its aims or priorities. The authority responsible for managing the ruraldevelopment programme must inform the Commission on the follow-up to the

recommendations contained in the evaluation report.

The actual follow-up to a particular recommendation would of course depend

on many factors, including the quality of the evaluation, the practicalfeasibility of the recommendation and the improvement it would potentially

lead to. However, it is expected that all well substantiated recommendationsrelating to the running programmes be seriously considered and as far as

possible implemented where the types of conditions just mentioned arefulfilled.

The results of the evaluations should in full or summary, be available to thepublic once the commissioning body has approved the evaluation report. The

Commission hopes to make at least the executive summaries of the reportsavailable on the Internet.

7.5. Co-financing

Article 49(2) of the Council regulation stipulates that the EAGGF Guarantee

section may, within the framework of the financial resources allocated to theprogrammes, participate in the financing of evaluations relating to rural

development in the Member Sates. A separate application for funding to theCommission is not required, but a prior consultation as explained in Section

26

7.1 must take place. [Furthermore, Article 6 of the transitional rules6 specifies

that also the ex ante evaluation, although it is carried out prior to thesubmission of the plan would be eligible if paid after 31 July 1999].

Article 40 of the implementing regulation sets certain eligibility conditions forthe co-financing. Participation in the financing shall only apply to evaluations

that effectively contribute to the evaluation at Community level due to theirscope, particularly through their replies to the common evaluation questions

and through their quality. The requirements concerning quality are elucidatedbelow (7.6) and those concerning the scope are set out in Part 2. Article 40

also adds certain ceilings for the financial participation. The co-finance shallnot exceed 50% of an upper limit which, except in duly justified cases, is 1%

of the total costs of the rural development programme. Within each ruraldevelopment programmes at least 40% of the co-financing shall concern the

ex post evaluation.

In line with the principles of Sound Financial Management7 only additional

costs would be eligible, i.e., expenditure over and above what is linked to theday-to-day tasks of the public administrations. That means that variable costs,

for example consumables, travel, or external assistance that are necessary forcarrying out the evaluation would be eligible. Additional personnel engaged

on a temporary basis exclusively in order to help with the evaluation wouldalso be eligible, but salaries for the personnel in the public service would not.

Overhead costs and deprecation (e.g., of computers) of the public servicewould not be eligible.

7.6. Assessment of the quality of evaluation reports

The authority responsible for managing the rural development programmes

and the Commission must assess the quality of the individual evaluations.Adequate quality of the evaluation is one of the conditions for co-financing of

the evaluations (Article 40). Recognised methods must be used for assessingthe quality. This would normally entail the use of the following criteria:

Meeting needs: Does the evaluation adequately address the information needs and fitthe terms of reference?

Relevant scope: Is the rationale of the programme and its outputs, results and

impacts examined fully, including both intended and unexpected policy interactions

and consequences?

Defensible design: Is the evaluation design appropriate and adequate to ensure that

the full set of findings, along with methodological limitations, is made accessible for

answering the evaluation questions?

Reliable data: To what extent are the primary and secondary data collected/selected

appropriate, offering an adequate degree of reliability for the intended use?

Sound analysis: Is quantitative and qualitative information appropriately and

systematically analysed according to the state of the art so that evaluation questionsare answered in a valid way?

6 Commission regulation (EC) No 2603/1999 of 9.12.1999 laying down rules for the transition to the ruraldevelopment support provided for by Council Regulation (EC) No. 1257/1999, OJ L 316, 10.12.1999, p. 26.

7 See, for example, Datasheet N° 22 (notified 24.04.1997) concerning the eligibility of Structural Funds’expenditure (‘Cost incurred by Public Administrations, including Salaries of Member States’ Civil Servants)

27

Credible findings: Do findings follow logically from, and are they justified by, the

data analysis and interpretations based on carefully described assumptions and

rationale?

Impartial conclusions: Are recommendations fair, unbiased by personnel orstakeholders’ views, and sufficiently detailed to be operationally applicable?

Clear report: Does the report clearly describe the programme being evaluated,

including its context and purpose, together with the procedures and findings of theevaluation, so that information provided can easily be understood?

These criteria are explained in the Volume 1 of the MEANS collection.“Evaluating socio-economic programmes: evaluating design and

management” It should be consulted for further detail and explanation aboutthe assessment of the quality of evaluations, cf., the reference documents in

Annex V.

7.7. Timetable for evaluation

Activity a Deadline

(where applicable)Definition of common evaluation questions : 1999

Definition of common criteria and indicators: End 1999

Establishing the terms of reference for the ex ante

evaluation:Reporting on ex ante evaluation: With the plan

Establishing terms of reference for mid term

evaluation:Reporting on mid term evaluation: Not later than

31.12.2003

The responsible authority informs theCommission about the follow up to the mid term

recommendations:

The Commission elaborates a Community level

synthesis on mid term evaluation:

Upon receipt of

individualevaluation reports

Optional update of the mid term evaluation: By 31.12.2005

The responsible authority informs theCommission about the follow up to the

recommendations in the update:

Establishing terms of reference for the ex postevaluation:

Reporting on the ex post evaluation: Max 2 years after

end of program-

ming period

The Commission elaborates a Community level

synthesis:

Max 3 years after

end of

programmingperiod & receipt

of reports.

28

Annex I

Common evaluation questions

This is an indicative list of the common evaluation questions. It contains only the main

questions, not any sub-questions, criteria and indicators. The implementing regulationon rural development requires, as discussed in Part 4, the Commission to define common

evaluation questions with criteria and indicators. The present annex would inconsultation with the Member States be improved and complemented with sub-questions,

criteria and indicators as required.

Common cross-cutting evaluation questions(transversal to the individual chapters of the Council regulation)

1. To what extent has the assistance influenced the population level, composition

and distribution in rural areas?

2. To what extent has the assistance been conducive to securing employment?

3. To what extent has the assistance been conducive to provide an appropriate levelof income to the rural community?

4. To what extent has the market situation been improved through the assistanceespecially from redeploying production, improving quality and competitiveness?

5. To what extent have environmental concerns been integrated into ruraldevelopment programming so as to improve the environmental aspects of

activities in rural areas, including agricultural practices?

6. To what extent have programming and implementation helped in producing the

anticipated impacts?

Common chapter-specific evaluation questions

(related to the individual chapters I to IX of the Council regulation)

Chapter I – Investments in agricultural holdings

I.1.- To what extent have the investments improved the income of beneficiaryfarmers?

I.2.- To what extent have the investments contributed to improve efficiency atholdings?

I.3.- To what extent have the investments contributed to the reorientation offarming by redeployment of production and diversification of activities?

I.4.- To what extent have the investments improved the quality of farm products?

I.5.- To what extent has the diversification of on-farm activities originating from

the investments helped maintain employment?

I.6.- How significant are the impacts of the investments on the rural environment?

29

I.7.- To what extent have the investments improved the quality of the production

process, notably by improving the working conditions and animal welfare andhygiene?

Chapter II – Setting up of young farmers

II.1.- To what extent has the setting up aid covered the costs arising from setting up?

II.2.- To what extent has the setting up aid contributed to the earlier transfer of

holdings?

II.3.- To what extent has the aid influenced the number of young farmers of either

sex setting up?

II.4.- To what extent has the setting up of young farmers actually achieved,

contributed to safeguard employment?

II.5.- How significant was the synergy with the aid for early retirement in achieving

earlier transfer? (cf., question IV.4)

Chapter III – Training

III.1.- To what extent has vocational training of individuals of either sex assisted in

achieving efficient and competitive structures?

III.2.- To what extent has the improved level of training contributed to employment,

more specifically to enhance job quality?

III.3.- To what extent has vocational training promoted environmentally sustainable

management and practice in agriculture and forestry?

III.4.- To what extent has vocational training enabled farmers to conform to

standards in the field of hygiene and animal health?

III.5.- To what extent has vocational training been conducive to the uptake of rural

development activities?

Chapter IV – Early retirement

IV.1.- To what extent has the aid for early retirement contributed to the earlier

transfer of farms and the rejuvenation of the agricultural population?

IV.2.- Was the income offered to the transferors and to farm workers appropriate in

terms of offering them a fair standard of living and making them abandonfarming?

IV.3.- To what extent has the reassignment of released land to non-agriculturalactivities contributed to the quality of environment and the preservation of

landscape?

IV.4.- To what extent has the aid for early retirement in conjunction with the aid for

setting up of young farmers contributed to the improvement of agriculturalstructures? (II.5)

Chapter V – Less favoured areas and areas with environmental restrictions

V.1.- To what extent have the allowances contributed to farm income?

30

V.2.- To what extent have the allowances helped in ensuring a continuation of the

agricultural land use?

V.3.- To what extent have the allowances in less favoured areas helped in

maintaining the farming population?

V.4.- To what extent have the allowances contributed to the protection of the

environment?

V.5.- To what extent has differentiation of the allowances with a view to the natural

handicaps and environmental problems and according to the regional, sectoraland structural situation, improved the effectiveness and efficiency of the

assistance?

Chapter VI – Agri-environment

PM: Biodiversity (habitats and damage to them from farm pollution)

PM: Rural landscapes (biophysical features, appearance of habitats and agriculturalecosystem, cultural & historical features)

PM: Natural resources (soil, water, )

Chapter VII – Improving processing and marketing of agricultural structures

VII.1.- To what extent have the investments helped increase the competitiveness and

added value of agricultural products?

VII.2.- To what extent have the producers of the basic agricultural products benefited

from the investments?

VII.3.- To what extent have the investments improved human health conditions

thanks to the quality of the products and the working conditions?

VII.4.- How significant are the impacts of the investments on the environment and

natural resources?

Chapter VIII – Forestry

VIII.1.-To what extent would forest resources be improved or safeguarded in the

short, medium or long term due to the aid?

VIII.2.-To what extent has the economic function of forests been improved in a

sustainable manner?

VIII.3.-To what extent have the forestry measures contributed to other socio-

economic or social functions?

VIII.4.-To what extent has biological diversity been conserved or improved?

VIII.5.-To what extent have the protective functions of forest been better ensured?

VIII.6.-How significant is the anticipated stocking of carbon dioxide due to

afforestation and forest improvement on a time scale relevant to theinternational engagements?

31

Chapter IX – Promoting the adaptation and development of rural areas

IX.1.- Have the assisted actions, specifically those undertaken to improve the livingconditions in rural areas, contributed to maintain the population in rural areas?

IX.2.- To what extent has diversification of activities originating from the assistancecontributed to the maintenance or creation of employment in rural areas?

IX.3.- To what extent have the actions contributed to maintain or improve the incomeof the rural population?

IX.4.- How significant have the assisted actions contributed to the ruralenvironment?

32

Annex II

The recommended

common structure for the reports

The implementing regulation on rural development requires the structure of the

evaluation reports for the individual programmes to follow, as far as possible, arecommendation for a common structure for the evaluation reports to be providedby the Commission, c.f., Article 45(3).

This requirement is discussed in Section 6.2 of the Guidelines. The presentannex presents the recommended common structure and explains its content in

more detail.

(a) Executive summary

The executive summary should contain at least the main findings and conclusionsof the evaluation. It should not exceed 5 pages.

(b) Introduction

The introduction should provide general and contextual information about the

programme: specific national policies, social and economic needs justifying theassistance, definition of beneficiaries or other target groups. The introduction

should provide information on previously implemented actions. It should alsoinclude the key (updated) characteristics of the implementation: actors involved,

institutional context, time frame, general budgetary information, brief descriptionof priorities and measures.

The introduction must also elucidate the evaluation process itself: recapitulationof the terms of reference, purpose and scope of the evaluation, the programme

specific and the common evaluation questions. It should also briefly outline anyprevious evaluations of relevance to the programme.

(c) Methodological approach

The evaluation design and its consequences should be made understandable in

this section. It should describe the general design of the evaluation and themethods used in the evaluation process:

– Sources of data, techniques for data collection (questionnaires, interviews;size and selection criteria for samples, …); information about how the

indicators are calculated in order to assess the quality and reliability of thedata and identify possible biases;

– Techniques for replying to the evaluation questions and arriving atconclusions.

Any problems or limitations connected to the methodological approach should beclearly stated.

33

(d) Presentation and analysis of the information collected

(This section is expected to make up the larger part of the total volume ofthe evaluation report).

1. The secondary and primary data (see Chapter 5) used for answering theevaluation questions should be thoroughly presented and explained. Detailed

data may go into an annex. The evaluation must go beyond descriptiveinformation; hence analysis and interpretation of the data must be a major part of

the report.

1.1. The information and analysis concerning the financial and administrative

inputs should at least comprise:

- The forecast expenditure for the period 2000-2006 for implementing of the

rural development programme as a whole as well as a split up to theindividual measures (c.f., Point 8 and Point 12 in the in Annex to the

implementing regulation about measures and codification of expenditure).Account should be taken of updated forecasts where they exist.

- The actual expenditure so far, for the entire implementation and theindividual measures and their uptake. The development, compared to the

previous evaluation, in the trend of the financial indicators should beanalysed where relevant.

- The financial effectiveness (actual spending in relation to forecastexpenditure for the whole programming period).

- Where relevant, the follow up to the recommendations of previousevaluations.

By no means, the evaluation should limit itself to simply stating whether or notthe financial planning or forecasts are being met.

1.2. The information related to measures and their output should at least involvethe following:

– The uptake of the measures by the beneficiaries should be explained inrelation to the population of potential beneficiaries.

– The actions and the concrete outputs deriving from the individual measuresshould be presented accompanied by relevant output indicators.

– The efficiency should be calculated where relevant by relating the financialinformation above to the nature and amount of direct physical output.

2. The text presenting the answers to the evaluation questions must not belimited to descriptive information but must include elements of analysis:

– Answers must be provided for (a) the questions for the individual programmeand (b) for each common question (including its sub-questions) which is

relevant to the programme.

� The answers must at least include an analysis and discussion of the

suitable indicator(s) in relation to the criteria and target level for the

34

question. The indicators must be presented, both those for the common

evaluation questions and those for the question relating to individualprogrammes, for the entire geographical area covered by the programme

and for any further geographical split, as explained in 2.4 and 4.3. If thisis the case, the range of values obtained for the indicator must be

discussed.

� The answer should also include an analysis and discussion of any other

relevant quantitative and qualitative information from the surveys orenquiries or from other sources, for example statistics, research or other

evaluations.

- For practical reasons, the answers and the indicators should be easily

identifiable, preferably by using the numbered references contained in AnnexI (or in any document that would possibly later supersede it) to the common

questions and sub-questions.

- While answering the evaluation questions, the focus and manner in which this

should be done would have to differ between the three types of evaluation.This differentiation derives in part from the requirements in Article 43(1) and

Article 44(2) of the implementing regulation as explained in Box 3.1 and inpart from the fact that it takes time before certain impacts become apparent.

However, it is expected that at least the ex post evaluation provides acomprehensive answer to all relevant questions, c.f., the requirement

contained in Article 44(2) of the implementing regulation. In the case of anex post evaluation, a validation of modifications to the programme (modified

priorities, reallocation of resources based on previous evaluation results)should also be included.

(e) Conclusions and recommendations

The findings (relating to the common evaluation questions and to the individual

programmes) from the analysis of the data and other information should, as a firststep, enable conclusions about the coherence between the implemented measures

and the pursued objectives and the balance between the different measures withina programme.

The findings should help draw conclusions on the contribution of the assistanceto the objectives of the specific rural development programme and to the general

objectives, at Community level, of the Council regulation on rural development.In addition, the evaluator should provide recommendations, deriving from his

analysis and conclusions. Recommendations should be useful to programmemanagers and deciding authorities for the continuation of the programme. In the

case of a mid term evaluation, concrete proposal for modifications to theprogramme (modified priorities, reallocation of resources) should be included if

relevant.

The conclusions and recommendations relate to the effects of the programme as a

whole (c.f., the cross-cutting questions) and to measures within each of the nine(as implemented) specific chapters of the Council regulation on ruraldevelopment. Commonly, they would have to include an element of judgement

on the part of the evaluator.

35

(f) Annexes

Detailed information such as the full terms of reference, complete data sets,analytic details, detailed monographs or the structure of questionnaires may be in

an annex..

36

Annex III

Ex ante evaluation

The implementing regulation in its Article 43(1) lists the key requirements for ex anteevaluation of rural development programmes. These include:

(A) Analyse disparities, gaps and potentials of the current situation

(B) Assess the consistency of the proposed strategy with the situation and targets

Verify the consistency with the Common Agricultural Policy and other policies

(C) Assess the expected impact of the selected priorities for action

Have regard to the issues raised in the common evaluation questions

(D) Quantify their targets where they lend themselves thereto

(E) Verify the proposed implementing arrangements

The detailed elements below follow this structure and can be considered as the

components that a proper ex ante rural development evaluation would deal with inorder to comply with these regulatory requirements.

Please note that several other parts of the present Guidelines discuss the ex ante

evaluation as well: box 2.1, box 3.1, figure 3.1, section 6.2, box 6.1, section 7.1,section 7.3, section 7.5 and section 7.7.

A. Analysis of disparities, gaps and potentials of the current situation in the

region / sector concerned

(1) Analysis of the situation of the region(s)/ sector(s) concerned, the main strengths

and weaknesses, the opportunities for and threats to rural development (SWOTanalysis)

– Description of the current situation, using quantified data, and includingthe agricultural sector, demographic situation, rural economy, human

resources, employment, environment

– A coherent ranking of disparities to be addressed

– Definition of driving forces towards sustainable rural development

(2) Analysis of previous results (where relevant)

– Relevance of the existing strategy

– Lessons from past experience, such as best practices, key incidents

affecting policy implementation, aspects relating to effectiveness, cost-efficiency of the measures.

– Problems with evaluability and monitoring

37

B.1 Assessment of the consistency of the proposed strategy

(1) Justification of the strategy and its priorities for actions

(2) Consistency between operational and global objectives

– How will compatibility be ensured when more than one rural development

plan applies in the region

(3) Balance between the different support measures

– Justification of the balance of measures, including, where appropriate, the

assessment of the degree of interaction between measures, i.e. thepossibilities for synergy and conflicts between measures and priorities of

the programme.

– Justification of financial resource allocation

– The extent to which the agri-environmental measures apply throughout theterritory in an appropriate manner

B.2 Consistency with the Common Agricultural Policy and other policies

(1) Extent to which the programme integrates Community policies and priorities

– Assessment of the consistency with other instruments under the CommonAgricultural Policy including, for example, the provisions concerning

‘Environmental protection requirements’ and ‘Modulation’ as introducedin Articles 3 and 4 of the common rules for direct support schemes under

the common agricultural policy8 as introduced under the reform in thecontext of Agenda 2000

– Integration of other environmental requirements into the policy of ruraldevelopment programming9

(2) Respect of Community policies and principles

– Competition policy

– Equal opportunities between men and women

(3) Respect of international and national obligations, e.g. environmental obligations

C.1 Assessment of the expected impacts of the selected priorities

(1) Evaluation of the expected impacts and results from the measures of the planrelated to the:

– global objectives, i.e. impacts

– specific objectives, i.e. results

– operational objectives, i.e. outputs

8 Council Regulation (EC) N° 1259/1999 of 17.05.1999 establishing common rules for direct support schemes

under the common agricultural policy, OJ L160 of 26.06.1999, p. 113

9 c.f., Decision No. 2179/98/EC of the European Parliament and of the Council on the review of the European

Community programme of policy and action in relation to the environmental and sustainable development“Towards sustainability”, Kyoto, CBD,

38

C.2 Have regard to the issues raised in the common evaluation questions

Article 44(1) of the implementing regulation lists these issues in a condensed form.The analysis of the expected impacts should hence include the living conditions and

structure of the rural population, employment and income from farm or off farmactivities, agricultural structures, agricultural commodities, quality, competitiveness,

forest resources and environment.

D. Quantification of targets

(1) Quantification of objectives and gaps, key disparities and identification ofrelevant indicators

– The quantification of objectives and definition of target levels should bedone according to the level of analysis: operational objectives, specific

objectives and global objectives

E. Verification of the proposed implementing arrangements

(1) Clear definition of responsibilities

– competent authorities and bodies responsible

– economic and social partners to be consulted, agricultural and

environmental bodies to be associated, e.g., with regard to agri-environmental measures

– role, composition and rules of procedures of any monitoring committee

(2) arrangements for monitoring and evaluation, including attention to the common

evaluation questions

(3) Procedures and selection criteria defined in order to reinforce the efficiency of

project selection

39

Annex IV

Glossary of evaluation terms

Coherence

Assessment of whether a better complementarity or synergy could be found within a

programme and in relation to other programmes. The internal coherence refers to thecorrespondence between the resources allocated to a programme and its objectives. The

external coherence refers to the adequacy between the evaluated programme and otherrelated programmes, e.g. Community aids for early retirement and related national state

aids.

Counterfactual situation

Situation which would have occurred without the public assistance. Also referred to

"policy-off" situation.

Criterion

Characteristic on which a judgement can be based. The criterion must be explicitly

defined. A measure would usually be judged on several criteria. Indicators are thendefined for each criterion. "Reduction of costs" is an example of a criterion to examine

the efficiency of agricultural holdings.

Deadweight effect

Change in the situation of the beneficiary that would have occurred even without the

public funding. For instance, an agricultural holding might have invested even in theabsence of co-finance.

Displacement effect

Effect obtained in a given geographical area to the detriment of another area. Forexample, when a job is created in an assisted area to the detriment of another job which is

lost outside the area concerned.

Effectiveness

Assessment of the effects in relation to the objectives of the evaluated programme. An

action will be effective when the objectives have been attained. For example, the ratiobetween the “number of kilometres” of water pipes that should have been constructed

(quantified objective) and the “number of kilometres” that have actually been constructedcould serve to assess the effectiveness of an agricultural measure concerning irrigation.

Efficiency

Assessment of the achieved effects in relation to the inputs (financial or administrative)mobilised; i.e., how economically have the inputs been converted into outputs, results or

impacts. Could the same result have been achieved with less resources, or more resultswith the same resources?

40

Exogenous factor

External factors partly or entirely responsible for the changes observed. The evaluationmust take into account such factors (e.g., market prices) in order to assess the net effect

of the assisted action.

Gross effect

Change observed consequently to the implementation of the measure. The observation of

gross effect is not sufficient to conclude properly on the effects imputable to the assistedaction. Deadweight, displacement and substitution effects, exogenous factors must also

be assessed to conclude on the net effects.

Impact

Effects of the programme in the medium or long term. There can be expected,

unexpected, positive or negative impacts, depending also on the influence of exogenousfactors. Direct and indirect beneficiaries can be affected by the impacts of a programme.

Indicator

For the purpose of the present guidelines: information in a form suitable for assessing or‘indicating’ the effects of an assistance. They help in quantifying and simplifyinginformation about complex phenomena. They represent more than the raw data on whichthey are based. Measurement produces raw data, which may be aggregated andsummarised to provide statistics; statistics can be analysed and re-expressed in the formof indicators, which fed into the evaluation or decision-making process.

There are programme indicators and context indicators. As an example, a programmeindicator for the criterion “Reduction of costs” might be the “Ratio between costs and

turnover on assisted holdings”. A context indicator relating to the income in thegeographical area covered by the programme might be compared to a programme

indicator such as “Gross farm income on assisted holdings”.

For the purpose of these evaluation guidelines, the programme indicators are linked tocriteria.

Input

Resources mobilised to implement the programme: financial means, material, legal andorganisational resources.

Leverage effect

Fact that the public funding induces private spending among the beneficiaries.

Net effect

Effect completely imputable to the programme. Deadweight, substitution and

displacement effects have been subtracted from gross effect to estimate the net effect.

41

Output

What the programme finances. For example: buildings, storage equipment, touristfacilities, …

Primary data

Data collected ad hoc directly in the field at the time of the running evaluation.

Relevance

Appropriateness of the objectives of a programme in relation to the sectoral needs and

socio-economic problems to which the programme should respond.

Result

The most immediate impact, directly identifiable once the action has been implemented.

It occurs as soon as the public intervention has been completed. For instance, whentourist accommodation is created or upgraded, a result would be an increased

accommodation capacity; when transport infrastructure has been created or upgraded thetravelling time within or from the area would be reduced.

Secondary data

Existing information, e.g., statistics, monitoring data, data from previous evaluations, …

Substitution effect

Fact that an effect is obtained in favour of the target beneficiaries but at the detriment of

non-eligible persons or organisations. Substitution effect appears, for example, when aperson finds a job thanks to a programme but at the same time another person looses

his/her job.

Sustainability of results

Effects are sustainable when they last in the long term, and after the end of the

programme.

Utility

The fact that the impacts observed correspond to sectoral needs and to identified socio-

economic problems. Unlike relevance, utility does not appreciate the intervention byreferring to the objectives of the assisted actions.

42

Annex V

Reference documents

1. COMMISSION PUBLICATIONS

1.1 MEANS documents on evaluating socio-economic programmes, Office for

Official Publications of the European Communities. (available in English andFrench).

Volume 1: “Evaluating socio-economic programmes: Evaluation design andmanagement”

Volume 2: “Evaluating socio-economic programmes: Selection and use of

indicators for monitoring and evaluation”

Volume 3: “Evaluating socio-economic programmes: Principal evaluation

techniques and tools”

Volume 4: “Evaluating socio-economic programmes: Technical solutions for

evaluating in partnership”

Volume 5: “Transversal evaluations of impacts on the environment, employment

and other intervention priorities”

Volume 6: “Glossary of 300 concepts and technical terms”.

1.2 Other general Commission documents

A handbook on "Environmental Assessment of Regional Development Plans and

EU Structural Funds Programmes", Commission Document (EN,DE,FR,EL)

“The new programming period: methodological working papers. The Ex-Ante

Evaluation of the 2000-2006 interventions” Document of the Commission services.

“The new programming period: methodological working papers. Indicators for

monitoring and evaluation.” Document of the Commission services.

“Evaluating EU expenditure programmes – Ex-post and intermediate evaluation” ,

Office for Official Publications of the European Communities (in English andFrench)

“Guide to Cost-Benefit Analysis of Major Projects”. European Commission, DGXVI (EN, FR, DE, EL, ES, PO, IT)

“Counting the Jobs: how to evaluate the employment effects of Structural Fundinterventions” (EN, FR, DE)

“Understanding and Monitoring the Cost-Determining Factors of InfrastructureProjects”. European Commission, DG XVI (EN, FR, DE, EP, ES, IT)

43

2. LITERATURE ON AGRICULTURE AND RURAL DEVELOPMENT

Baldock, D. (1999). Indicators for High Nature Value Farming Systems in Europe.In: F.M. Brouwer and J.R. Crabtree (Eds.) Environmental Indicators and

Agricultural Policy. Wallingford, CAB International, 1999, pp. 121-135.

Brouwer F.M. (1995), Indicators to Monitor Agri-environmental Policy in the

Netherlands, Agricultural Economics Research Institute (LEI-DLO), The Hague:The Netherlands.

Brouwer F.M; and van Berkum S. (1996), CAP and Environment in the European

Union: Analysis of the effects of the CAP on the environment and assessment ofexisting environmental conditions in policy, Wageningen Pers, Wageningen: The

Netherlands.

Brouwer, Floor and Philip Lowe (Eds.) (1998). CAP and the rural environment intransition: A panorama of national perspectives. Wageningen, Wageningen Pers.

Brouwer F.M. and J.R. Crabtree Eds. (1999). Environmental indicators andagricultural policy. Wallingford, CAB International, CABI Publishing,

Wallingford, UK

Brouwer, F.M. (1999) Agri-environmental indicators in the European Union:

Policy requirements and data availability. In: F.M. Brouwer and J.R. Crabtree(Eds.) Environmental Indicators and Agricultural Policy. Wallingford, CAB

International, 1999, pp. 57-72.

Campbell I. (1998), Guide de l'analyse de l'environnement des politiques et des

programmes agricoles, Bureau de l'environnement, Direction générale despolitiques, Agriculture et Agroalimentaire Canada.

Delbaere, B. C. W. (1998), 'Facts and Figures on Europe's Biodiversity - State andTrends 1998/99', European Centre for Nature Conservation, Tilburg.

European Commission, Agriculture, environment, rural development; Fact andFigures – A Challenge for Agriculture, 1999

Institut Francais de l'Environnement, Orleans (1997). “Agriculture etenvironnement: les indicateurs”

Jesinghaus, J. (1999). Agricultural sector pressure indicators in the EuropeanUnion. In: F.M. Brouwer and J.R. Crabtree (Eds.) Environmental Indicators and

Agricultural Policy. Wallingford, CAB International, 1999, pp. 45-55.

Jongman, R. H. J. (1996), 'Ecological and Landscape Consequences of Land Use

Change in Europe', European Centre for Nature Conservation, Tilburg.

Lowe, P., N. Ward and C. Potter (1999). Attitudinal and institutional indicators for

sustainable development. In: F.M. Brouwer and J.R. Crabtree (Eds.)Environmental Indicators and Agricultural Policy. Wallingford, CAB International,

1999, pp. 263-278.

44

Moxey A., Whitby M., and Lowe P. (1998) Environmental indicators for a

reformed CAP, Monitoring and evaluation policies in agriculture, Centre for ruraleconomy, University of Newcastle Upon Tyne

OECD (1994), Creating rural indicators for shaping territorial policy, OECD: Paris.

OECD (1997), Environmental Indicators for Agriculture, OECD: Paris.

Parris, K. (1999). Environmental indicators for agriculture: Overview in OECDcountries. In: F.M. Brouwer and J.R. Crabtree (Eds.) Environmental Indicators

and Agricultural Policy. Wallingford, CAB International, 1999, pp. 25-44.

Rametsteiner, E. (1999).Criteria and indicators: Experience in the forestry sector.

In: F.M. Brouwer and J.R. Crabtree (Eds.) Environmental Indicators andAgricultural Policy. Wallingford, CAB International, 1999, pp. 247-262.

Tucker, G. (1999). Measuring the impacts of agriculture on biodiversity. In: F.M.Brouwer and J.R. Crabtree (Eds.) Environmental Indicators and Agricultural

Policy. Wallingford, CAB International, 1999, pp. 89-103.

Wascher D.M., Piorr H.-P. and Kreisel-Fonck (1998), Agri-environmental

Indicators for Landscapes, European Centre for Nature Conservation, Tilburg: TheNetherlands.

Wascher, D.M., M. Múgica and H. Gulinck (1999). Establishing targets to assessagricultural impacts on European landscapes. In: F.M. Brouwer and J.R. Crabtree

(Eds.) Environmental Indicators and Agricultural Policy. Wallingford, CABInternational, 1999, pp. 73-87.

3. INTERNET SITES

1) http://europa.eu.int/comm/dg06/eval/index_en.htm

Evaluation in DG VI, including recent evaluation reports

2) http://www.inforegio.cec.eu.int/wbdoc/docoffic/evaluation/evaluation_en.htm

Contains documents on evaluation of structural programmes relating toObjectives 1, 2, 3 and cohesion: regulations, working documents on evaluation,

electronic newsletter on evaluation activities, results/lessons from evaluation,etc.

3) http://www.europeanevaluation.org/

The European Evaluation Society (EES) whose goal is to promote theory,

practice and utilisation of evaluation. Includes links to other evaluation sites.

4) http://europa.eu.int/comm/dg06/index_en.htm &

http://europa.eu.int/pol/agr/index_en.htm

45

Information about the Common Agricultural Policy and rural development

including information about the Commission's department responsible foragriculture

5) http://europa.eu.int/comm/dg19/evaluation/en/index.htm

General information about evaluation in the European Commission (website of

the Directorate General for Budgets)