1 the agri-environmental footprint index users’ manual an output of the ae-footprint project...

1

THE AGRI-ENVIRONMENTALFOOTPRINT INDEX

USERS’ MANUAL

An output of the AE-FOOTPRINT Project SSPE-CT-2005-006491

Project Team:

S.R. Mortimer, J.R. Park, A.L. Mauchline, K.A. Haysom, D.B. WestburySchool of Agriculture, Policy And Development, University of Reading, UK

G. Purvis, G. Louwagie, G. NortheySchool of Biology and Environmental Science, University College Dublin, Ireland

J.A. FinnTeagasc (Irish Agriculture and Food Development Authority), Environmental Research Centre, Ireland

K. Knickel, N. Kasperczyk Institute für Ländliche Strukturforschung, Johann Wolfgang Goethe Universität Frankfurt, Germany

J. Primdahl, H. Vejre, J. Vesterager, L. Kristensen, K. TeilmanDanish Centre for Forest, Landscape and Planning, University of Copenhagen, Denmark

L. Podmaniczky, K. BalázsInstitute of Environmental and Landscape Management, Szent Istvan University, Hungary

G. Vlahos, S. ChristopoulosDept of Agricultural Economics and Rural Development, Agricultural University of Athens, Greece

L. Kröger, J. Aakkula, A. Yli-Viikari, J. PeltolaMTT Agrifood Research Finland, Economic Research, Finland

For further information:

Dr Simon [email protected]

Draft version March 2008

© AE Footprint, 2008

2

CONTENTS

How to use this manual 3

Introduction 4

Which evaluation? 6

Flow of AFI methodology 7

Step 1. Setting the scope of the evaluation 8Step 2. Identifying environmental concerns 12Step 3. Assign weights to the major AE-Issues 17Step 4. Identifying farm-level indicators 20Step 5. Collecting farm-level indicator data 24Step 6. Converting indicator values to scores 26Step 7. Weighting indicators in AE-Dimensions 28Step 8. Calculating the AFI 30 Step 9. Conducting a sensitivity analysis 34

References 35

Appendix A. 36

- - - - - - - Draft version March 2008, not for circulation - - - - - - -

3

HOW TO USE THIS MANUAL

This manual is intended for those involved in the evaluation of European agri-environmental schemes (AES). The manual describes a methodology suitable for evaluation of AES which was developed as part of the Agri-Environmental Footprint Project funded by the European Commission. The manual provides guidelines and examples for the use of the methodology in AES evaluation in non-technical language.

Who is this manual for?This manual will primarily be of use to agri-environmental policy analysts, AES policy makers and those tasked with implementing AES evaluation. It will also be of interest to other groups, such as national and regional farming organisations; environmental NGOs and special interest groups; agencies responsible for wider environmental monitoring and evaluation; and scientific groups involved in environmental research linked to agricultural landscapes.

How to use this manualThis manual begins with a general introduction to the methodology and it’s uses. A stepwise description of the AFI methodology is then provided on the blue pages, with descriptions of technical terms and illustrated examples or options provided in numbered boxes on the facing yellow pages. Four illustrated examples (A, B, C & D) of the use of the method for different purposes are provided on the green pages at the back of the manual.

References to other literature and online resources are denoted by superscript numbers in the text and are listed towards the end of the manual.

Supplementary materialThis manual is supplied with a CD-ROM containing a MS-PowerPoint version of this manual and a MS-Excel template for use in the calculation of the AFI. The Excel template automatically generates graphical representations of the results.


4

INTRODUCTION

The Agri-Environmental Footprint ProjectThe AE-Footprint project was an EU Specific Targeted Research Project (EU project code SSPE-CT-2005-006491) which ran from April 2005 to March 2008. The project developed a common methodology to assess the environmental performance of AESs.

The Agri-Environmental Footprint Index (AFI)The methodology developed is called the Agri-Environmental Footprint Index (AFI) and is based on Multi-Criteria Analysis1. It is a step-wise process that can be adapted for use in the evaluation of agri-environment scheme policies in all EU member states in any agri-environmental (AE) context. The methodology is particularly suited to the evaluation of multi-objective schemes that encompass wide environmental concerns. The AFI aggregates measured agri-environmental indicators using a hierarchical structure and the application of weights derived through stakeholder participation. Purvis et al. (in prep)2 provides a full description of the methodology.

The AFI is a farm-level index and aims to quantify the agri-environmental performance of individual farms and farm types (through the aggregation of AFI scores for a large sample of individual farms). The potential benefits of this approach for both AES evaluation and basic monitoring of the environmental impact of farming are described below.

The uses of the AFI methodologyThe methodology described in this manual has been worded for the specific purpose of AES evaluation. Within a common agri-environmental context, this use of the AFI methodology can allow the following:

Basic evaluation outputs:• measurement of the environmental impact or performance of farms• measurement of environmental impact that can be aggregated across a larger sample of similar farms to establish a comparative baseline measure• measurement of changing environmental impact over time using repeated snap-shot assessments in comparison to the established baseline

AES evaluation outputs:• evaluation of multiple objectives within a single framework• comparison of the environmental impact of AES participating and non-participating farms• identification of changes in the relative environmental impact of AES participating and non-participating farms over time• identification of unexpected side effects of AESs (positive or negative)


5

INTRODUCTION

Extensions to the uses of the AFI methodology in AES evaluation:• measurement of the effectiveness of agri-environmental policies by tracking AFI scores over time within specific AES policy backgrounds• identification of improvements in AES design

The AFI methodology is NOT designed to:• allow the comparison across different AESs, across different farming types or across different geographical regions/EU Member States• assess the cost effectiveness/cost-benefit of different AES

Alternative uses of the AFI methodology (not described in this manual)The methodology described in this manual has been worded for the specific purpose of AES evaluation. There are other potential uses of the AFI, for example:• evaluation of the environmental impact of other agri-environmental policy mechanisms, such as cross compliance or regulation• tracking/information tool for farmers/regulators to monitor changes in the environmental impact of individual farms over time• ex-ante analysis to give an idea of the future impact of proposed policies

Which evaluation is appropriate for my needs?There are several different types of AES evaluation that can be conducted using the AFI methodology. There are four examples of different types of AES evaluations illustrated in the Examples section at the end of this manual. A simple decision key is shown on page 6 to help inform the decision on which evaluation is the most appropriate for different situations.


6

WHICH EVALUATION?

AES / AEM / AE Policy

Broad Narrow

Strictly policy-oriented

evaluation

Wider environmental

evaluation

Strictly policy-oriented

evaluation

Wider environmental

evaluation

A B C D

Q1.

Q2.

Example

To determine which evaluation is correct for your needs, work through the flow diagram below to establish the type of scheme and type of evaluation you wish to conduct. The flow diagram will help you identify the most relevant example to refer to during the evaluation process.

Q1. How would you describe the specificity of the environmental objectives of the scheme under evaluation?

- Broad = scheme with wide-ranging environmental objectives, - Narrow = scheme with a highly focused and quite specifically stated environmental objective(s) relating to a single, or limited number of particular agri-environmental concerns

Q2. What type of evaluation do you wish to conduct?

- Strictly policy-oriented evaluation, based solely based on the stated AES objectives- Wider environmental evaluation = evaluation based on the AES objectives PLUS additional, wider environmental concerns

Examples: Refer to page X.- A = Strictly policy-oriented evaluation of a broad scheme- B = Wider environmental evaluation of a broad scheme- C = Strictly policy -oriented evaluation of a narrow scheme- D = Wider environmental evaluation of a narrow scheme


7

AFI METHODOLOGY

Set the scope of the evaluation (pages 8-11)Step

1

Specify Assessment Criteria by identifying relevant environmental concerns and appropriate farm management

strategies to deal with them (pages 12-16)

Step 2

Assign weights to the identified environmental concerns and management strategies to deal with them according to their relative importance in the evaluated context (pages 17-19)

Step 3

Identify appropriate farm-level indicators for the agreed assessment criteria (pages 20-23)

Step 4

Collect farm-level indicator data (pages 24-25)Step

5

Convert indicator values to scores (pages 26-27)Step

6

Weight the indicators (pages 28-29)Step

7

Calculate the AFI (pages 30-33)Step

8

Conduct a sensitivity analysis of the results to changes in scores or weights (page 34)

Step9

Having established the scheme under assessment and the type of evaluation you wish to conduct, you are ready to start the step-wise process of the AFI methodology.

There are nine steps in the methodology as described in the flow diagram below. These steps are based on the stages in multi-criteria analysis1 and they should be followed in this order.


8

The first step of the AFI methodology is to establish a clear understanding of the scope and context of the evaluation. The following is a list of points which need to be considered before embarking upon the evaluation process itself. By setting out these considerations at the beginning, many of decisions to be taken at subsequent steps in the AFI methodology become clearer.

a) Specify the agri-environmental scheme/measure/policy to be evaluatedFor clarity, identify the specific AES scheme, measure or policy under evaluation.

b) Specify the agri-environmental context (Box 1)Establish the farming type(s) and geo-climatic region to be considered in the evaluation.

c) Specify the type of evaluation (Box 2)Use the decision key on page 6 to establish the scope of evaluation to be conducted. The options are;

A strictly policy-oriented evaluation - an evaluation solely based on the stated AES objectives; A wider environmental evaluation – an evaluation based on the AES objectives PLUS additional, wider environmental issues/concerns.

d) Specify the time frame (Box 3)If undertaking a repeated, time-series evaluation; set the time frame and frequency of assessments.

e) Sampling strategy (Box 4)Determine whether the evaluation is to be a comprehensive (all farms included) or a sample-based assessment. If the latter, establish the number of farms needed for statistically-representative samples, and any eligibility criteria.

f) Comparisons (Box 5)Establish any sample comparisons that need to be made as part of the evaluation process.

g) Spatial scale (Box 6)It is important to define the term “farm-level” in order to establish the basic unit of evaluation. The resolution of the evaluation should be carefully considered when the evaluated AES has a spatially-defined component;e.g.

schemes with specific boundaries for eligible land schemes where only a certain subset of fields is under agreement

Step 1. Setting the scope of the evaluation


9

Box 2. Type of evaluation Strictly policy-oriented: evaluation solely based on the AES objectives. This option allows a very focussed AE scheme evaluation that only includes assessment criteria that are stated in the AES objectives or scheme literature. Wider environmental evaluation: evaluation based on the AES objectives and additional, wider environmental concerns. This option allows a more holistic evaluation of both AES objectives, and wider environmental concerns that are not explicitly stated in the scheme literature. The inclusion of additional assessment criteria that go beyond the strictly stated objectives of an AES allows identification of unintended, potentially positive or negative side effects of an AES.

Box 1. Agri-environmental context Farm type: for example arable, general cropping, dairy, poultry, mixed farms, or all farm types. Geo-climatic region: examples could include; a specific geo-climatic region associated with a particular type of farming, a specifically defined area, such as a conservation or nitrate vulnerable zone, or an administrative area.

Box 3. Time frame Examples could include; a single snap-shot, a repeated time-series at fixed intervals, or at the start and end of a policy implementation period.

Box 4. Sampling strategy Number of farms: This is dependent on the extent of the evaluation, the resources and time available etc. However, it is important to have a large enough sample size to ensure it is statistically representative for the defined farm context. Eligibility criteria: For example, it might be decided to only include farms that have been in an AES for a minimum period of time, or that are above a defined minimum size.

Box 5. Comparisons Scheme participating vs. non-participating farms: This is a very important comparison. Snap-shot evaluation of mean AFI scores for participants alone will not identify the benefits of the scheme over and above the background changes in the agri-environment. Comparison to a baseline: If non-participants are not available, then comparison of the AFI score for participating farms can be tracked in comparison to their own baseline (e.g. before scheme participation) or over the course of time whilst participating.

Box 6. Spatial scaleFor part-farm schemes and schemes with spatially-defined boundaries it is important to decide whether, for example, to assess the whole farm or just the areas under agreement.


10

Step 1 continued

h) Advisory groupsThe methodology incorporates the participation of stakeholders and technical advisors in designing a customised form of the AFI. Such input of specific, technical and local knowledge helps to clarify the evaluation objectives and their relative importance and ensures the evaluation is appropriate to the local agri-environmental context. It is important at this stage in the methodology to identify and recruit the assistance of relevant stakeholders and technical advisors. Stakeholders (Box 7)For the purposes of this methodology, a stakeholder can be defined as someone who can affect or who is affected by the agri-environment scheme or the local agri-environment. There is no ideal composition for a stakeholder group, but it is advisable to include representatives from the four groups defined4 below. Further considerations for the selection of stakeholders are discussed in Box 7:

Group Examples

Group 1 = People who may affect Farmersthe local agri-environment

Group 2 = People who may be affected Neighbouring producersby the local agri-environmental conditions Local authoritiesand farming practices NGOs

Group 3 = People who act as regulators Ministries officialsfor the agri-environment Local administrators

Group 4 = Environmental/Agronomic Local agronomists/consultantsExperts Researchers

There is no ideal number of stakeholders, but a group of up between 10 -15 members is manageable in generating discussion and building consensus.

Technical advisors (Box 8)There are several points in the AFI methodology where specialist discussions with technical experts may be required. These advisors can be consulted individually, or at discussion meetings involving members of the wider stakeholder group as necessary. This point is discussed further in Box 8.


11

Box 7. StakeholdersThe composition of the stakeholder group will vary according to the evaluation being undertaken, but some considerations for the selection of the final group might be to: include national-level stakeholders when nationally important issues are involved or for the wider-environmental impact evaluations include scheme-specific stakeholders and possibly the scheme policy makers when conducting a scheme-specific evaluation consult scheme designers for their original intentions for the scheme technical specialists and environmental experts familiar with the agri-environmental context

Box 8. Technical advisorsIt might be advisable to specifically convene the technical experts at certain stages in the methodology (e.g. subsequent Steps 4, 6 and 7) to enable cross-discipline discussions in a complex evaluation. The specific creation of a Panel of Technical specialists may be useful for this purpose.


12

Step 2. Identifying environmental concerns and ways to deal with them

When considering evaluation of complex AESs with multiple objectives, an important stage in the process is to structure these objectives in a way that permits an expression of their relative importance (weighting).

The AFI approach is to:

a) Identify and classify the relevant AE-Issues (Box 9)Three fundamental AE-Issue categories (see Box 9 for definitions) can be used to structure most AES evaluations. These relate to the protection, maintenance or enhancement of:

Natural Resources (NR) Biodiversity (B) Landscape (L)

These three categories encompass most current issues, however, further AE Issues can be included where appropriate (Box 10).

b) Identify and classify management practices that influence the identified AE-Issues (Box 11)Within the farming context being evaluated, the influence of management practices can be considered with reference to three categories or Management Options (see Box 11 for definitions) relating to:

Crop and Animal Husbandry (CAH) Physical Farm Infrastructure (PFI) Natural and Cultural Heritage (NCH)

Steps 2a and 2b can be used to create an Assessment Criteria Matrix (ACM) that provides a framework for describing and summarising the specific concerns for the agri-environment to be evaluated (AE-Issues x AE-Management Options - see Box 12):

AE Issues

NR B LOther

(see Box 10)

CAH

PFI

NCH

ManagementOptions


13

Box 9. Identify the relevant AE-Issues Definitions for the three main categories of agri-environmental Issues used in the AFI methodology are: Natural Resources (NR): protection and enhancement of; soil quality, quality and quantity of ground water, air quality and other non-renewable abiotic resources Biodiversity (B): conservation of wildlife, protection and creation of important habitats for wildlife, maintenance of the genetic diversity of cultivated crops and livestock and utilisation of the functional (ecological) benefits of natural biological processes in production systems Landscape (L): protection and enhancement of; landscape quality in terms of aesthetic appearance and multifunctional value, including cultural/regional identity, amenity, recreational and wider societal value

Box 10. Possible further AE IssuesFurther agri-environmental issues can be added to the evaluation structure (ACM) should new trends or dynamics in agri-environmental or related policy emerge, for example: Environmental risk and control: flooding, fire, avalanches Climate change: greenhouse gases, land use, irrigation Health/welfare: nutritional quality of food, food safety/security, farm workers’ health, animal welfare Energy and food supply: energy crops, sustainability of local food supply

Box 11. Identify the relevant AE-Management OptionsThe definitions for the three main categories of agri-environmental management options used in the AFI methodology are: Crop and Animal Husbandry (CAH): farmer management of crop/animal husbandry systems and production methods without the need for major infrastructural investment i.e. routine and ongoing farming practices Physical Farm Infrastructure (PFI): farmer management of basic farm infrastructural facilities i.e. investment in farm layout and functional design/provision of production facilities Natural and Cultural Heritage (NCH): farmer management of the wider countryside beyond the management of the farm ‘production process’, i.e. management of non-crop habitats, culturally-valued features and general appearance of the wider countryside

These definitions can be visually understood by reference to an aerial photograph of a farm provided in Appendix A, which illustrates the concept of the AE-Management Domain categories

Box 12. The Assessment Criteria MatrixThe ACM is essentially a universal template relevant to all possible farming contexts. When filled with customised details for a specific AE-evaluation, it provides a description of locally desirable management practices nested within specified agri-environmental objectives, and a detailed description of what is to be evaluated


14

Step 2 continued

c) Assessment criteria (Box 13)The process of filling an Assessment Criteria Matrix (ACM) with relevant criteria (see Box 13) should begin with reference to the specified objectives of the AE-Scheme being evaluated. For example, AES objectives could be to:i) “Improve opportunities for enjoying the countryside” ii) “Maintain and/or extend wildlife habitats”iii) “Protect the quality of water resources

These objectives need to be worded as assessment criteria based on appropriate farm management practice within relevant dimensions of the ACM, for achievement of the stated objectives. This is best done by formulating an assessment criterion as a question; e.g. i) “Have appropriate opportunities for public access and enjoyment of the countryside been provided by the farmer?” ii) “Are wildlife habitats appropriately managed/have they been enhanced?”iii) “Is the application of animal manures to land hazardous to water quality?”iv) “Do facilities for storage of animal manures constitute a hazard to water quality?”

Such criteria are placed in the appropriate dimension (cell) within the ACM (see below, and Box 13 for further discussion):

A single cell in the ACM is termed an AFI “Dimension”

NR B L

CAHIs the application of animal manures to land hazardous to water quality?

PFI Do facilities for storage of animal manures constitute a hazard to water quality?”

Have opportunities for public access and enjoyment of the countryside been provided?

NCHAre wildlife habitats appropriately managed/have they been enhanced?


15

Box 13. Assessment CriteriaAssessment Criteria can be defined as questions relating to condition, state and/or management practice that affect environmental quality. Ultimately, the AFI methodology requires that Assessment Criteria be converted into one or more measurable indicators that permit assessment of environmental quality with respect to the identified criteria.

In creating an ACM, evaluators with the assistance of stakeholders should aim to define assessment criteria relevant within each AE-Dimension (matrix cell). The objectives of AE-Schemes are often stated in very general terms, however, the interactions between farming practice and such objectives are often many and complex. All such interactions need to be carefully teased out during the stakeholder consultation process.

For example, an AE scheme might only cite protection of the quality of water resources as one of its primary objectives. Such an objective must be ‘translated’ into assessment criteria relevant to the evaluated farming context. In the case of a livestock farming system, Groundwater quality can be protected by appropriate management options with respect to Crop and Animal Husbandry (to prevent diffuse-source pollution) AND through appropriate management options with respect to Physical Farm Infrastructure (to prevent point-source pollution). Therefore, evaluation of this broader policy objective will require at least one assessment criterion in each of the NR x CAH, and the NR x PFI cells of the ACM (see p. 16). Natural watercourses also feed into the groundwater supply. Therefore in order to comprehensively evaluate the original policy objective in this context, it may be necessary to identify an appropriate assessment criterion for the NR x NCH cell of the ACM relating to a farmer’s protection of these natural features as part of his wider management of the countryside (see example ACM on p. 16). The process of identifying comprehensive assessment criteria requires the considered input of stakeholders to identify ALL the likely influences of a farmer’s management on the environment.

In more complex evaluations it might be helpful to break down consideration of the major Agri-Environmental Issues into sub-Issues. For example, subdividing Natural Resources (NR) into ‘Air’ Soil’ and ‘Water’ issues; and Biodiversity into ‘Genetic’ (Crops and Livestock), ‘Wildlife’ and ‘Functional’ (ecological utilisation) issues.

If it has been decided that an evaluation is to be strictly policy-focused, the included assessment criteria should cover all aspects of the stated policy. A wider and more holistic environmental evaluation requires broader thinking to identify ALL potentially important environmental issues or problems that face the area under evaluation. These additional concerns should be expressed in the ACM as additional assessment criteria that go beyond the stated objectives of the evaluated policy.


16

Box 13. Assessment criteria (continued)

NR B L CAH

• Is the application of animal

manures hazardous to water quality?

• Is the application of inorganic fertilisers hazardous to water quality?

• Is farm husbandry damaging to soil structure?

• Is the application of animal manures to land hazardous to air quality

• Are extensively managed cropping areas important for wildlife maintained?

• Are field conservation margins used and correctly managed?

• Is the use of pesticides hazardous to biodiversity

• Are rare animal breeds or crop varieties used?

• Have traditional husbandry methods that contribute to the character of the traditional countryside been maintained?

• Are husbandry practices that conflict with the aesthetic and multifunctional value of the countryside (e.g. land-spreading of animal manures) done in a sympathetic way?

PFI

• Do facilities for storage of

animal manures constitute a hazard to water quality?

• Does animal housing constitute a hazard to air or water quality?

• Are field boundary habitats maintained and correctly managed for wildlife?

• Do facilities for storage of animal manures and/or livestock housing facilities constitute a hazard to biodiversity

Have opportunities for public access and enjoyment of the countryside been provided?

Have traditional farm buildings and other farm infrastructure been maintained to enhance the aesthetic appearance of the countryside?

NCH

• Have natural riparian

buffer margins been maintained to protect water quality?

• Are wildlife habitats (including riparian margins) appropriately managed/have they been enhanced as wildlife habitats?

• Are specific measures implemented for the protection of specified wildlife populations?

• Have natural landscape features been protected and properly managed?

• Have historic & archaeological features been protected and properly managed?

• Has the farmer been given training in local natural and cultural heritage?

An example ACM with generic assessment criteria that might feature in each AE-Dimension


17

Step 3. Assign weights to the major AE-Issues and categories of Farm Management Options according to their relative

importance in the context being evaluated

Agri-environmental measures (especially within multi-objective schemes) are often not of equal importance in achieving the scheme objectives, therefore an accurate evaluation needs to consider the relative importance of the different AE-Dimensions of the AFI. The third stage in the AFI methodology is a weighting process that considers the relative importance of the major AE-Issues, and that of the three categories of farm Management Options in addressing each of the major AE-Issues, in the farming context being evaluated.

Weighting of the AE-Issues reflects the relative importance of abiotic Natural Resources, Biodiversity and Landscape Issues, and in practice the relative demand for these environmental services, in the context being evaluated. The weighting of AE-Management Options within each major AE-Issue, reflects the relative importance of the three categories of farm management practice (CAH, PFI, NCH) in addressing each of the major AE-Issues in turn, and in practice the potential for supplying or providing these services.

The weighting process needs to be performed by the stakeholders consulted throughout the AFI methodology. Stakeholders can be contacted and engaged in this process in a variety of ways, but a suggested protocol for stakeholder interactions can be found in Appendix B. Throughout the weighting process, it is extremely valuable to have stakeholders meet in a face-to-face exercise in which both specialist technical knowledge and direct practical experience of the farming system being evaluated can be exchanged and inform discussions so that consensus can be reached.

a) Assigning weights to the AE-IssuesThe methods for assigning weights between 0 and 1 are shown in Box 14.The AE-Issues should be weighted according to the perceived importance of Natural Resources, Biodiversity and Landscape as services or problems in the region under evaluation. For example in the following table, Landscape is deemed to be the most important Issue with Natural Resources and Biodiversity of comparatively less importance:

Some points that should be considered during the weighting process are discussed in Box 15.

Natural Resources Biodiversity Landscape

0.25 0.25 0.5 ∑=1


18

Box 14. Assigning weights to the AE-Issues and AE- Management OptionsThe ACM has been constructed to simplify the complex task of assessing the relative importance of multiple criteria; by presenting an evaluation as a 3x3 matrix, there are only ever three categories to be considered in the weighting process at any one time.

Suggestions of weighting methodsThere are various available methods for a group of stakeholders to assign weights and collate the results.

Distribution of 100 percentage points: Ask each stakeholder to distribute 100 percentage points between the categories according to their relative importance. Calculate the average value amongst all stakeholders for each category, and convert the result from percentages to proportions on a scale of 0 to 1. Ranking method: Ask each stakeholder to assign the most important category a value of 10 and the other categories appropriate relative rankings on the scale 0-10. Convert the averaged ranking values for each category to proportions on a scale of 0 to 1. Pairwise comparisons: Use Thurstone’s conjugate comparison5 and the Guilford method6 to evaluate the relative importance of the categories. Consensus building: Each stakeholder initially defines their own weights and then through facilitated, constructive discussion, a group consensus is agreed as initial opinions are modified.

Box 15. Considerations during the weighting process For scheme-specific evaluations, the major AE-Issue weights may already have been determined by policy makers during scheme conception. Therefore, unless there is wish to reconsider any existing statement of policy priorities with respect to the major Issues, this participatory approach to weighting maybe unnecessary. For wider environmental evaluations, the stakeholders should always participate in the described weighting process, using their professional knowledge and opinions, broadened through group discussions. For any evaluation, it is important to clearly describe the objectives of the evaluation to the stakeholders at this stage, for example, with respect to regional/national priorities and the other considerations established at Step 1. It is also important to clearly describe the concepts and distinction between the AE- Issues and the AE-Management Options to the stakeholders. Refer to the definitions in Boxes 9 and 11, and the visual representation of AE-Management Option categories provided in Appendix A.


19

Step 3 (continued)

b) Assigning weights to the AE-Management Options (Box 14)The AE-Management Options should be weighted according to the perceived importance of each Management category within each AE-Issue. These weightings should reflect the likely effectiveness of each of the Management Option categories in influencing and achieving AE-Issue goals.

For example:

In this example, it is considered that a farmer’s management of Crop and Animal Husbandry (CAH) is much more important in achieving objectives regarding abiotic Natural resources (protection of soil, water, air) than either management of Physical Farm Infrastructure (PFI) or management of the wider countryside (NCH). In contrast, all categories of farm management are considered equally important for Biodiversity objectives and a farmers management of the wider countryside (NCH) is considered most important in achieving Landscape Issues.

c) Considerations during the weighting processSome points that should be considered during the weighting process are discussed in Box 15.

Natural Resources

Biodiversity Landscape

Crop and Animal Husbandry

0.7 0.33 0.2

Physical Farm Infrastructure

0.2 0.33 0.2

Natural and Cultural Heritage

0.1 0.33 0.6

Totals ∑=1 ∑=1 ∑=1


20

Step 4. Identifying farm-level indicators for the concerns expressed by assessment criteria

a) Choosing farm-level indicators (Box 16)This stage of the methodology requires the identification of measurable, farm-level indicators for assessment of the criteria specified in each cell of the Assessment Criteria Matrix (AE-Dimension). This process essentially converts the developed ACM, which describes what has to be evaluated, into an Indicator Matrix (IM), which specifies the environmental quality measures to be used in evaluating conditions/practices on individual farms. The example below illustrates a possible IM based on the generic ACM shown on page 16.

The chosen indicators should be able to reflect actual environmental quality, or potential environmental hazard. Some general considerations for indicator selection are discussed in more detail in Box 16.

NR B L

CAH • Manure application - water hazard indicator• Inorganic fertilizer - water hazard indicator• Soil compaction indicator• Gaseous emissions hazard indicator

• Proportion of farm extensively cropped• Proportion of field margins with conservation strips• Pesticide use - wildlife hazard indicator• Use of rare breeds/crops indicator

• Use of traditional husbandry indicator• Remedial husbandry measures indicator

PFI • Animal waste storage - water hazard indicator• Animal housing-air/water hazard indicator

• Relative length of traditional field boundaries indicator•Animal waste storage - water hazard indicator• Animal housing - biodiversity hazard indicator

• Public access indicator• Farm yard buildings/ tidyness indicator

NCH • Proportion of river margin protected with semi-natural riparian zones

• Wildlife habitats indicator• Specified wildlife conservation measures indicator

• Management of natural landscape indicator• Management of historic and archaeological features indicator• Farmer - heritage education indicator


21

Box 16. Choosing farm-level indicatorsIt is highly desirable that a separate Panel of Technical Specialists including members drawn from the wider Stakeholder Group should be constituted for the purpose of choosing appropriate indicators.

The chosen indicators should be: relevant to the evaluation objectives measurable at the farm-level able to measure environmental change over time reliable in terms of the relationship between the indicator value and environmental condition obtainable with the resources available

In creating an Indictor Matrix, the same indicator may validly be used in more than one AFI-Dimension. For example, a reduction in nutrient inputs may be a useful indicator for the CAH dimension of both Natural Resources (specifically water quality) and Biodiversity objectives; however, such an indicator may very likely be assigned quite different weights (relative importance) within different dimensions (see Step 3).

Indicators can relate to, but should not directly seek to assess actual compliance with contractual obligations stipulated in the scheme being evaluated. This is important, since the selection of management indicators that merely evaluate contractual obligations will provide only an evaluation of policy implementation, and will not necessarily indicate actual environmental benefit. Indicators should be chosen on the basis that they are independent arbiters likely to reflect actual environmental conditions, or the level of perceived environmental risk.

As assessment seeks to track change in agri-environmental quality over time, indicators that express actual environmental state would normally be preferred7. However, such data can be difficult, frequently impossible, and nearly always prohibitively expensive to collect. Proxy ‘driver’ indicators relating to farming practice that have a clearly understood relationship with actual environmental conditions are frequently the only available cost-effective option.


22

b) Using multi-metric indicator functions (Box 17)

In many cases, identified assessment criteria will relate to relatively complex environmental issues, e.g. an important assessment criterion within the NR x CAH dimension might relate to a farmer’s use of animal manures. The environmental hazard associated with land spreading has multiple facets, including the type of manure being applied (solid or slurry), the application technology used and the time (season) of application, because the pollution risk from N and P contamination of water sources and gaseous losses of NH3 will differ in different seasons.

In such cases, it is unlikely that a single statistic relating to the land-spreading of organic manures can realistically evaluate environmental quality/hazard, and so it will be necessary for evaluators to design a ‘multi-metric’ indicator that aggregates multiple individual statistics relating to manure disposal in order to effectively measure the environmental hazard associated with a particular farmer’s practice. An example of a relatively simple multi-metric indicator structure for assessing the adequacy of field boundary management for protecting biodiversity is given in Box 17.

c) Limiting the total number of indicators used in the IM (Box 18)

As multiple criteria methods work best with a total of between 4 and 20 indicators, it is advisable to limit the number of indicators used within any individual AFI Dimension (cell of the IM) to a maximum of about 5. The use of multi-metric indicator functions (see above) enables this limitation to be met without unduly simplifying very complex issues. Further important considerations about reporting the results of a very holistic AFI evaluation in the event of there being more than about 20-25 indicators in the IM are discussed in Step 8.

d) Checking potential sources of data using a pilot study (Box 19)

There are likely to be numerous potential sources of farm-level data that can be used in the development of environmental indicators. It is always useful to check the availability and accessibility of such data by conducting a small-scale pilot study at this stage in order to compile a realistic Indicator Matrix.

e) Ask the wider Stakeholder Group to approve the finalised Indicator Matrix (Box 20)


23

Box 18. Total number of indicators per AE-DimensionIn order to avoid double counting and redundancy, multiple indicators within the same AFI-Dimension should address clearly different assessment criteria, or be complementary and necessary for accurate evaluation of a single criterion. By limiting the total number of indicators used per AE-Dimension to 5, the subsequent process of weighting the relevant importance of indicators within cells (Step 3) is also made easier.

Box 19. Check sources and availability of indicator dataA wide range of farm-level data sources that have been built up from regular monitoring efforts may be of use in providing data necessary for the quantification of the developed Index. These sources may include:•National databases•European databases (e.g. FADN, IACS, FSS)•Research reports, journal articles, monitoring survey outputs

In most contexts, it is unlikely that all the required data will be available from existing databases and customised farm surveys will very likely need to be conducted (see Step 5). Prior to finalising the IM and committing resources to large-scale data collection, it is advisable to check that all the required data are accessible. Running an initial small-scale pilot study for a small number of farms is a good way to check on data availability.

Box 20. Approval of the finalised Indicator Matrix by the stakeholdersAlthough not essential, it is strongly recommended, that a meeting of the wider Stakeholder Group is also convened to discuss and approve the use of the final form of the Indicator Matrix before resources are committed to full-scale data collection. This will ensure that the most appropriate indicators have been selected for accurate assessment of the identified criteria, and that there is general confidence in the utility of the developed Index.

Box 17. A multi-metric indicator functionAssessment Criterion = Are field boundary habitats managed for biodiversity?Multi-metric indicator = Proportion of hedgerows appropriately managed for wildlife, in particular farmland birds; measured as the average combined score derived from two categorical management practice measurements:

1. Timing of hedge cutting AFI Score

January – February 10

March – August 0

September - December 5

2. Frequency of hedge cutting AFI Score

Annual 2

Left unmanaged 5

Mixture of annual and 2-3 years 6

Every 2-3 years 7

Mixture of 2-3 years and some unmanaged

10


24

Step 5. Collecting farm-level indicator data

a) Large-scale data collection This stage involves the collection of farm-level indicator data for large, statistically representative samples of farms, either from existing databases, from a specifically commissioned farm survey, or most likely, from a mixture of these possibilities (see comments in Box 19 regarding the checking of data availability).

b) Farm survey data collection (Box 21)If specifically commissioned data collection is necessary, an appropriate farm survey questionnaire needs to be developed (see Box 21).


25

Box 21. Farm survey data collectionQuestions to be used in a farm survey questionnaire need to be structured carefully to ensure that farmers can easily and quickly provide accurate answers from which the necessary information for indicator construction can be extracted. Appendix C provides a protocol for interactions with farmers and land owners in order to collect indicator data through survey methods.


26

Step 6. Converting indicator values to scores

Converting indicator values to scores (Box 22)Having collected indicator data from evaluated farms, these data need to be converted into AFI scores on a standardised scale of 0-10, whereby 0 = ‘very bad’, and 10 = ‘best possible’ environmental quality. The conversion from original indicator values (expressed on different scales) to a standardised indicator score needs to take account of several factors.

There is a definite need at this stage to consult technical experts, scientific literature, scheme documentation etc. to ensure that the correct scaling system is developed for each indicator. This task and the rules governing transformation will be unique to each indicator used within specific applications of the index as the relationship between indicator values and AFI scores will be strongly dependent on the evaluation undertaken, the farming type and agri-environmental context. Also, a particular indicator that occurs in more than one Dimension of the Indicator Matrix is likely to require different scaling according to whether it is relating to, for example, Natural Resources or Biodiversity.

Some potential relationships between indicator values and indicator scores include:

Linear relationships:

Non-linear relationships:

Categorical scaling:

Binary scaling:

Value X Y

Score 0 1 2 3 4 5 6 7 8 9 10

Value X Y

Score 0 1 3 5 8 10 7 5 3 1 0

Value A B C D

Score 0 1 2 3 4 5 6 7 8 9 10

Value No Yes

Score 0 1 2 3 4 5 6 7 8 9 10

NB. ‘Bad’ = 0 and ‘Good’ = 10


27

Box 22. Converting original indicator values to standardised AFI scoresFor each indicator there needs to be a set of rules guiding the transformation of measured indicator values into the standardised AFI scoreing scale of 0 - 10. Considerations during this process include:

An initial pilot study (step 4d) will considerably help the process by providing knowledge of the current range in indicator valuesThe scoring range needs to be set so as to permit future documentation of both likely improvement and deterioration in environmental conditions. This is achievable by ‘benchmarking’ the central value of the standardised AFI scoring scale (5), to the average indicator value observed in an initial small-sale pilot study. If on subsequent evaluation of a much larger sample the observed indicator mean is considerably different, an adjustment of scaling may be necessary to ensure such benchmarking. Other criteria, including concepts such as ‘Good Agricultural and Environmental Condition (GAEC) and strategically set optimum targets for best environmental quality may be similarly ‘benchmarked’ at the lower and upper ends of the standardised (0-10) scale, respectively.

Examples of ways to transform indicator values to AFI scores:

A simple indicator:Assessment Criterion = Does livestock production impact on air quality?Indicator = Stocking density measured as livestock units (LU) per ha Scoring system:

LU/ha Value >1.8 <=1.8 <=1.4 <=1 <=0.6 <=0.2

AFI Score 0 2 4 6 8 10


28

Step 7. Weighting indicators in AE-Dimensions

Weighting the indicators (Box 23)This is the final stage of weighting in the AFI process and should be done by technical experts, using either the arithmetic average of individual views, or working as a group towards a consensus view. It may be beneficial to convene a meeting of technical experts, so as to generate discussion and an improved collective understanding amongst the individuals involved.

During this process, the indicators within each dimension of the Indicator Matrix should be given a weight that reflects their relative importance in achieving aims within the particular dimension of the evaluation, and NOT on their quality (information value or accuracy) as indicator statistics (see example indicator matrix showing indicator weights in Box 23). The indicator weights within each AE-Dimension should always sum to 1 (see Step 3 for a reminder about the basic weighting process).

The weighting of indicators within AE-Dimensions is an optional part of the AFI process, depending on the scope and aims of the evaluation (see Box 24)


29

Box 23. Weight the indicatorsExample indicator matrix showing indicator weights in brackets:

NR B L

CAH • Manure application - water hazard indicator (0.4)• Inorganic fertilizer - water hazard indicator (0.3)• Soil compaction indicator (0.2)• Gaseous emissions hazard indicator (0.1)

• Proportion of farm extensively cropped (0.3)• Proportion of field margins with conservation strips (0.3)• Pesticide use - wildlife hazard indicator(0.2)• Use of rare breeds/crops indicator (0.2)

• Use of traditional husbandry indicator (0.6)• Remedial husbandry measures indicator (0.4)

PFI • Animal waste storage - water hazard indicator (0.6)• Animal housing-air/water hazard indicator (0.4)

• Relative length of traditional field boundaries indicator (0.5)•Animal waste storage - water hazard indicator (0.3)• Animal housing - biodiversity hazard indicator (0.2)

• Public access indicator (0.5)• Farm yard buildings/ tidyness indicator (0.5)

NCH • Proportion of river margin protected with semi-natural riparian zones (1.0)

• Wildlife habitats indicator (0.6)• Specified wildlife conservation measures indicator (0.4)

• Management of natural landscape indicator (0.3)• Management of historic and archaeological features indicator (0.3)• Farmer - heritage education indicator (0.4)

Box 24 Weighting indicators is optional Indicator weights allow the evaluation to reflect the relative importance of underlying environmental criteria as measured by the chosen indicators. The influence of individual indicators is greatest in a relatively narrow policy-focused evaluation, where only a limited number of AE-Dimensions have been populated in the ACM/IM. This is because when many AE-Dimensions are empty, the earlier weighting of AE Issues and AE Management Options has only a limited effect on the final AFI score. The weighting of indicators within populated AE-Dimensions is then the best way to reflect the perceived relative importance to the environmental issues being addressed by the evaluated policy. In a holistic evaluation going beyond the stated aims of policy (so that most of the AE-Dimensions are populated), the overall AFI score will most strongly reflect the relative importance assigned to the AE Issues and AE Management Options. In such situations, because of the hierarchical mathematical structure of the AFI, weighting indicators within AE-Dimensions will have relatively little additional impact on the Index. The necessity to weight indicators therefore greatly depends on the the kind of evaluation being undertaken and the structure of the AFI created.


30

Step 8. Calculating the AFI

An Excel template is provided for the calculation of the final AFI score for each farm. This is done by summing scores multiplied by relevant weights at each level within the hierarchical AFI structure as shown in Boxes 25 and 26.

For a scheme evaluation, the average AFI score for participating farms compared to the average AFI score for the non-participating farms in the sample allows a straight-forward impact assessment of the scheme. This comparison is automatically displayed in the Excel template as a bar chart, however box-and-whisker plots (produced using other statistical software packages) can display the same information along with the range in the observed data.

The final AFI score provides a useful summary of the findings, however, there is a lot of very useful and important data that can be displayed by breaking down the AFI into constituent parts. For example, the stacked graphs below show the relative contribution to the final AFI scores of the three agri-environmental Issues:

6.53 4.87

0

1

2

3

4

5

6

7

8

AF

I sco

re

Participating

Non-participating

0

1

2

3

4

5

6

7

Participating Non-participating

AF

I sc

ore Landscape

Biodiversity

NaturalResources


31

Box 25. Hierarchical structure of the Agri-Environmental Footprint Index showing the three nested weighting stages

The AFI is calculated by an aggregative process multiplying indicator scores (S) by weights and summing at each successive level of the index: firstly Indicators within Dimensions, Management Options within Issues, then Issues within the overall index. For example:

NRxCAH = 6.1

NRxPFI = 7.5

NRxNCH = 4.9

BxCAH = 9.1

BxPFI = 6.3

BxNCH = 4.5

LxCAH = 8.5

LxPFI = 7.4

LxNCH = 3.9

AFI

Natural Resources (0.5)

Biodiversity (0.3)

Landscape (0.2)

We = AE Issue Weights

CAH PFI NCH (0.6 0.3 0.1)

Wf = AE Management Option weights

CAH PFI NCH (0.4 0.5 0.1)

CAH PFI NCH (0.1 0.8 0.1)

Wi = Indicator weights

I1 = 0.1 I2 = 0.3 I3 = 0.1 I4 = 0.4 I5 = 0.1

I1 I2 . . .

I1 I2 . . .

I1 I2 . . .

I1 I2 . . .

I1 I2 . . .

I1 I2 . . .

I1 I2 . . .

I1 I2 . . .

Wi x S I1 = 0.1 x 4 I2 = 0.3 x 6 I3 = 0.1 x 2 I4 = 0.4 x 7 I5 = 0.1 x 9

(∑Wi x S) x Wf 6.1 x 0.6 7.5 x 0.3 4.9 x 0.1

NR = 6.4

B = 7.24

L = 7.16

(∑ (∑Wi x S) x Wf) x We) 6.40 x 0.5 7.24 x 0.3 7.16 x 0.2

AFI = 6.8


32

Step 8 (continued).

Another graphical representation of the constituent parts of the AFI is to show the relative contributions of the nine Dimensions to the final score:

This way of presenting the outcome of an evaluation is very useful in instances where the total number of indicators in the IM exceeds about 20-25. since in such cases, aggregation of the AFI to higher levels may well result in a statistic that is relatively insensitive to change in its constituent parts.

0

2

4

6

8

10NRxCAH

NRxPFI

NRxNCH

BxCAH

BxPFIBxNCH

LxCAH

LxPFI

LxNCH Participating

Non-participating


33

Box 26. Example of AFI calculation for a single farm

Ex

am

ple

Far

m

1

AE

ISS

UE

Wei

ght

(We)

DIM

EN

SIO

NW

eigh

t (W

d)IN

DIC

AT

OR

Wei

ght

(Wi)

sco

resc

ore

x W

i∑

(sc

ore

x W

i)∑

(s

core

x

Wi)

x W

d

∑ (

scor

e x

Wi

x

Wd)

∑ (

scor

e x

Wi

x

Wd)

x W

e

Nat

ural

Res

ourc

es0.

25C

AH

0.70

Pes

ticid

e in

puts

0.50

84

6.20

4.34

5.74

1.44

Leve

l of

soil

com

pact

ion

0.10

40.

4N

itrat

e lo

ss0.

204

0.8

Leve

l of

gree

nhou

se g

as e

mis

sion

s0.

205

1∑

1.00

PF

I0.

20P

ropo

rtio

n of

man

ure

stor

age

faci

litie

s se

aled

cor

rect

ly1.

002

22.

000.

40∑

1.00

NC

H0.

10P

ropo

rtio

n of

riv

er le

ngth

pro

tect

ed w

ith b

uffe

r st

rips

1.00

1010

10.0

01.

00∑

1.00

∑1.

00

Bio

dive

rsity

0.25

CA

H0.

33R

atio

of

sprin

g:w

inte

r cr

ops

0.33

72.

314.

951.

652.

940.

73P

ropo

rtio

n of

live

stoc

k fr

om r

are

bree

ds0.

335

1.65

Pro

port

ion

of f

arm

land

as

exte

nsiv

e gr

assl

and

0.33

30.

99∑

0.99

PF

I0.

33P

ropo

rtio

n of

hed

gero

w p

rote

cted

with

buf

fer

strip

s/gr

ass

mar

gins

0.70

53.

267

3.87

1.29

Leng

th o

f be

etle

ban

ks0.

302

0.6

∑1.

00N

CH

0.33

Pro

port

ion

of w

oodl

and

unde

r ap

prop

riate

man

agem

ent

1.00

00

0.00

0.00

∑1.

00∑

1.00

Land

scap

e0.

5C

AH

0.40

Div

ersi

ty o

f cr

oppi

ng ty

pes

0.40

41.

47.

402.

965.

672.

84E

vide

nce

of m

ixed

far

min

g0.

6010

6∑

1.00

PF

I0.

40P

ropo

rtio

n of

hed

gero

w le

ngth

rem

oved

0.40

93.

429

5.28

2.11

Pro

port

ion

of tr

aditi

onal

bui

ldin

gs u

nder

app

ropr

iate

man

agem

ent

0.50

31.

25Le

vel o

f fa

rm ti

dine

ss0.

106

0.6

∑1.

00N

CH

0.20

Pub

lic r

ight

s of

way

(m

/ha)

0.50

10.

53

0.6

Pro

port

ion

of h

isto

ric &

arc

haeo

logi

cal f

eatu

res

unde

r ap

rop.

man

gmnt

0.50

52.

5∑

1.00

∑1.

00∑

1.00

AF

I=5.

01


34

Step 9. Conduct a sensitivity analysis of the results to changes in scores or weights

As standard good practice, it is usual to undertake an analysis of the sensitivity of the evaluation to changes in the weights or scores. The main reason for undertaking a sensitivity analysis is to question and validate the “robustness” of the AFI. At many stages we are in fact making best estimates based on the thoughts of a limited number of stakeholders or experts and thus it is important to question the degree to which the AFI changes if the values of Issues, Foci and indicator weights change (i.e. effectively representing the thoughts of a different set of stakeholders). Sensitivity analysis usually involves testing the degree to which the output variable (in this case the AFI score at aggregate level) changes in response to changes in a single input variable. However, sensitivity analysis of the AFI is slightly complicated by the fact that a single input variable cannot be changed. For instance we cannot just change weighting of a single Issue (i.e. Biodiversity) as this affects the weighting of the two related Issues (Natural Resources and Landscape).

Therefore, a suggested approach is to look at the range of weights used in the AFI calculation in turn to see if the vagueness about the inputs or disagreements between people makes any difference to the final AFI scores (especially the difference between the average participating and non-participating AFI scores). The indicator scores are measured values and therefore can’t be manipulated, but the weights and the conversion rules can be contentious and changes in them may shift the order of final AFI values.


35

REFERENCES

1. Office of the Deputy Prime Minister (ODPM) (2000) DLTR Multi-Criteria Decision Analysis Manual. Downloaded 12 October 2007. http://www.communities.gov.uk/archived/publications/corporate/multicriteriaanalysismanual

2. Purvis, G., Louwagie, G., Mortimer, S. Finn, J., Park, J. & others (in prep). The Agri-environmental Footprint Index (AFI): a harmonised method for evaluating the effectiveness of EU agri-environment schemes.

3. http://ec.europa.eu/agriculture/envir/index_en.htm#intro 4. Beopoulos, N., and Vlahos, G. 2005: Desertification policies in Greece:

Implementing policy in an environmentally sensitive livestock area. In: Wilson ,G.A., and Juntti, M. (eds): Unravelling Desertification, policies and actor networks in Southern Europe. Wageningen Academic Publishers, pp 157-178.

5. Thurstone, L.L. (1927). A law of comparative judgement. Psychological Review, 34, 278-286.

6. Guilford, J. P. (1936). Psychometric methods. New York: McGraw-Hill.7. Smeets, E., Weterings, R., 1999. Environmental indicators: Typology

and overview. EEA Technical report 25, European Environment Agency, Copenhagen, 19 pp.


http://www.communities.gov.uk/archived/publications/corporate/multicriteriaanalysismanual

http://www.communities.gov.uk/archived/publications/corporate/multicriteriaanalysismanual

http://ec.europa.eu/agriculture/envir/index_en.htm#intro

1 the agri-environmental footprint index users’ manual an output of the ae-footprint project...

Documents

evaluation of aes

aes evaluation

environmental science

environmental concerns12step

environmental research

manualthis manual

wider environmental

ae footprint