Evaluating Conservation Projects

Current Approaches, Perceived Challenges, and Future Directions

The ProblemDespite effort put into conservation, success at improving state of biodiversity not widespread:23% mammals, 46% amphibians, 61% of reptiles evaluated by IUCN are on their Red List of threatened species 180 million hectares of forest lost between 1980 and 1995; 12 million ha/yr.How can evaluation be modified to help inform and improve conservation projects?

Presentation OverviewBackgroundThree ideal components for evaluationCurrent evaluation frameworks used in conservation LFA, RBM, PCMAnalysis of these frameworks with respect to ideal componentsFuture directions, barriers, next steps

A Paradigm Shift in Conservation

the major limitation of conservation biology: conservation problems are usually more social, economic and political than biological. U.S. trade relations with developing countries, for example, will have a far greater impact on biological diversity than will biodiversity projects funded by USAID. (McNeely,1989, p. 418; emphasis added).

Conservation and Development

The Importance of Evaluation

Conservation/Development projects areComplexDynamicInterdisciplinaryLong term

Need adaptive management = systematic way of learning by doing where evaluation is incorporated throughout the project cycle

Ideal Evaluation Components

Analysis of Evaluation FrameworksLooking at conservation projects specifically

Inclusion: To what degree are the 3 ideal components of evaluation being incorporated into these frameworks in theory and in practice?

Challenges: What are the challenges to integrating these components in existing frameworks? Is it possible?

Future Directions: How can evaluation of conservation projects provide better information for project improvement? What are the barriers?

Main Approaches to Evaluating Conservation Projects

Logical Framework Analysis (LFA)

Results-Based Management (RBM)

Project Cycle Management (PCM)

Logical Framework Analysis (LFA)Source: Modified from Stem et al. (2005)

Results Based Management (RBM)InputsActivitiesOutputsOutcomesImpactsThe Results Chain:

Project Cycle Management (PCM)Source: Modified from Salafsky and Margoluis, 1999 Source: Hockings et al., 2000

LFA & Evaluating Conservation ProjectsIn Theory main criticisms:Oversimplified and rigidFocus on measurable indicators Limits participation when logframe constructed at proposal stage

In Practice:Look at 10 GEF funded conservation projects showed little participation, no account of local perspectives, and focus on biological

RBM & Evaluating Conservation ProjectsIn Theory main criticisms:Over-focus on quantitative performance indicatorsLack of attention to process Attribution of cause and effect

In Practice:Report from CIDA (CIDA, 2002) indicates that RBM in practice keeps control in hands of donors as opposed to local communities, and CIDA is reluctant to address qualitative indicators.

PCM & Evaluating Conservation ProjectsIn Theory:Strength is the iterative cyclical approachWeakness lies in cost and time requiredMargoluis and Salafsky (1998) version less focus on measurable indicators, more focus on focus groups, interviews, community involvement

In Practice:Look at case studies showed very little community involvement in evaluation in either approach

What can be drawn from the analysis?Evaluation of conservation projects characterized by:A heavy emphasis on quantitative biological indicators Lack of inclusion of community perspectives in evaluationsLimited participatory approaches

What can be drawn from the analysis?Degree of difficulty in incorporating 3 components into frameworks:

Future Directions in Evaluation of Conservation ProjectsNeed to question appropriateness of LFA and RBM for informing project improvementNeed to look at PCM as a more useful approach for learning, with the caveat of:Shifting away from emphasis on biological data onlyIncreasing acknowledgement of importance of community perspectivesIncreasing participatory approaches

Future Directions in Evaluation of Conservation Projects (contd)How?Interdisciplinary evaluation teamsQualitative data collection methods such as interviews, focus groups, PRA Flexibility from donors on the evaluation processLearning from experience of others organizations as well as disciplines

Barriers

Lack of resources and capacityUncertainty about approaches and methodsResistance to changeResistance to giving up controlConcern of validity of participatory approaches

Next StepsMeta-evaluation of evaluation in conservation projectsComparison of information gained, participant satisfaction, and decisions taken amongst the different evaluation frameworks usedStudy of the barriers practitioners face in implementing these components, and how to overcome themGuidelines on how to better incorporate these three components into evaluation

ConclusionsEvaluation of conservation projects can provide better information to inform project improvement if they include the three components of:Integrated socio-economic informationCommunity perspectivesParticipatory approaches

By adapting evaluation tools, approaches and lessons learned, can learn how a project can better meet social and biodiversity goals

Thank You !

This presentation is focused on looking at how conservation projects are evaluated current approaches, challenges or barriers, and future directions.

Despite all of the efforts put into conservation, success at improving the state of biodiversity is not widespread

Project evaluation can provide a significant impact in the learning process to improve these projects, and yet does not seem to be carrying out this role in conservation at the present time

My over-riding question therefore is what needs to be done to ensure that evaluation of conservation projects are contributing the information that is needed to assess what is working well, what isnt working well, and what needs to happen to improve a project.To start, Im going to give a bit of a background or context for the presentation starting with explaining what I mean by conservation projects, and how the field has changed over the past few decades, and then talking about some of the characteristics of these projects that make evaluation essential for understanding how these projects are doing.

Im going to talk about 3 components to evaluation that ideally should be included in evaluation of conservation projects

Ill take a brief look at the three main frameworks currently being used to evaluate conservation Logical Framework Analysis, Results-based Management, and Project Cycle Management

Im going to look in turn at each of these approaches and discuss how these frameworks incorporate the 3 ideal components in theory and in practice, with respect to conservation projects in particular.

Ill summarize some of the key points that emanated from the analysis including the compatibility of each of the approaches to providing the information we need to improve conservation and development projects;

Conclude with the change that needs to happen, what some of the barriers are, and next steps to improve the evaluation of conservation projectsTo start off Id like to set the context of the type of projects I am talking about.

Traditionally biodiversity conservation projects have been thought of protected area creation and management.However biodiversity conservation as a field has undergone a paradigm shift in the past 3 decades from protected area creation at the expense of local community members, to initiatives that integrate biodiversity conservation objectives with social and economic development.

This quote by McNeely points to the need for this shift in thinking: the major limitation of conservation biology is that conservation problems are ..

This paradigm shift in theory means that there is increased acknowledgement that conservation problems are social problems, and are a result of the social, economic, cultural and political influences on an individuals land use and resource use decisions and therefore these aspects need to be addressed in conservation project planning and implementationSo rather than just protected areas, what I mean by conservation projects is a wide diversity of initiatives that include things such as:

Agroforestry systemsEducation programsSustainable agricultureEndangered species recovery plansSustainable forest managementOther strategies

Im speaking about it mostly in a developing world context. Where biodiversity is high, poverty is high, use of agriculture high.Conservation problems are social problems and as such they are:Complex involving multiple actors, locations, agency levels, etcDynamic in a constantly changing environmentInterdisciplinary are an integration of cultural, social, economic, political and ecological influencesLong term in that there is a long time lag in seeing change and results

In this case evaluation is especially important in order to:Find out how a project is working (what worked well, what didnt work well)And to adapt and improve upon the program - what can be done to maximize positive impacts and minimize the negative

Information is needed to make decisions today on activities that may not show results for years

This idea is summarized in the term adaptive management, defined by Margoluis and Salafsky (1998) as a systematic approach to learning by doingIn order that the necessary information is gained in order to adaptively manage a project to improve a project for the better a project that is complex, dynamic, interdisciplinary and reflects this paradigm shift of integrating biodiversity conservation with community development, Ive proposed that the following ideal components are necessary in an evaluation process:

Integration of socio-economic with biological informationConservation are social problems therefore looking just at biological data is not going to provide input to improving solutionsBlack-foot ferret example evaluating the project based on biological indicators only shows large project success, because the numbers of the species are increasing due to captive breeding programs. However, their habitat is being destroyed rapidly due to intensive agriculture and residential land uses this is what will ultimately lead to the species demise, therefore the biological information that is being collected is not successfully assessing how well the project is doing in terms of reducing the endangerment of this species.

2) Incorporation of the perspectives of local community membersThose who are directly impacted by a project are in the best position to sense any changes. Local community members can: Provide important information to fully understand the context in which the project is operatingBring attention to unintended positive and negative impactsProvide suggestions for change that may not have been considered by project administrators

3) Participatory decision-makingThis component consists of true participation not simply consultation on or during the evaluation but participation in the decision-making of the evaluation process, collection of data, analysis of the information, and decision-making on what to do with the resultsSuch participation provides many benefits including:Context specific solutions that meet both community and conservation goalsInvolving participants in decision-making can best bring about desired behaviour changesOwnership of the projectEmpowerment in decision-making

All 3 of these components are needed in order to truly evaluate how a project is doing, and to encourage project improvement.So what I wanted to look at, therefore, was:

First of all - what approaches are being used to evaluate conservation projects?

1) INCLUSION: To what degree are these 3 ideal components community perspectives, participatory approaches, and interdisciplinary information being incorporated into these evaluation approaches in theory and in practice. For all three of the frameworks, the guidebooks described in detail the value of involving key stakeholders at all stages of an evaluation process, and the need to look at social as well as biological information, however I found that what is stated in theory didnt always occur in practice.

2) COMPATIBILITY: Are the frameworks compatible with the three components?

3) FUTURE DIRECTIONS: How can evaluation of conservation projects be improved to provide better information for project improvement

From my literature review, there are 3 main approaches that are currently being used for conservation projects based off of three approaches to project management, and dictated mainly by the main funding agencies that support conservation and development projects.

LFA being used by several Commissions of the World Conservation Union (IUCN), the UNDP, and used to be used by GEF, CIDA, USAIDRBM currently used by CIDA and USAID, and recently as of February of this year by GEFProject Cycle Management used by the IUCN Commission on Protected Areas, WWF, and TNC.

Im only going to go through these briefly here, as the majority of you are familiar with these evaluation frameworks:

LFA consisting of a logic matrix to outline the goals (impacts), purpose (outcomes) outputs and inputs for a project, concentrating on how inputs can be turned into results.Results-based management came out of the increased need for public accountability and therefore focuses on results as opposed to inputs. Outputs and outcomes are what are looked at mostly in evaluations, as these are measurable within the funding timeframe of a project.And project cycle management which emphasizes an interative cyclical approach to project management where evaluation feeds into all stages of project implementation.

In my analysis I looked at two versions of PCM

This one, eminating out of the work done by the Biodiversity Support Program in the 1990s, described by Salafsky and Margoluis. This model was influenced by LFA, RBM, work done on Biodiveristy Support Programs, strategic planning, and the concept of adaptive management. So what were some of the key findings from my analysis of the literature?

Logical Framework Analysis has been discussed to death in the literature! The strength of the approach lies in its usefulness as a planning tool, and its simplicity. However for complex conservation and development projects its main limitations lie in:

An oversimplification of how a projects initiatives can lead to results paying little attention to possible non-linear processes of change, and other processes such as interaction of group members that can influence project success. This linear process stems from business and logistics planning where projects are predictable and linear something unseen in complex world of social and environmental issuesThe focus on measurable indicators to fill out the matrix in the case of social problems, where social goals are less tangible than # of species off the endangered list such focus on the measurable is impractical. The focus on the measurable means that much information that can provide information on how a project is doing and how it can do better is being missed e.g. rich qualitative data, perspectives etc.Participation with a logframe is possible as we saw in Heather Perkins presentation before lunch, but when a logframe has to be put together as part of a funding proposal, it negates the possibility of local community participation.

I took a quick look at 10 GEF funded conservation projects available on the GEF website to see the degree to which the 3 ideal evaluation components were included. Unfortunately there is often very little about the way an evaluation is done written up in these reports. From my look, there was little evidence of participatory approaches, of the inclusion of community perspectives, and a focus on quantitative indicators that linked results directly with project objectives, without looking at possible unintended impacts.

References:

Des Gasper, 2002Lucy Earle, 2002Crawford, 2005Results-based management has been a more recent adoption by many funding agencies in fact, GEF just recently adopted an RBM approach to evaluation while I was in the middle of writing my paper in February!

There are 3 main characteristics of an RBM approach that make it challenging to see its compatibility with conservation projects:

The first, as seen in LFA, is an emphasis on quantitative performance indicators. Even though the CIDA guidelines talk about the importance of qualitative as well as quantitative information, the guides suggest quantifying such indicators like participation, satisfaction, etc., into numbers in order to compare before and after effects. An example is seen in CIDAs (1999) definition of qualitative indicators as judgement and perception measures of congruence with established standards, the presence or absencee of specific conditions, the extent and quality of participation, or the level of beneficiary satisfaction, etc.. This focus of RBM is incompatible with the needs of evaluating social as well as biodiversity goals, as many of these goals such as empowerment, improved well-being, food security are not easily measurable by numbers.

2) Attribution of cause and effect RBM came out of a call for increased governmental accountability to public spending, and therefore is very much rooted in the idea that visible effects/impacts can be directly attributed to a projects or programs initiatives and activities. However in a complex situation such as conservation and community development, there are any number of external factors that influence human behaviour and land use decisions from economic policies to political atmosphere to social interactions and it is extremely difficult to claim with certainty that a projects actions is responsible for the results seen.

3) As seen in LFA, RBM also focuses on outcomes and outputs at the expense of the process to reach these outcomes. Patrick Kilby talks about pro-social citizenship behaviour the will to volunteer, the influence of personal relations on project success, informal and formal conflict management, etc., and how this can influence the results or outcomes of a project. By not looking at this aspect of project management, we are missing out on important information that can help improve a project.

Mark Schacter from the (Canadian) Institute on Governance in a 1999 paper felt that RBM was more appropriate for projects which are 1) self-contained; 2) have tangible outputs; 3) have discrete and well defined problems; 4) have a clear observable progression from inputs to outputs to outcomes; and 5) where the organization has a high degree of control over design and direction. None of these characteristics are fitting of conservation and development projects, which instead have intangivle outputs, unclearly defined problems, progress from input to outputs not clear, immediate cause and effect relationships not easily observed.

Similar to LFA, the guidebooks for using RBM such as CIDAs RBM policy espouse that participatory and multidisciplinary approaches should be included at all stages of the evaluation process. However, a 2002 report by CIDA on how the RBM process has been adopted to date indicates that:The focus of RBM in practice is more on compliance than on learningDonors still have the upper hand, even though the intention of many projects is to pass control to local partnersThose interviewed in the report expressed that CIDA was reluctant to address qualitative indicators, and focused more on things that could be counted.

References:Crawford, 2005Lucy Earle, 2002Mark Schacter, 1999Kilby 2004 RBM incomptable with empowerment

The main strengths of PCM, in both approaches looked at, lie in the iterative cyclical approach to project management where evaluation is a key part of each stage of implementation.

Key weaknesses lie in the increased amount of time and resources required to conduct this continuous evaluation, as opposed to a one-time evaluation at the end of project activities.

Despite these benefits, a look at the case studies and examples provided in both of these publications had a very limited amount of public participation to get perspectives on the project activities, or to involve them in decision-making.What messages can be drawn from the analysis?

Evident from this analysis of the literature that the current situation of how conservation projects are evaluated are characterized by:An over-emphasis on quantitative biological indicatorsLack of attention to community perspectivesLimited participatory approaches

In terms of the compatibility of these 3 frameworks to the three components discussed at the beginning of the presentation:

I feel that the overemphasis of both LFA and RBM on measurable indicators limits and challenges its compatibility with a qualitative assessment of intangible social goals seen through the integration of socio-economic information and community perspectivesBoth could be compatible with participatory approaches however if participation continues to be limited in project planning and implementation, it is questionable how effective participation in evaluation will be. It also raises the question of whose objectives are being evaluated? At what point does the community have a chance to express their goals for the project, which are the focus of the evaluation?PCM shows the most compatibility to all 3 of the ideal evaluation components and offers the most benefits to conservation and development projects through the iterative project cycle and evaluation at all project stages. However there is limited evidence that this is occurring in practice.

Plus there is the idea of whose objectives are being tracked. Where does the community get involved in deciding project objectives?

What messages can be drawn from the analysis?

Evident from this analysis of the literature that the current situation of how conservation projects are evaluated are characterized by:An over-emphasis on quantitative biological indicatorsLack of attention to community perspectivesLimited participatory approaches

In terms of the compatibility of these 3 frameworks to the three components discussed at the beginning of the presentation:

I feel that the overemphasis of both LFA and RBM on measurable indicators limits and challenges its compatibility with a qualitative assessment of intangible social goals seen through the integration of socio-economic information and community perspectivesBoth could be compatible with participatory approaches however if participation continues to be limited in project planning and implementation, it is questionable how effective participation in evaluation will be. It also raises the question of whose objectives are being evaluated? At what point does the community have a chance to express their goals for the project, which are the focus of the evaluation?PCM shows the most compatibility to all 3 of the ideal evaluation components and offers the most benefits to conservation and development projects through the iterative project cycle and evaluation at all project stages. However there is limited evidence that this is occurring in practice.

Plus there is the idea of whose objectives are being tracked. Where does the community get involved in deciding project objectives?

What needs to happen in order to improve the evaluation of conservation projects, in order to provide the information that is essential for assessing how well a project is doing, and how it can be improved for better reaching both social and biological goals?

Need interdisciplinary evaluation teams to increase the integration of socio-economic information:Need to increase use of qualitative approaches to get community perspectives and socio-economic information:Interviews, focus groups, PRANeed to increase true participatory approaches to evaluations and overall project management. This needs to be done in a way that ensure culture and gender sensitivity. Need to ask:Are the goals of the project valid? Are they a priority?What is working well in carrying out the strategies to reach the goal?What is not working well?What does the community want to see changed and why?What are the hidden impacts and outcomes, both positive and negative?What can happen to better reach the goals

Need to ensure evaluation for learning Action Research approach of act-observe-reflect-act-oberve-reflectNeed to ensure evaluations are used presented in multiple formats in order that all audiences can benefit from what was learned

How?Horizontal learning across disciplines there is ample information out there from the fields of education and social development on evaluation approaches, their successes and failures with approaches ranging from Responsive Evaluation to Participatory Evaluation to Utilization-Focused Evaluation -A lot of this is applicable to evaluating conservation projects, and can help guide the creation of appraoches specific to the context of conservationSome of the barriers to adopting such changes that have been mentioned in the literature with respect to increasing the importance of evaluation in general, but that are also applicable to incorporating these three ideal components include:

Lack of resources and capacity for conducting evaluationsUncertainty about which approaches and methods can be used to get the information needed for a good evaluation for project improvementResistance to change from current practices, or resistance to wanting to give up the control of the evaluation and project implementationAnd concern with the validity of participatory and qualitative approaches in general

As I said at the beginning, this research was based on a literatue review as opposed to empirical research, so some of the next steps that can help further guide the improvement of evaluation appraoches in conservation projects include:

To do a meta-evaluation of evaluation across conservation projects looking at the degree to which these components are being incorporated in practice, where it has been successful, and what were the factors that contriuted to this success

To compare information gained, participant satisfaction and decisions taken based on the different evaluation frameworks used

To put together guidelines drawing on the successes and challenges of other fields experience in evaluation, such as education and social development to better incorporate these 3 components into evaluation