performance measurement in the conservation community status, progress, barriers, and next steps...
TRANSCRIPT
Performance Measurementin the Conservation Community
Status, progress, barriers, and next steps
Elizabeth O‘Neill, Conservation Auditor, WWF International
& Matt Muir
on behalf of the
Measuring Conservation Effectiveness Summit Research Committee
INTRODUCTION
Purpose: Survey of 29 conservation implementers and funders to ask about their Systematic Performance Measurement (SPM) practice.
Implementors:• “OECD-DAC principles for evaluation”
• “annual plans and performance reviews”
• “peer-review audits”
• “implementation of the Open Standards”
Funders:• “communication tool for our board”
• “rigorous post-grant monitoring”
• “constant and direct assessments of progress”
• “qualitative approach to evaluate grantee results”
Plan
Adapt Do
Check
INTRODUCTION
Respondents:
15 conservation implementers: • ~2.1 billion US$
• ~7,000 projects
• 16,000 staff
14 conservation funders: • ~1.1 billion US$
• ~1,000 grants
• 578 staff
1. SPM is widely acknowledged as important...
2. ...but is not done to a great extent across our community…
3. …however, we can improve by leveraging the progress we have made…
4. and overcoming several critical obstacles.
What we learned:
INTRODUCTION
SPM IS IMPORTANT IN CONSERVATION
1. SPM is widely acknowledged as important in conservation.
SPM is generally viewed favorably and as a priority by most managers and boards.
SPM IS IMPORTANT IN CONSERVATION
Strongly disagree
Strongly agree
Moderately disagree
Moderately agree
? ?
SPM IS IMPORTANT IN CONSERVATION
When SPM is implemented, we say the primary driver is to improve effectiveness.
Do we understand why projects fail?
Are actions cost-effective?
Are we having intended impacts?
Are we adapting and improving?
What can be learned to improve?
Can we demonstrate credible results?
85 to 90% of all respondents say answering these
questions is important
FUNDERS
Do we understand why projects fail?
Are actions cost-effective?
Are we having intended impacts?
Are we adapting and improving?
What can be learned to improve?
Can we demonstrate credible results?
% of respondents
SPM IS IMPORTANT IN CONSERVATION
Not at allVery wellMinimally
wellSomewhat
well
0 100
FUNDERS ORGS
Do we understand why projects fail?
Are actions cost-effective?
Are we having intended impacts?
Are we adapting and improving?
What can be learned to improve?
Can we demonstrate credible results?
SPM IS IMPORTANT IN CONSERVATION
Not at allVery wellMinimally
wellSomewhat
well
% of respondents0 100
FUNDERS ORGS $$$
Do we understand why projects fail?
Are actions cost-effective?
Are we having intended impacts?
Are we adapting and improving?
What can be learned to improve?
Can we demonstrate credible results?
SPM IS IMPORTANT IN CONSERVATION
Not at allVery wellMinimally
wellSomewhat
well
% of spending0 100
SPM IS NOT DONE WIDELY
2. That we often cannot answer key performance questions with confidence is likely because...
...SPM is not done to a great extent across our community.
For every conservation dollar spent, we say that only 10-30 cents worth is guided by SPM…
SPM IS NOT DONE WIDELY
Survey Question: Of the total budget for your organization’s conservation efforts, what % is guided by SPM?
…and only about 5% of projects go through a full SPM cycle (i.e., plan-do-check-adapt).
only ~350-500
only ~2500
Of ~7000 projects
have good conservation plans in place, but…
currently undertaken by implementing organizations,
have completed the SPM cycle
SPM IS NOT DONE WIDELY
By $$$
Practice tends to break down at certain places in the cycle
DESIGN
IMPLEMENT
MONITOR
EVALUATE& ADAPT
SHARE
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
KEY INGREDIENTS TO PROGRESS TO DATE
3. To improve our ability to measure effectiveness, we can leverage several ingredients key to our progress to date.
SPM:
The importance of having a mandate
Key ingredients to the progress we’ve made to date:
KEY INGREDIENTS TO PROGRESS TO DATE
(1) Institutional mandate
(2) Presence of an SPM champion
(3) A vision for what could be accomplished with SPM
(4) Evidence that SPM helps
(5) Dedicated SPM staff
(6) Dedicated SPM funding
CRITICAL OBSTACLES TO PROGRESS
4. To improve our ability to measure effectiveness, we also must overcome several critical obstacles.
CRITICAL OBSTACLES TO PROGRESS
OBSTACLE 1: LACK OF TIME TO DO SPM
• 90% of respondents indicate this is an extreme or major obstacle
CRITICAL OBSTACLES TO PROGRESS
OBSTACLE 2: LACK OF MONEY
Most implementers indicate they invest no more than 1-5% of total conservation spending in SPM.
By Org
% of annual conservation spending
OBSTACLE 3: LACK OF STAFF DEDICATED TO SPM
CRITICAL OBSTACLES TO PROGRESS
578foundation
staff
By Funders
19 dedicatedSPM staff
(30:1)
By Org
16,400organizational
staff 71 dedicatedSPM staff
(230:1)
CRITICAL OBSTACLES TO PROGRESS
OBSTACLE 4: FEELING THAT SPM IS TOO COMPLEX
Strongly disagree
Strongly agree
Moderately disagree
Moderately agree
OBSTACLE 5: LACK OF PRESSURE/DEMAND FROM ‘ABOVE‘
By Org By $$$
CRITICAL OBSTACLES TO PROGRESS
Smaller organizations seem to do SPM to a greater extent than larger ones.
Is SPM practiced well at your organization?
Key obstacles identified:
• Lack of donor pressure
• Lack of board pressure
• Lack of demand from upper management
Insufficient time + Weak funding + Few staff + Limited understanding + Lack of demand from ‘above’ =
An apparent low priority placed on SPM
Unanswered research questions:
Why is SPM a low priority? What are root causes?
What will it take to make major advances?
CRITICAL OBSTACLES TO PROGRESS
1. SPM is important in conservation
We say it is important and a top priority.
We want to be able to answer key questions regarding effectiveness.
But we generally cannot answer these questions.
2. SPM is not done widely
Only 10-30% of current conservation spending is guided by SPM.
Very few projects do more than basic design plus implementation.
Things break down at more rigorous design and monitoring, evaluation, adaptation, and sharing learning.
CONCLUSIONS
Performance Measurement in the Conservation Community
3. We can improve by leveraging key factors to progress to date...
Performance Measurement in the Conservation Community
4. ...and by overcoming several critical obstacles.
CONCLUSIONS
Time
Money
Dedicated SPM staff
Knowledge/understanding
Demand from ‘above’
Institutional mandate
Champions for SPM
A vision for what SPM can do
Evidence that SPM helps
THANK YOU!!
Frog illustrations © Ted Kahn 2010Original artwork used with permissionNeotropical Conservation Foundationwww.neotropicalconservancy.org
Thank you to all survey respondents!
Summit Research CommitteeSheila O'Connor (chair)Bernd CordesWilliam CrosseBrett JenksRichard MargoluisMatthew MuirElizabeth O'NeillNick SalafskyKristin Sherwood