The Story of Washington’s Murder(according to Bill Sutherland)12th Dec 1799 Inspects plantation in snow13th Sore throat14th Dec 2:00 AM Breathing difficulties • Mr Rawlins, Estate overseer, provided mixture molasses, vinegar• Mr Rawlins ordered by Washington to remove 0.75 pint blood 10:00 AM Dr James Craik, personal physician, arrives• Preparation dried beetles applied to throat• 1 pint blood removed• No improvement so another 1 pint blood removed• Vinegar in water gargle – near suffocation• 2 more pints blood removed3:00 PM Dr Elisha Dick, prominent physician arrives• 1.8 pints blood removed• Pulse low10:10pm Died

Total > 6.5 pints removed

Pierre Charles Alexandre Louis(1787-1872)

• Introduced numerical methods to examine the effectiveness of medical interventions

• Showed blood letting ineffective

How Practitioners Make Decisions Myth Anecdote Observation Comparison Replicated RRC

AcademicNow

Practitioners

Future Academic

Practitioners

Documented

Source: Sutherland 2007

Measuring Effectiveness

An Overture

Measuring Conservation Effectiveness SummitMay 5-6, Palo Alto, California

Research On Over 220 MeasuresSystems in Different Fields

3542

55

30

56

0

10

20

30

40

50

60

Business &Mngmt

Education &Soc Serv

InternationalDvlpmt

Public Health& Pop

Envt &Conservation

Ap

pro

ac

he

s R

ev

iew

ed

Business & ManagementFamily Tree

1500

1900

1970

1980

1990

2000

EFFECTIVENESSACCOUNTING & CERTIFICATION

Balanced Scorecard

Cost Accounting

Activity Based Costing

Double-EntryBookkeeping

Engineering Standards

Auditing by Accountants

Process Certification

Scientific Mngmt

Total Quality Mngmt

Managing for Results

Six Sigma

Process Reengineering

Performance Benchmarking

Learning Organization

Community of Practice

Outcome Evaluation

Operations Research

Reflective Practice

Social Learning

STATUS ASSESSMENT

Stock Market Index

Economic Production

Econ Welfare Indicators

Outcome Evaluation

Accounting Standards

BenchmarkingChaordic Systems

Q1. What Is Measuring Effectiveness?

1500

1900

1970

1980

1990

2000

EFFECTIVENESSACCOUNTING & CERTIFICATION

STATUS ASSESSMENT

Status Question:How are Species and Ecosystems Doing?

?

Effectiveness QuestionAre Our Actions Leading to Desired Results?

?

Certification Question:Do We Meet External Standards?

?

Why We Are Focused on Effectiveness

EFFECTIVENESSACCOUNTING & CERTIFICATION

STATUS ASSESSMENT

??The Focus

for This Summit

Q2. What is the Unit for Measuring Effectiveness?

Actions

Threats

Conservation TargetWildlife and Habitat

Direct

Indirect

Costs of measuring change

Level of

confid

en

ce

Tim

e t

o s

ee a

n im

pact

Outputs

Outcomes

Impacts

Q3. How Do You Measure Effectiveness?

1500

1900

1970

1980

1990

2000

EFFECTIVENESSACCOUNTING & CERTIFICATION

STATUS ASSESSMENT

External Summative Evaluations

Participatory Formative

Evaluations

Project Cycle Based

Monitoring

Need to Integrate Measuring Effectivenessinto an Iterative Project Cycle

ImplementManagement &

MonitoringPlans

D

Develop aMonitoring

Plan

C

Develop aManagement Plan:Goals, Objectives,

& Activities

B

Start

Clarify Group'sMission

Iterate

Use Results toAdapt & Learn

EDevelop

ConceptualModel Basedon Local SiteConditions

A

AnalyzeData and

CommunicateResults

E

The AMProjectCycle

Check

DoAdapt

Plan

The Basic Systematic Performance Management (SPM) Cycle

Biodiversity ConservationProjects Come In All Shapes and Sizes

1. A community fishing ground in the Pacific2. A TNC preserve or provincial park3. Coordinated efforts to manage state and federal

funding for Puget Sound or Lake Ontario4. The WWF Markets Initiative5. A funder’s grantmaking strategy for the California

Coastal waters, or the Coral Triangle

CMP AWF CI TNC WCS WWFBiodiversity Targets

Focal Targets

Conservation Outcomes

Focal Conservation Targets

Landscape Species

Long-Term Goals

Threats Threats Pressures Threats Threats Threats

Objectives Milestones Objectives Targets Project Targets

An Excerpt from the CMP’s Rosetta Stone Analysis

Similar Concepts, Different Terms

Conservation Measures Partnership’s Open Standards Are One Flavor of SPM

• Developed by leading orgs & agencies

• Draws on many fields• Open source &

common language• Used around the world

Q4. How Much Should We Invest inMeasuring Effectiveness?

SPM does not specifyM&E Design !!

The “Burden of Proof” Depends on Risks,Costs, and Timeframe for Decision Making

Generally Invest More in Measures When:• Stakes are high (high cost of error or inaction)• Potential to leverage learning• Costs of measures are low relative to actions

MarislaFoundation

PackardFoundation

Q5. Should We Mandate Measuring Effectiveness?The Bell Curve: Treating Cystic Fibrosis

A Surprise – The Best Get BetterIt’s the centers in the top quartile that are improving fastest….they are at risk of

breaking away. What the best may have, above all, is a capacity to learn and adapt – and to do so faster than everyone else.

The Bell Curve Leads to Uncomfortable Questions Will being in the bottom half be used against doctors in lawsuits? Will we be

expected to tell our patients how we score? Will our patients leave us? Will those at the bottom be paid less than those at the top? The answer to all these questions is likely yes.

Poor

BelowAvg

AboveAvg

Best

Distribution of Cystic FibrosisTreatment Center Success

Rates

Ultimately We Are Not Competing Within Conservation, But Against Other Social Causes

ConservationWorthy Cause A Worthy Cause B

Q7. How Do We Roll Out Measuring Effectiveness Both Internally….

After a guy gave an ‘air ball’Six Sigma presentation…the standard joke

was that the guy “decided to leave” before his elevator reached the ground floor.

Q7. How Do We Roll Out Measuring Effectiveness Both Internally….and Externally?

If We Create a Culture of Data Sharing…

?

Q8. How Do We Continue this Conversation Going Forward?

Our Fundamental Hypothesis

-

0 %

100 %

- Failure

+ Success

+Re

so

urc

es f

or

Co

ns P

roje

cts

Our Fundamental Hypothesis

--

0 %

100 %

- Failure

+ Success

++

Re

so

urc

es f

or

Co

ns P

roje

cts

Investment inSPM

Measuring Effectiveness Can Transformthe Practice of Conservation

• Practitioners in field using measures results to plan and implement effective projects

• Managers and donors getting better information to improve work

• All stakeholders learning from each other• Increased public support for conservation• Ultimately, better conservation outcomes