theme 3 - iisd · module 12: adaptive management and evaluation stakeholder feedback • each taw...
TRANSCRIPT
Theme 3 Adaptive Thinking and
Committing to Action to
Prepare for the Future
Module 12: Adaptive Management and Evaluation
Module 12Adaptive Management
and Evaluation
Module 12: Adaptive Management and Evaluation
Learning objectives
After completing this module, you will be able to:
1. Describe the principles of adaptive management and
summarize how watershed management can respond
to anticipated and unanticipated uncertainties.
2. Explain the role of monitoring and evaluation (M&E) and
key performance indicators and describe how to
implement appropriate M&E systems for adaptive
watersheds.
Module 12: Adaptive Management and Evaluation
Module 12: Adaptive Management and Evaluation
Key questions
• What is adaptive management? Why is it necessary? What are the key
elements of adaptive management?
• How is adaptive management implemented?
• How can we develop key performance indicators (KPIs), including long term
and near term?
• How can we develop effective M&E systems?
Module 12: Adaptive Management and Evaluation
Rationale for adaptive systemsBased on Holling, 1978
Brought together principles of ecology, sociology and
economics in the context of environmental management
and policies.
• Our knowledge of a system is often surpassed by our
ignorance of aspects of it; plan for uncertainty.
• Humans have dealt with uncertainty through trial and
error.
• Ecological systems demonstrate a resilience that has a
counterpart in institutional and other systems.
• Complex systems (including watersheds) comprise
multiple variables that shift and move. Management
policies often try and minimize this variability, but must
instead understand and adapt with them.
Module 12: Adaptive Management and Evaluation
Adaptive management
• Monitor and review impacts to understand what’s working
• Revise actionsfor maximum impact
• Implementing key actions based on goals
• Establishing goals and responsibilities Plan Do
CheckCorrect
Module 12: Adaptive Management and Evaluation
Cycle of strategic adaptive management
Module 12: Adaptive Management and Evaluation
Adaptive ecosystem-based management in watersheds
Source: Adapted from Roy, Barr and Venema, 2011
Module 12: Adaptive Management and Evaluation
Visioning your watershed of the futureAdaptive watershed management brings together ecological, social, hydrologic,
climatic, economic and other priorities to develop management systems.
Fisheries
IrrigationAgriculture
Sanitation
and Waste
Drought
Local
bioenergy
Recreation
Source: IISD, 2010
Module 12: Adaptive Management and Evaluation
Piloting portfolioof ecosystem initiatives• Implementation stage of strategic adaptive ecosystem management
• Manager pilots ecosystem initiatives aimed at achieving a shared long-term
outcome
Module 12: Adaptive Management and Evaluation
Need for formal review (after Holling 1978)
1. Variability of ecological systems provides a self-
monitoring system that maintains resilience.
2. Regular review, even when the program is performing
well, and the use of well-designed pilots throughout the
life of the program to test assumptions related to
performance, can help address emerging issues and
trigger important adjustments for better impact.
3. Monitoring the wrong variable can falsely indicate no
change even when drastic change is imminent.
4. Impacts may not be gradual; they may appear abruptly
due to climate, human, or other drivers or pressures.
5. Existing impact assessment methods (e.g., cost-benefit
analysis, input output, cross-impact matrices, linear
models, discounting) do not recognize importance of
these interactions.
Module 12: Adaptive Management and Evaluation
Formalize review systems
• Specified time period identified in planning stages
• Annual to 5-year reviews usually recommended
• Monitoring key performance indicators (KPIs) may
indicate shorter time interval needed to respond to
changing conditions
• When performance of an initiative is sensitive to a
specific input parameter or if impacts are potentially
serious but uncertain, KPIs can indicate key times for
additional review.
• Stakeholder feedback, including new scientific
information
Module 12: Adaptive Management and Evaluation
Key performance indicators
(KPIs)• KPIs are the primary tool during this stage
• Different levels and time scales: Process; Outcomes, near-term/long-term;
Impact on ecosystem, community and resilience KPIs
Module 12: Adaptive Management and Evaluation
Anatomy of a KPI Anatomy of a KPI
Title and axis labelunits
source
data legend
Target
Source: UNEP/DEWA/GRID-EUROPE, GEO Data portal: Compiles from Forest Stewardship Council (FSC) data)
Module 12: Adaptive Management and Evaluation
Key characteristics of a KPI
SMART indicators
• Specific: Target a specific area of improvement
• Measurable: Quantify/qualify progress
• Achievable: Results must be possible in time and resources available
• Responsible: Specify who will actually do it
• Time-bound: Specify when the results can be achieved.
Module 12: Adaptive Management and Evaluation
Near-term KPIs
• Activity-based outputs to measure completion of tasks and their results
• Early gauge of whether hypothesis upon which the initiative is based was
correct
• Can the actions deliver a positive ecosystem benefit?
Outputs
(knowledge
generated or
services
delivered)
Ecosystem initiative results show the impact of water retention
on fish population and water availability
KPI: % increase in downstream fish population
Target: 20%
Activities
(ecosystem
management
and adaptation
projects)
Development of water retention structures to improve fish
habitat and mitigate flood damage in the region.
KPI: Progress in developing small dams
Target: Completed on schedule and budget
Module 12: Adaptive Management and Evaluation
Long-term KPIs
• Measure performance of actions based on impacts, priorities and
effectiveness as agreed-upon by stakeholders.
Results Chain
Ecosystem Initiative #1
Water retention pilot initiative
Long-term outcomes (change in state of society, economy, environment)
Improved water access for irrigation during dry seasons, flood damage mitigation and improved fisheriesKPI: Flood damage mitigated downstream; total fish populationTarget: 20% increase in fish population
Intermediate outcomes (increased capacity, awareness, access,etc.)
Awareness among regional land use managers, watershed managers and communities about the relationship between water storage, floods/drought and community resilience.KPI: # of planners attending planning sessionsTarget: Specific # of influential roles/people identified in the impact strategy)
Module 12: Adaptive Management and Evaluation
Selecting your KPIs
• Aim to measure your strategic objectives
• Aim for a manageable number of KPIs
• Choose SMART KPIs, including a mix of near-term,
long-term, process, ecological, social, etc.
• Look for recurrent and consistent patterns
• Select KPIs based on available data, and also inputs
from key stakeholders to ensure the most legitimate,
evidence-oriented and culturally appropriate
measurement.
• Ensure inclusion of gender-responsive KPIs, as well as
those representing the needs of vulnerable groups
Module 12: Adaptive Management and Evaluation
Long-term KPIs
Results Chain
Ecosystem Initiative #1
Spillway Pilot Initiative
Ecosystem Initiative #2
Fish Ladder Pilot Initiative
Ultimate Outcomes
(change in state of
environment, society,
economy)
Restoration of salmon population and hydropower that can meet demand
KPI: Total salmon population
Target: 20% above baseline counts within 5 years
Intermediate
Outcomes
(new/improved policy
or practice)
More frequent spillway operation
KPI: Total spillway operation time
Target: X hours more per month
Permanent increase in fish ladder capacity
KPI: salmon count immediately downstream of ladder
Target: X% of upstream count
Intermediate
Outcomes
(increased
awareness, capacity
or access)
Awareness among hydropower policy-makers that
increased spillway operation is a feasible means to
increase salmon population
KPI: # of hydropower planners and policy-makers
attending presentation on results of spillway experiments
Target: (this target should include the specific names of
influential persons identified in the impact strategy)
Awareness among hydropower policy-makers that
improved fish ladder technology can increase salmon
population
KPI: # of hydropower planners and policy-makers attending
presentation on results of fish ladder experiments
Target: (this target should include the specific names of
influential persons identified in the impact strategy)
Outputs
(knowledge
generated or
services delivered)
Ecosystem initiative results showing the impact of
spillway operation on salmon population
KPI: % increase in downstream salmon population
Target: 20%
Ecosystem initiative results showing the impact of fish
ladder operation on salmon population
KPI: % increase in downstream salmon population
Target: 20%
Activities
(ecosystem
management
projects)
Ecosystem initiative to test the impact of increased
spillway operation on salmon population (including
salmon population and stream flow monitoring)
KPI: Progress toward completion of ecosystem experiment
Target: Completed on schedule
Ecosystem initiative to test the impact of improved fish
ladder design on salmon population (including salmon
population and stream flow monitoring)
KPI: Progress toward completion of ecosystem experiment
Target: Completed on schedule
Module 12: Adaptive Management and Evaluation
Activity: KPIs - Part I (20 min)
Based on your draft watershed plan and priorities, develop 10 illustrative KPIs,
including actions, and social, economic and environmental impact indicators.
Once you have 10, test these against the SMART criteria
KPIs
# of small check dams built (#) (Act.) Outputs/actions
Fish population increased (#) (Env.) Short/long-term outcomes
Fisheries-based economic growth ($/year) (Econ.) Short/long-term outcomes
Volume of flood-water retained (m3) (Env.) Short-term outcomes
# of irrigable acres increased (Econ.) Short/long-term outcomes
# of people with better access to water
(disaggregate by gender and community) (Soc.)
Long-term outcome
Improved resilience to floods and drought (Soc.) Ultimate outcome
Module 12: Adaptive Management and Evaluation
Activity: KPIs - Part II
KPIs Type SMART?
# of small check dams built (#) (Act.) Outputs/action
s
Fish population increased (#) (Env.) Short/long-
term outcomes
Fisheries-based economic growth ($/year) (Econ.) Short/long-
term outcomes
Volume of flood-water retained (m3) (Env.) Short-term
outcomes
# of irrigable acres increased (Econ.) Short/long-
term outcomes
# of people with better access to water
(disaggregate by gender and community) (Soc.)
Long-term
outcome
Improved resilience to floods and drought (Soc.) Ultimate
outcome
Module 12: Adaptive Management and Evaluation
Monitoring, evaluation and reporting• Recognize the role of monitoring and reporting in the development and
implementation of KPIs, and the adaptive watershed.
Module 12: Adaptive Management and Evaluation
Monitoring, evaluation and reporting• Monitoring: Systematic process of observation. Follows
a course of activities, comparing what happens with
what was expected. Observes delivery of ecosystem
services and that progress is made towards meeting
societal goals
• Evaluation: Assesses achievement against preset
criteria. Evaluation of an EM plan determines extent to
which ecosystem service levels meet intended
purposes; identifies lessons learned for subsequent
phases of EM
• Reporting: The process of regular sharing of
information with project stakeholders and the public to
ensure participation, transparency, accountability and
innovation.
Module 12: Adaptive Management and Evaluation
Developing effective monitoring and evaluation systems• Begin by reviewing the high-level KPIs appropriate for
your adaptive watershed plan.
• Identify primary users of evaluation, including those
most affected by actions, people whose perception will
control whether or not evaluation is used to guide
evolution of our EM plan
• Identify existing monitoring systems available in your
region.
• Compile ecological, social and economic measures that
could help with the completion of your KPIs
• Prioritize component parameters to understand how
systems are functioning, and how they respond to
development, climate and other pressures.
Module 12: Adaptive Management and Evaluation
Climate-hydrologic monitoring systems
Module 12: Adaptive Management and Evaluation
A role for citizen science
• Citizen science includes the collection and
analysis of data by members of the general
public, typically as part of a collaborative project
with professional scientists.
• Citizen Science is gaining popularity in different
parts of the world, particularly in the field of
weather and water-related monitoring.
Module 12: Adaptive Management and Evaluation
Integrated evaluation
• Along with information and data on climate, land
and water systems, incorporate key information
and data related to KPIs selected for the health of
the watershed.
• E.g., for community water access, you need to
think about access data (e.g. time travelled to get
water, or hours of running water/day)
Module 12: Adaptive Management and Evaluation
Integrated evaluation
Integrated Evaluation Systems
Evaluation Summary and Reporting
Summarizing available
data/info
First-hand monitoring
systems
Citizen science efforts
with formal protocols
Monitoring protocols for all desired elements (the what, when, how, and who
of monitoring systems)
KPIs selected for process and outcomes
Stakeholder priorities and requirements
Module 12: Adaptive Management and Evaluation
Stakeholder feedback
• Each TAW initiative should have an expert team that reviews feedback and
develops ways to respond.
• Evaluation should incorporate sound data, but also views of key
stakeholders to ensure that both qualitative and quantitative information are
captured.
• Aggregated stakeholder feedback can tell an important story about an
emerging issue or an unintended consequence of an initiative.
Module 12: Adaptive Management and Evaluation
Activity: monitoring and evaluation (20 min)
1. Looking back at the illustrative KPIs you developed for your watershed, can
you link them clearly to broader adaptive watershed priorities?
2. Are all the primary users represented in your evaluation? Where are the
gaps? (e.g., women’s use of local fisheries not included)
3. Identify key/potential sources of data for these KPIs; specify quantitative
(incl. analytical) and qualitative (deliberative) approaches
4. Set priorities, areas and frequency of monitoring.
Module 12: Adaptive Management and Evaluation
Final discussion points
1. Your monitoring plan is a key tool to evaluate your
watershed adaptive plan and change course as
necessary.
2. Ensure that your KPIs and monitoring take into account
climate-related vulnerabilities as well as women and
Indigenous community needs.
3. Your KPIs and monitoring systems should be adaptive
and can change with changing priorities.
4. Consider how evaluations will be reported to different
stakeholders. There is a range of reporting mechanisms
including data portal, websites, published “report cards”
and meetings.