skunk works evaluation tools: how do we know if we are having an impact?
TRANSCRIPT
![Page 1: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/1.jpg)
Skunk Works Evaluation Tools:
How do we know if we are having an impact?
![Page 2: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/2.jpg)
What are we trying to achieve?
Strategies
Mind Shaping
Process: Activities
to demonstra
te application
Capacity
Building
Accessible ES Tools
Improved State of HWB and Biodiversity (Ecosystem Services)
Outcomes
Use of ES information in decisions
![Page 3: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/3.jpg)
Operationalizing Outcomes
• Material living standards
• Physical health• Mental health• Social
connections• Spiritual
fulfillment• Cultural
identity• Security• Life
satisfaction• Equity
Use of ES and Policy
Decisions
Improved ES Improved HWB
• Increased policy documents with reference to ES as desired outcomes or factors in decision-making
• Increased discussion of ES in policy meetings
• Clean drinking water
• Reliable water flow in the dry season
• Reduced, in-stream sediment concentration
![Page 4: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/4.jpg)
What other factors influence our intended outcomes?
Individual experience and personalities
Social and institutional context
Political climate
Other initiatives in place
Resources
• Land-use change (outside of decision)
• Climate change • Extreme events
(e.g., fire or flood)
• Other sources not related to natural resources
• Background factors (physical, mental, social health)
• Biophysical context
Use of ES and Policy
Decisions
Improved ES Improved HWB
![Page 5: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/5.jpg)
So what to we need to consider?1) Stage of Evaluation
– At what stage are we in completing our strategies?
2) Design – How confident can we be that our intervention
caused the outcomes?
3) Data Collection Methods– Which methods are most valid to measure
expected outcomes?
![Page 6: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/6.jpg)
Evaluation Stages
Developmental• What is promoting innovation in
the project?
Formative• How well are our strategies
getting us to our goals?
Summative (Impact)• Did it work?
![Page 7: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/7.jpg)
Design
• Control and Experiment group/site receive same measurement before and after intervention
Quasi-experiment/
BACI• Control and experiment group/site
randomized and measured post-intervention
Posttest Only
• Pre and post measure the experimental group/site
One Group Pre-Posttest
• Control and Experimental group/site non-randomly assigned, measured post intervention
Natural Experiment
• Post intervention measurement onlyCase Study ✔
![Page 8: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/8.jpg)
Some Example Methods…
![Page 9: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/9.jpg)
To Test Mind Shaping:
e.g.• Stakeholder focus on ES• Mediation of different ideas• Understanding and Awareness of ES
Method: Mental Models
![Page 10: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/10.jpg)
Method 1: Mental Models
Tells us: To what extent ES are being incorporated into Structure and Content of Knowledge and preferences
• Accomplished during an interview (1hr)• All or sample of participants, individual or group• Content analysis and stakeholder comparison• Improved with pre-post design, control group,
and/or randomization• Necessary: control for the hypothesized
intervention
![Page 11: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/11.jpg)
To Test ES use in Decisions: e.g.• Policy documents that refer to ES as
desired outcomes or factors in decision-making
Method: Content Analysis
![Page 12: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/12.jpg)
Method 2: Content Analysis
Tells us:The frequency and parameters of ES used to make a decision, and the relative influence of ES to other factors • Desktop document review• Can combine with clarifying interviews
with document preparers• Either inductive or deductive analysis (a
rubric helps)
![Page 13: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/13.jpg)
To Test Increased HWB: e.g. • Physical Health• Life Satisfaction and Happiness• Political Voice and Governance
Method: Community-level Survey
![Page 14: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/14.jpg)
Method 3: Community-level Survey
Tells us:Baseline and change values for actual and perceived HWB attributes
• Usually best developed based on ideas from interviews or focus groups
• In-person, paper or Web-based• Distributed among the target population• Some indicators may already exist from
local health departments or other agencies
![Page 15: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/15.jpg)
A note on HWB measurements Qualitative and Quantitative
approaches necessary to hone in on appropriate indicators
![Page 16: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/16.jpg)
Other Methods
• Network analysis: measure increase in information exchange, collaboration, and social capital related to ES and/or HWB
• Focus groups/interviews: Just about anything
• Participant Observation: Participant interaction with the tools and which are most effective at what
• Observation Protocols: Adaptation of ES specific terminology in meetings or trainings
![Page 17: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/17.jpg)
Water quality
high
low
time
Conservation impact site
Control
Difference in mean values
improved water quality!
To Test Improved ES
e.g. improved water quality and dry-season flow
Methods: BACI and modeled controls
![Page 18: Skunk Works Evaluation Tools: How do we know if we are having an impact?](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649e545503460f94b4b583/html5/thumbnails/18.jpg)
Method 4: BACI and modeled controlsTells us: Water quality relative to before the conservation impact and to control areas • Experimental backbone: Before-After-Control-
Impact (BACI) (conservation impact)• Modeled controls and exploration of “minimum
monitoring”• Post-treatment/impact only• Experimental controls to validate NatCap
models: model “no action” as a comparison