how evaluation helps you design and implement programs ... · program design & evaluation –...
TRANSCRIPT
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
How Evaluation Helps You
Design and Implement
Programs That Work for Your Audience
Carolyn J. Cumpsty Fowler, Ph.D., M.P.H.
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
do you do what you do?
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
The Public Health Approach … again
1. Surveillance & systematic data collection
Assess the problem.
2. Identify risk and protective factors
Identify causes.
3. Develop & evaluate interventions
Develop programs and policies that work.
4. Implementation. Scale up effective policies & programs.
Implement & Disseminate
Original graphic from http://www.who.int/violenceprevention/approach/public_health/en/index.html - adapted by Carolyn Cumpsty Fowler 2012
Why Do So Many Programs Fail?
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
The Public Health Approach … again
1. Surveillance & systematic data collection
Assess the problem.
2. Identify risk and protective factors
Identify causes.
3. Develop & evaluate interventions
Develop programs and policies that work.
4. Implementation. Scale up effective policies & programs.
Implement & Disseminate
Original graphic from http://www.who.int/violenceprevention/approach/public_health/en/index.html - adapted by Carolyn Cumpsty Fowler 2012
Modifiable barriers to implementation
of effective programs:
Overly broad problem definition
Incomplete diagnosis
Unrealistic goals
Poorly defined objectives
Inadequate implementation planning
Working in a vacuum
Turf wars
Planning gaps
Cruise control and tunnel vision
Absent or inadequate evaluation
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
Which is the most critical modifiable
barrier to implementation of effective
programs?
Overly broad problem definition
Incomplete diagnosis
Unrealistic goals
Poorly defined objectives
Inadequate implementation planning
Working in a vacuum
Turf wars
Planning gaps
Cruise control and tunnel vision
Absent or inadequate evaluation
Which is the most critical modifiable
barrier to implementation of effective
programs?
Overly broad problem definition
Incomplete diagnosis
Unrealistic goals
Poorly defined objectives
Inadequate implementation planning
Working in a vacuum
Turf wars
Planning gaps
Cruise control and tunnel vision
Absent or inadequate evaluation
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
When is evaluation inadequate?
What is
expected to
change
Outcome
This graphic is adapted from one originally developed with Ronda Zakocs, PhD, MPH
Idea You invest a lot of time, energy and resources
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
What is
expected to
change
Outcome
Outcome
Evaluation
This graphic is adapted from one originally developed with Ronda Zakocs, PhD, MPH
Idea You invest a lot of time, energy and resources
What is
expected to
change
Outcome
Outcome
Evaluation
This graphic is adapted from one originally developed with Ronda Zakocs, PhD, MPH
Inadequate Evaluation: Too Little & Too Late
At this point, we
have very little
control over the
outcome
You invest a lot of time, energy and resources
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
What is
expected to
change
Program Planning Implementation Outcome
Outcome
Evaluation
This graphic is adapted from one originally developed with Ronda Zakocs, PhD, MPH
Inadequate Evaluation: Too Little & Too Late
At this point, we
have very little
control over the
outcome
?...?...?
Assess
Plan
Develop or modify
activities
Implement
activities
What is
expected to
change
Program Planning Implementation Outcome
Outcome
Evaluation
This graphic is adapted from one originally developed with Ronda Zakocs, PhD, MPH
Inadequate Evaluation: Too Little & Too Late
At this point, we
have very little
control over the
outcome
Problems Always Happen
Evaluation used during planning and implementation increases your
ability to influence the outcome
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
Assess
Plan
Develop or modify
activities
Implement
activities
What is
expected to
change
Program Planning Implementation Outcome
Formative Evaluation Process Evaluation Outcome
Evaluation
This graphic is adapted from one originally developed with Ronda Zakocs, PhD, MPH
Inadequate Evaluation: Too Little & Too Late
At this point, we
have very little
control over the
outcome
Problems Always Happen
Evaluation used during planning and implementation increases your
ability to influence the outcome
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
Panel on the Understanding and Control of Violence
Understanding and Preventing Violence, 1993
“The only way to determine if
something works is to try it, in a
way that lends itself to reliable
evaluation”
Evaluative Thinking
Assess the
Problem Identify the
“Causes”
Design & Evaluate
Programs &
Policies
Implement &
Disseminate
Programs & Policies
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
Mission Critical – Informed Beginnings
If the first button of one’s coat is wrongly
buttoned, all the rest will be crooked.
Giordano Bruno (1548-1600)
Assess the
Problem Identify the
“Causes”
Design & Evaluate
Programs &
Policies
Implement &
Disseminate
Programs & Policies
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
A [person] should look for what is, and not for
what [s]he thinks should be.
Albert Einstein
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
The Golden Rule of Evaluative Thinking:
ALWAYS Question Assumptions
Cartoon courtesy of i-heart-god.com
The Evaluative Thinking Process as Coach
“Most of the value in a logic model is in
the process of creating, validating and
modifying the model … The clarity of
thinking that occurs from building the
model is critical to the overall success
of the program.”
W. K. Kellogg Foundation Evaluation Handbook 1998; p. 43.
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
The Public Health Approach …with evaluation questions
1. Surveillance & systematic data collection
Have we asked the right questions? Have we defined the real problem? Have we checked?
2. Identify risk and protective factors
Have we identified modifiable critical factors: causal, predisposing, enabling & reinforcing?
3. Develop & evaluate interventions
What works and for whom? Who should deliver? What are the costs, risks, benefits, & potential to sustain?
4. Implementation. Scale up effective policies & programs.
How do we take effective & promising interventions to scale, & demonstrate impact?
Original graphic from http://www.who.int/violenceprevention/approach/public_health/en/index.html - adapted by Carolyn Cumpsty Fowler 2012
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
Are there other
important
questions that
evaluative
thinkers can
identify?
Always consider the intervention context
Factors that will support or inhibit our program
– resource issues
– physical environmental issues
– social environmental factors (social ecological)
Can you achieve this program; if yes, should it?
Where
we are
Where we
hope to be Change process
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
We can’t afford NOT to do evaluation!
(if we have limited resources)
Evaluation may help you discover
opportunities that help you implement
programs that work for your audiences
Program
Delivery
Injury
Prevention
Impacts
Problem
Analysis
Problem
Identification
Program
Design
Social and Political Contextual Influences
Capacity &
Infrastructure
Epidemiology & Assessment, Program Design, Implementation, Evaluation, Information-
Sharing, Human & Other Resource Issues, Funding, Partnerships, Socio-Political Influence
-Internal
- External
Graphic developed by Carolyn Cumpsty Fowler
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
Expanded Program Cycle Model developed by Carolyn Cumpsty Fowler PhD, MPH with input from Dr. Ronda Zakocs (2004)
The Expanded Program Cycle:
The set of activities involved in initiating,
planning, implementing, evaluating, and
improving programs.
Problem ID, Definition,
& Measurement
Monitoring of
Program Quality
Integrity
of Diagnosis
Reactive, Adaptive, Proactive
Program Responses
Intervention
Selection Strategy
Dissolution or
Institutionalization
Plan Quality
Resource
Mobilization
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
The Expanded Program Cycle can be viewed as a series of stages – each
with a desired outcome/product
Critical, evaluative thinking used throughout each step of the program
cycle, can serve as a program coach and build program capacity.
Resource
Mobilization
Problem ID, Definition,
& Measurement
Monitoring of
Program Quality
Integrity
of Diagnosis
Reactive, Adaptive, Proactive
Program Responses
Intervention
Selection Strategy
Dissolution or
Institutionalization
Plan Quality
Expanded Program Cycle Model developed by Carolyn Cumpsty Fowler PhD, MPH with input from Dr. Ronda Zakocs (2004)
E-valu-ation
Extremely Valuable Information
can provide
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
“I have not failed.
I have found ten thousand
ways that don’t work”
Thomas Edison
1847-1931
A multi-layered evaluation strategy is
needed to shed light on the change
process
Once we really believe this, we are more likely to prioritize and do evaluation
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
Many people undervalue evaluation
.. often because they don’t know what they’re missing
Where there’s a will, there’s a way
People who really want to do evaluation,
find ways to do it.
Evaluation may pay its own way
– $ invested in evaluation may save $$$ later
on in the program
– The earlier we start, the more we can save.
Evaluation can save your life – literally
and figuratively
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
Evaluation lessons from the fire department
“Size-Up”
The dangers of unfamiliar territory
The dangers of very familiar territory
The critical importance of sharing lessons learned Baltimore County Fire Department in action (2009) Photograph by Pete Hammond.
“A journey of a
thousand miles
begins with a single
step”
Lao-tzu (604 BC-531 BC)
Injury prevention
is an urgent
challenge but ..
don’t be
tempted to rush
into action
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
A journey of a thousand miles begins with a single step Lao-tzu 604 BC-531 BC
A new translation:
“A journey of a thousand miles begins at the spot under
one's feet, therefore deal with things before they happen;
create order before there is confusion”
As you embark on your injury
prevention program planning journey,
remember that evaluation is your
friend …and guide.
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
For Everything You Do
To Keep Kids Safe!
Carolyn Cumpsty Fowler, PhD, MPH
443-287-0541
Johns Hopkins Center for Injury Research and Policy: www.jhsph.edu/InjuryCenter
Valuable free web resources:
http://ctb.ku.edu
http://www.uwex.edu/ces/pdande/index.html
Program Design & Evaluation – Part 2 SAFE KIDS Conference
June 20, 2013
Carolyn Cumpsty Fowler, PhD, MPH
More excellent resources:
North Carolina Department of Health & Human Services (2011) Community Health Assessment Guide Book** http://publichealth.nc.gov/lhd/cha/docs/CHA-GuideBookUpdatedDecember15-2011.pdf
W. K. Kellogg Foundation www.wkkf.org
Evaluation handbook**
Logic Model Development Guide**
** These texts are free and available (as a pdf file) online
Coley, S.M., Scheinberg, C.A. (2013). Proposal writing:
Effective Grantsmanship (4th ed.) Los Angeles, CA: SAGE
Issel, L. M. (2013). Health Program Planning & Evaluation
(3rd ed.) Sudbury, MA: Jones and Bartlett