nccmt spotlight webinar: guide to policy-influence evaluation
TRANSCRIPT
Follow us @nccmt Suivez-nous @ccnmo
Funded by the Public Health Agency of Canada | Affiliated with McMaster UniversityProduction of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The
views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..
Guide to Policy-Influence Evaluation
Presenters:
Marla Steinberg, PhD CE
Anima Anand, PhD
Michele Hopkins, RSW, MSW
Shannon Bradley Dexter, MSc
January 25, 2017 1:00 – 2:30 PM ET
Follow us @nccmt Suivez-nous @ccnmo2
Housekeeping
Use Chat to post comments and/or
questions during the webinar
• ‘Send’ questions to All (not
privately to ‘Host’)
Connection issues
• Recommend using a wired
Internet connection (vs.
wireless),
• WebEx 24/7 help line
• 1-866-229-3239
Participant Side
Panel in WebEx
Chat
Follow us @nccmt Suivez-nous @ccnmo3
After Today
The PowerPoint presentation (in English and French)
and English audio recording will be made available.
These resources are available at:
PowerPoint: http://www.slideshare.net/NCCMT/
Audio Recording:
https://www.youtube.com/user/nccmt/videos
Follow us @nccmt Suivez-nous @ccnmo4
How many people are watching
today’s session with you?
Poll Question #1
A. Just me
B. 1-3
C. 4-5
D. 6-10
E. >10
Follow us @nccmt Suivez-nous @ccnmo
Your profession? Put a √ on your answer (or RSVP via email)
/
Epidemiologist Management (director,
supervisor, etc.)
Allied health
professionals (nurse,
dietician, dental
hygenist, etc.)
Librarian Physician / Dentist Other
5
Follow us @nccmt Suivez-nous @ccnmo
Guide to Policy-Influence
Evaluation
http://www.nccmt.ca/resources/search/241
Episode 30
6
NCC
Infectious
DiseasesWinnipeg, MB
NCC
Methods
and ToolsHamilton, ON
NCC Healthy
Public PolicyMontreal, QC
NCC
Determinants
of HealthAntigonish, NS
NCC
Aboriginal
HealthPrince George, BC
NCC
Environmental
HealthVancouver, BC
7
Registry of Methods and Tools
Online Learning
Opportunities
WorkshopsMultimedia
Public Health+
Networking and
Outreach
NCCMT Products and Services
8
Follow us @nccmt Suivez-nous @ccnmo9
Poll Question #2
How familiar are you with the
method or tool we are discussing
today?
A. I am not familiar with the method or tool
B. I have heard of the method or tool
C. I have used the method or tool
Follow us @nccmt Suivez-nous @ccnmo10
Presenter
Shannon Bradley Dexter, MSc
Senior Policy Analyst,
Innovation Strategy, Public
Health Agency of Canada
Follow us @nccmt Suivez-nous @ccnmo11
Presenter
Marla Steinberg, PhD CE
Evaluation Consultant
Follow us @nccmt Suivez-nous @ccnmo12
Presenter
Anima Anand, PhD
Project Lead, Healthy
Weights for Children, The
Bridge Youth & Family
Services
Follow us @nccmt Suivez-nous @ccnmo13
Presenter
Michele Hopkins, RSW, MSW
Project Coordinator, Healthy
Weights for Children, The
Bridge Youth & Family
Services
Evaluating Policy-Influence
NCCMT Webinar
January 25, 2017
Marla Steinberg, Ph.D. CEEvaluation Consultant
Anima Anand, Ph.D. and Michele Hopkins M.S.W.Healthy Weights for Children Project
Shannon Bradley DexterInnovation StrategyPublic Health Agency of Canada
Welcome
By the end of this webinar, you should be able to:1. List the complexities of evaluating policy-influence.2. Assess how the Guide to Policy-Influence Evaluation can support your work.
Establishing our Learning CommunityPoll: What is your interest in policy-influence?
A. I am a practitioner who does policy-influence work
B. I am a public health manager or decision-maker with a mandate for policy-influence work
C. I am a funder who funds policy-influence work
D. I am an evaluator who evaluates policy-influence work
E. I am a student interested in policy-influence work
F. Other
G. I don’t really know what policy-influence work is……………..
Poll Question #3
Have you ever…..
• Wondered how you get senior decision makers and politicians to take an interest in your program or issue?
• Wondered how best to support the scale-up of an intervention?
• Wondered how to evaluate your influence on policy?
• Wondered what should be considered a realistic achievement for policy-influence work?
The Guide to Policy-Influence Evaluation may be just what you need….
Needs, definitions, and the complexity of policy-influence work and evaluation
Setting the context for the guide
Origins of the Guide: The Public Health Agency of Canada’s Innovation Strategy (IS)
• Funds population health intervention research in diverse communities across Canada to build evidence about "what works" "for who" and "in what context" to reduce health inequities.
• Promotes and shares intervention results to inform future policy, program design, and other actions that will improve population health.
• Funding provided in three phases.
Current focus areas: Mental Health Promotion and Achieving Healthier Weights
IS Evaluation Requirements
• IS projects are required to undertake a comprehensive evaluation including assessing the uptake of evidence or knowledge generated through the project.
• Uptake is often achieved through the “influence on policy.”
THE ISSUE: Projects needed guidance, tools, and resources to effectively report on how IS projects were influencing policy. They
wanted to be able to tell their policy-influence stories!
Exploring the world of policy-influence work
Definitions
What is policy?
• Formal and wordy definition:
Policy is a means of governing action with the aim of attenuating or promoting particular phenomena occurring in the population.
Policies can outline rules, provide principles that guide action, set roles and responsibilities, reflect values and principles, as well as state intentions.
Policies can be enacted by all levels of government (federal, provincial, regional and municipal), community organizations, businesses, and schools.
Polices can guide programs, practice or education.
• Adapted from the National Collaborating Centre for Healthy Public Policy (2010) Accessed at: http://www.ncchpp.ca/docs/MethodPP_EN.pdf
• Endorsed by 95% of IS project respondents in the policy evaluation needs assessment conducted in the fall of 2013.
What is policy-influence work?
Policy-influence work supports the uptake or spread of evidence-based
interventions.
An example
“With partner organizations, we developed a food security questionnaire that went to all party leaders during the recent provincial election to see where they stand on food security issues. The next step will be holding them to some promises. We have also been working with the municipality to develop a food charter and food strategy. Prior to this we prepared a fact sheet on food security as a municipal issue, which was distributed to supporters and encouraged people to write in support of including food security in the Regional Plan”
Policy maker education
Policy development
Public awareness campaign
Advocacy
Other terms……….
What is unique about policy-influence work and its evaluation?
Policy-Influence is Complex
“Policy-influence is a highly complex process shaped by a multitude of
interacting forces and actors. Outright success, in terms of
achieving specific, hoped-for changes in policy, is rare, and the work that
does influence policy is often unique and rarely repeated or replicated,
with many incentives working against the sharing of ‘good practice’”
(Jones, 2011, p. 1)
A lot of policy-influence is about being there when the
window opens and being nimble to jump on board. A
lot of policy work is stand and wait and then jump when you need to act”
Bonnie Leadbeater – WITS (Mental Health IS project)
Policy-making and policy-influence are processes
All roads lead to Rome…..
There are a wide variety of activities that can influence policy
There are a variety of policy impacts:
(1) Conceptual – changing the thinking of key stakeholders
(2) Instrumental – changing actions
Carol Weiss
Jones (2011) offers further elaborations on therange of policy impacts:1. Framing debates and getting issues on to the political
agenda; drawing attention to new issues and affecting the awareness, attitudes or perceptions of key stakeholders
2. Encouraging discursive commitments; affecting language and rhetoric to promote the recognition of specific groups or endorsements of policy recommendations
3. Securing procedural change; changes in the process whereby policy decisions are made, such as opening new spaces for policy dialogue
4. Affecting policy content5. Influencing behaviour change in key actors: policy
change requires changes in behavior and implementation at various levels in order to be meaningful and sustainable
http://tclocal.org/food/Attributing Change is Difficult
Theories of change are useful (required!)
There are a variety of methods for evaluating policy-influence
Working Group of IS
Projects Created
Needs Assessment Completed
Guide Developed
Case Studies Added
Final Production
and Translation
The development of the guide
Recommends a limited number of useful (and free) resources that are organized around a four-step evaluation planning process
Four steps are aligned with other evaluation planning frameworks
CDC Framework for Evaluation in Public
Health
Includes three case studies:
WITS - anti-bullying program for primary schools from the mental health stream
Healthy Weights Connections -System change intervention to improve public health services for aboriginal children and families
Our Food - Creating a food strategy for Halifax
Includes completed evaluation plans for the cases developed through using the resources
User Experience Insights
Set of “raw” ingredients
http://www.otisfundraisingideas.com/product/our-ingredients
To be selected, prepared, adapted, and consumed as needed to suit your context.
What it is not……
A step-by-step recipe to be rigidly followed
What it is not……
A step-by-step recipe to be rigidly followed
A resource on how to evaluate the implementation, effectiveness or impact
Who is the audience
Funders
Projects or organizations involved in policy-influence work
Evaluators
Assumes a basic understanding of evaluation
Uses for the guide
Read it to understand the complexities of and options for policy-influence work and its evaluation
Uses for the guide
Read it to understand the complexities of policy-influence evaluation
Use some of the frameworks to develop evaluation requirements for funded projects
Uses for the guide
Read it to understand the complexities of policy-influence evaluation
Use some of the frameworks to develop project-specific evaluation requirements
Use the tools to work with projects to plan their policy-influence work and evaluation
Healthy Weights for Children Project
created an innovative family education model
‘Healthy Together’© (HT)
• Integrated healthy eating and physical activity within group learning to bring families together to learn to make healthier choices and build healthy relationships.
• Established a collaborative national network to guide program integration within core practice and policy.
• 21 sites in 8 Provinces and Territories (BC, AB, SK, MB, ON, NL, NB, NWT)
• 150+ trained professionals across Canada• 1,000+ children, youth and caregivers • Community context:
• Northern, rural, urban, Aboriginal, multi-cultural• Low income, immigrant/refugee, foster families
Snapshot of HT
Program effectiveness
• HT demonstrated that inclusion of healthy cooking/eating and physical activity within group learning was key to participant engagement, leading to positive behaviour change.
• Reduced consumption of sugar sweetened beverages• Increased physical activity• Reduced screen time
• Relationships that developed during HT sessions led to building social support for families, particularly in underserved communities.
HT ‘policy influence’ needs
• An effective policy development framework • Concerted effort by multi-sectorial partners to promote
healthier weight practices and policy• Tools to measure the impact of policy influence work
Complexity of HT policy influence
• In order to be effective, HT needed to
– Be responsive
– Be adaptable
– Build capacity
– Enhance knowledge uptake
HT policy influence - process
Policy Development
Placement on Policy Agenda
Policy Adoption
Policy Implementation
Policy Maintenance
Tool for HT Policy Influence Work –Goals & Activities
Practice/ Policy Goal Activities/Tactics Timeline
What is an example of a practice/ policy related to achieving healthier weights?
How do you envision Healthy Together influencing practice/policy discussions?
What practice/policy goal(s) do you think is possible within your sphere of influence?
What activities/tactics will you engage in, to achieve the above goals, and the estimated timeline?
Comments
Thank you for your participation!
Office/Agency
Organization/Network
Community
Province
Other
HT policy influence activities
• Program uptake/practice change by sites
• Establishing relationships with key decision makers
• Policy makers’ education/ capacity building
• Policy development/maintenance
• Policy implementation/evaluation
• Demonstration projects/pilots
• Electronic outreach through social media
• Briefings/presentations
• Adoption of HT approach into core practice• Communities/systems engaged in ‘buy-in’
• Stakeholders leverage partnerships to support uptake
Ultimate outcome:
‘Vulnerable children will experience greater equality of health outcomes and achieve healthier weights. ‘
HT policy outcomes
HT theory of change
• If families at-risk of developing unhealthy weights come together through programming that enables healthy eating, increases enjoyment of physical activity and strengthens the family bond, then members will improve their overall health and overall quality of life with skills and habits that will last.
How do I get the guide?
NCCMT:
http://www.nccmt.ca/uploads/media/media/0001/01/c3374b8be4b35e1340385f1b593d3bb9f50f6a38.pdf
the guide?
Wrap Up – Key Messages
• Policy-influence in order to support scale-up of evidence-based programs is a concern to many funders and organizations
• There is a burgeoning literature on how to do and evaluate policy-influence work
• The Guide to Policy-Influence Evaluation will help you navigate this literature with a select number of high quality useful resources
• The Guide can help to:• Develop evaluation requirements for policy-influence work
• Support projects and organization to do and evaluate policy-influence work
Thank You!Questions/Comments/Contact
www.healthy-together.ca
The Guide: http://www.nccmt.ca/resources/search/241
Follow us @nccmt Suivez-nous @ccnmo66
Your Comments/Questions
• Use Chat to post comments
and/or questions
• ‘Send’ questions to All (not
privately to ‘Host’)
Chat
Participant Side
Panel in WebEx
Follow us @nccmt Suivez-nous @ccnmo67
Poll Question #4
Could this method or tool be useful
in practice?
A. Very useful
B. Somewhat useful
C. Not at all useful
D. Don’t know
Follow us @nccmt Suivez-nous @ccnmo68
Your Feedback is Important
Please take a few minutes to share your thoughts
on today’s webinar.
Your comments and suggestions help to improve
the resources we offer and plan future webinars.
The short survey is available at: https://nccmt.co1.qualtrics.com/SE/?SID=SV_8ib8Yz9pRao
GLBz
Follow us @nccmt Suivez-nous @ccnmo69
Poll Question #5
Would you be willing to receive an
email from the NCCMT with a
follow-up survey in 2-3 months?
A. Yes
B. No
Follow us @nccmt Suivez-nous @ccnmo70
Join us for our next webinarNCCMT Spotlight on Methods & Tools: CDC Clear Communication Index
Date: Wednesday, February 1, 2017
Time: 1:00 – 2:30pm EST
Interested in evidence-based criteria for clear communication
of public health material? Do you develop and review public
health communication materials? Are you interested in
enhancing clarity and ease of understanding of these
materials? Join us for a webinar on how the Clear
Communication Index can help you.
Register at: https://health-evidence.webex.com/health-
evidence/onstage/g.php?MTID=eb180a59b9994f1d4da0729d
c3d31345d
Follow us @nccmt Suivez-nous @ccnmo71
Poll Question #6
What are your next steps? (Check all
that apply)
A. Access the method/tool referenced in the presentation
B. Read the NCCMT summary about the method/tool described today
C. Consider using the method/tool in practiceD. Tell a colleague about the method/tool
Follow us @nccmt Suivez-nous @ccnmo
Funded by the Public Health Agency of Canada | Affiliated with McMaster UniversityProduction of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The
views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..
For more information about the National Collaborating Centre for Methods and Tools:NCCMT website www.nccmt.ca
Contact: [email protected]