strategies for communicating evaluation findings · strategies for communicating evaluation...
TRANSCRIPT
Housekeeping notes
• This webinar will be recorded, and will be available on the Centre’s website as an educational resource.
• The slides have been sent to participants.• Log in with a phone whenever possible for optimal
audio quality.• We have staff online to assist with any technical
difficulties.• There will be a short evaluation survey for all webinar
attendees at the completion of the webinar.
Housekeeping notes
• This webinar will be followed by a question and answer period, however questions are encouraged throughout the presentation.
• Questions can be submitted electronically or verbally. Specifics around this process will be clarified at the end of the webinar .
Addressing questions:
Presenters
Lindsey ThomsonProgram Associate
Ontario Centre of Excellence for Child and Youth Mental Health
Trevor BosseQuality Assurance Associate
Child and Community Resources
What is the Centre?
Vision
Optimal mental health and well‐being for children and youth.
Mission
We bring people and knowledge together to strengthen the quality and effectiveness of mental health services for children, youth, and their families and caregivers.
5
What is the Centre?The plan outlines three key strategic priorities:
Goal 1Foster a culture of organizational learning to support agencies in using evidence to improve client outcomes
Goal 2Build and develop collaborative partnerships to sustain capacity within mental health services
Goal 3Be a true learning organization and lead by example
6
Overview
• To talk about the importance of communicating evaluation findings
• To suggest ways of tailoring communication to different stakeholders
• To have an agency share its knowledge exchange experiences
• To share ideas on innovative knowledge exchange approaches
• To share information on helpful resources
Goals of the webinar:
The Centre’s definition of knowledge exchange
“At the Centre, we see knowledge exchange as the continuous and purposeful flow of evidence and expertise that fosters mutual learning and leads to stronger mental health care for Ontario children, youth and families. By gathering, sharing and using the best available information, we work to amplify the impact of current knowledge, build meaningful connections, and strengthen our collective ability to respond to evolving challenges in the field”.
Various terms for knowledge exchange
Knowledge translation
Knowledge mobilization
Knowledge transfer
Implementation science
Why communicate evaluation findings?
• To share information about the evaluation process and its findings
• To facilitate understanding of the evaluation among different audiences
• To support decision‐making about the program and/or the evaluation
Torres, R.T., Preskill, H., & Piontek, M.E. (2005). Evaluation strategies for communicating and reporting: Enhancing learning in organizations (2nd ed.). Thousand Oaks, CA: Sage Publications.
Preparing a knowledge exchange plan
• Planning for knowledge exchange should occur while planning the evaluation project
• The plan should include at a minimum:o a listing of the possible audienceso information on the best ways of communicating the evaluation findings to each audience
Preparing a knowledge exchange plan
Knowledge exchange questions
What are our goals in sharing evaluation findings?
Who are the stakeholders?
What are the key messages of the evaluation?
Who needs to know what?
How can each audience best be reached?
What are their information needs?
How can I best present this information to enhance understanding and use?
How will you share sensitive or negative results?
Identifying audiencesPossible audiences are those who:
• sponsored or commissioned the evaluation.• will make decisions based on the evaluation results.• plan, develop or deliver the program being evaluated.
• participated in the evaluation and provided information.• participate in the program.• fund the program.
• are interested in the program (e.g. advocacy groups).• have a right to the information (e.g. legislators, taxpayers).
Adapted from: Torres, R.T., Preskill, H., & Piontek, M.E. (2005). Evaluation strategies for communicating and reporting: Enhancing learning in organizations (2nd ed.). Thousand Oaks, CA: Sage Publications.
Tailoring the communication to the audience
The communication strategy will depend on audience characteristics such as:• accessibility (e.g. geographic location)
• reading ability
• familiarity with (or interest in) the program
• role in decision‐making
• familiarity with evaluation
Adapted from: Torres, R.T., Preskill, H., & Piontek, M.E. (2005). Evaluation strategies for communicating and reporting: 2nd edition. Thousand Oaks, CA: Sage Publications, Inc.
An example ‐ presentationAudience Message Delivery method Messenger Evaluation
What are the general demographics of the audience?What may be their expectations? How can I reach them, let them know about the presentation?
What problems or questions should be addressed?What are the concluding remarks and actionable messages?
Using visual aids, short examples, or anecdotes helps to engage the audience. Posing and answering questions are techniques to encourage a two‐way flow of information and prevent a “knowledge dump.” The pace should be adjusted to the knowledge of the participants.
What are the required knowledge and skills for this presentation and who has them? Credibility and affiliations?
What are the realistic, achievable objectives of the presentation? What are the benefits to the users/receivers? What specific knowledge or message can they take back to their organizations and share?
Reporting findings – evidence tables
Evaluation question Evidence Conclusion Recommendation
Have children aged 6‐12 demonstrated increased social competence?
Self report – Statistically significant change on the SSRS and CGI scales
Parent ratings –Statistically significant change on the SSRS and CGI scales
Observation – Positive change on the BOSS
Qualitative interviews – campers, parents and staff
Children and youth leave the program with increased social competence
Seeing as this program meets a specific need for this population of children and youth with learning disabilities, funding and support for this program should continue.
Example: Evaluation of a therapeutic summer residential program
Adapted from: Evaluation Implementation Grant 889 ‐ Integra
Our Valued Stakeholders
Sharing Evaluation Results
Funders:• Ensure program is meeting the needs of
clients• Program accountability• Resource allocation
Our Valued Stakeholders
Sharing Evaluation Results
Clients:• Why do we evaluate?• At a glance, how successful are we at
achieving what we’ve set out to do?• How are you improving your services?
Our Valued Stakeholders
Sharing Evaluation Results
Governing Bodies (Board of Directors):• How did we do?• How are we improving?
Sharing Results
1. Comprehensive Final Report2. One‐pager for external stakeholders3. One‐pager for Board of Directors4. Crafty ways of sharing results5. Presenting Results to Teams
Sharing Evaluation Results
Sharing Evaluation Results
2012 Knowledge Exchange
• Offered to all CCR staff in June 2012• Increasing the capacity of staff surrounding
program evaluation and research, and how they contribute to evidence‐informed practice and organizational learning
Sharing Evaluation Results
2012 Knowledge Exchange
What we found…• 99% of attendees (n=109) agreed or strongly agreed that they were satisfied with the workshop
• 100% agreed or strongly agreed that the format of the session was effective for their learning
Innovative knowledge exchange strategies
• Poster
• Video
• Infographic
• Photovoice
• Board game
• Drama/theatre
• Storytelling/graphic novel
• World Café
Acknowledgements: Work at Adventure Central is part of CYFAR Project # 2003-41520-01560. This research was supported by the Susan Barkman Research and Evaluation Award, National Association of Extension 4-H Agents.
Theresa M. Ferrari, Nate Arnett, & Kristi Lekies – Ohio State University Extension
Youths’ Perspective on Their Long-Term Participation in an Urban 4-H Youth Development Program: Results of a Qualitative Study
I’ve done
“things you
would never
imagine you
would do…It
has just opened
new doors.
“There is a lot of team building ….before we even became group
leaders and actually worked here, we were still doing team building activities in our groups when we were younger. That helped us to adapt to other
people, like problem solving skills and being able to communicate.”
“There is a lot of team building ….before we even became group
leaders and actually worked here, we were still doing team building activities in our groups when we were younger. That helped us to adapt to other
people, like problem solving skills and being able to communicate.”
“I like it a lot here. Not only is it
educational, but it is fun here as well.”
“I like it a lot here. Not only is it
educational, but it is fun here as well.”
Participants
•16 participants between 11 & 16 years old
•Participated for at least 3 years (range from 3 to 7 years)
Purpose of the Study
Explore the process and outcomes of long-term participation in a youth development program.Research Questions
1. How has participation been helpful?
2. What skills & attitudes have youth acquired?
3. What opportunities have been afforded to them?
4. What program features have captured their interest & engaged them in sustained participation?
•Youth obtain developmental benefits from consistent participation in well-run, quality youth development programs.
•Frequency, duration, and breadth of participation relate to more positive outcomes.
•Joining and persisting in out-of-school activities is a dynamic process.
•Beginning to understand features that contribute to engagement.
Program DescriptionResearch BaseResearch Base
Methodology
•Partnership between OSU Extension 4-H Youth Development & Five Rivers MetroParks in Dayton, OH
•Opened in October 2000
•Hub of out-of-school activities 4 days a week
•Youth aged 5 to 18
•Computer lab and homework assistance
•Programs focus on science, nature, literacy, & healthy living, workforce skills, leadership, & cultural literacy
•Diverse array of full-time staff, part-time staff, & volunteers
Implications & Recommendations
ResultsResults
Data Collection & Analysis
•Four focus groups
•Qualitative analysis for themes
Youth Cited Three Key Themes Related to their Long-Term Participation:
•Multiple OpportunitiesFrom an early age, youth had new and challenging opportunities to demonstrate leadership, have a “real” work experience, travel, go outdoors and engage with nature, and get homework help.
•Having Fun and LearningBeyond keeping them out of trouble, youth recognized that they were learning and having fun while learning skills.
•BelongingMultiple references to “home” and “family” were used to describe the program environment to include staff and peers.
•Use of a model of positive youth development programming connecting program features, developmental needs, and outcomes
• Increasing opportunities for youth as they get older
•Connecting youth with other leadership opportunities
•Potential model for other urban 4-H and youth development programs to implement
Photovoice
Source: “Capturing the unique strengths and challenges of thepedestrian environment in Fort Frances, ON” –Safe KidsCanada and local partners
Storytelling/Graphic novel
Source: http://www.youth.society.uvic.ca/sites/default/files/images/PAGE%204-1.jpg
Summary• Knowledge exchange should be planned in the early stages of an evaluation project, and should happen at various points throughout the project.
• Communication strategies should be tailored to the audience.
• A range of communication strategies should be considered.
• Learning from others’ knowledge exchange experiences can be helpful.
Questions?
To submit questions electronically, use the Question box located in the control panel.To submit questions verbally, use the ‘Raised Hand’ icon also located in the control panel.
Upcoming Webinars
To infinity and beyond: Sustaining evaluationMelissa Jennings & Surbhi Bhanot‐Malhotra
May 21 at 1 p.m.
Final report preparation and writingLindsey Thomson & Jennifer Puddicombe
June 4 at 1 p.m.
ContactFor more information on this webinar or topic
Lindsey ThomsonProgram Associate
Ontario Centre of Excellence for Child and Youth Mental HealthPhone: 613‐737‐2297 ext. 3426
Trevor BosseQuality Assurance Associate
Child and Community ResourcesPhone:(705) 222‐5000 ext. 2818