Evaluating the Impact of your Capacity Building Plan …...Why is Evaluation Important? • If you can’t see success, you can’t learn from it • If you can’t recognize failure,
37
Evaluating the Impact of your Capacity Building Plan for Transition June Gothberg, Ph.D. Brian Molina, M.A. Western Michigan University Jacque Hyatt, M.Ed. TransCen
Evaluating the Impact of your Capacity Building Plan for Transition
June Gothberg, Ph.D. Brian Molina, M.A.
Western Michigan UniversityJacque Hyatt, M.Ed.
TransCen
Presenter
Presentation Notes
All three of us – short introduction with a little background for the context of evaluation i.e,., Good afternoon, we are so glad you’ve stayed after three intensive days already to attend our workshop! We are planning to make this a very interactive session and keep you awake. I am June Gothberg, part of the NTACT team from WMU. One of my roles is to help lead evaluation efforts for our team and those we serve. My background includes 10 years in the classroom with students on IEPs and also with adjudicated youth. I worked as a transition specialist, a vocational evaluator, and a state administrator as well. My Ph.D. is in EMR and I love working with teams to help them gather, understand, and make decisions based on their data. I am Brian Molina, part of the NTACT team from WMU. I am also on the NTACT evaluation team. My background includes… I am Jacque, part of the NTACT team from TransCen. I am part of the NTACT technical assistance team providing expertise in rehabilitation services. My background includes…
Why is Evaluation Important?
Presenter
Presentation Notes
June sets stage… With all the demands for data, why is evaluating implementation efforts so important?
Why is Evaluation Important?
• What gets measured gets done
• If you don’t measure results, you can’t tell success from failure
• If you can’t see success, you can’t reward it
• If you can’t reward success, you’re probably rewarding failure …..
Presenter
Presentation Notes
June sets stage… What gets measured gets done If you don’t measure results, you can’t tell success from failure If you can’t see success, you can’t reward it If you can’t reward success, you’re probably rewarding failure ….. If you can’t see success, you can’t learn from it If you can’t recognize failure, you can’t correct it If you can demonstrate results, you can win public support From: Osborne & Gaebler, 1992, Chapter 5, “Results Oriented Government”
Why is Evaluation Important?
• If you can’t see success, you can’t learn from it
• If you can’t recognize failure, you can’t correct it
• If you can demonstrate results, you can win public support
June sets stage… What gets measured gets done If you don’t measure results, you can’t tell success from failure If you can’t see success, you can’t reward it If you can’t reward success, you’re probably rewarding failure ….. If you can’t see success, you can’t learn from it If you can’t recognize failure, you can’t correct it If you can demonstrate results, you can win public support From: Osborne & Gaebler, 1992, Chapter 5, “Results Oriented Government”
http://www.nsttac.org
The don’ts
The doesn’t
Lessons Learned
Presenter
Presentation Notes
Brian What we’ve learned from team plan analysis: If teams don’t anticipate their outcomes, they can’t tell if they’ve achieved them. If teams don’t plan for evaluation, evaluation is not likely to happen. When evaluation doesn’t happen, teams don’t know if they have accomplished what they planned.
Evaluation Planning: When?
• Formative vs. Summative(Ongoing vs. at completion, quality of the plan vs. attainment of the outcomes)
“When a cook tastes the soup, it is formative evaluation; when the dinner guest tastes the soup, it is summative evaluation.”
~Harvey, 1998
Presenter
Presentation Notes
Jacque
Evaluation Planning: How?
• Quantitative vs. Qualitative
• Good evaluations often include both
Presenter
Presentation Notes
Jacque
8
Presenter
Presentation Notes
Jacque The most effective evaluations are created during strategic planning and enable teams to: make the most of resources increase the quality of data collected ensure the results will be constructive
NTACT Six Levels of Evaluation
http://www.transitionta.org
Evaluation Level One: Quality, Usefulness, and Relevance
Evaluation Level Three: Organizational Policies, Procedures, and Support
Evaluation Level Four: Program Implementation
Evaluation Level Five: In-school and Post-school Outcomes
Evaluation Level Six: Evaluation Use and Dissemination
Presenter
Presentation Notes
June
http://www.nsttac.org
Quality
Relevance
NTACT Level 1
Usefulness
Presenter
Presentation Notes
Level 1 – Participant reactions Likert-like scale evaluations (quantitative) of institutes, cadre, workshops, trainings Achievement of intended outcomes Usefulness of information Relevance of materials Qualitative open ended questions What worked and what didn’t
Level 1: Quality, Usefulness, and Relevance
http://www.transitionta.org
Presenter
Presentation Notes
June I am sure you all have seen this evaluation or something like it. This is an example from one of our State Transition Leadership Institute. You can clearly see this is a low level evaluation that provides a quick understanding of how people perceived the institute. We used a very simple 3 point Likert-like scale for the evaluation followed by open-ended questions to allow collection of richer descriptions. This is a good example of one way to include both quantitative and qualitative methods in your evaluation tools. Please note, the Likert-like scales are not limited to level one evaluation, they may be used to collect higher level data as well.
http://www.nsttac.org
Participant Learning Outcomes
NTACT Level 2
Presenter
Presentation Notes
Brian Pre-post tests (quantitative) New knowledge and skills of participants: student, teacher, and parent instruments Analysis of products (qualitative) Development of IEPs Level 2: Participant Learning Outcomes focuses on measuring the knowledge, skills, and perhaps attitudes participants gained. Besides “liking it,” we would hope participants learned something from their professional development experience. Measures must be based on the learning goals prescribed for that particular program or activity. This means specific criteria and indicators of successful learning must be outlined prior to the beginning of the professional development experience. Analysis of this information provides a basis for improving the content, format, and organization of the program or activities. Depending on the goals of the program or activity, evaluation can involve: a pencil‑and‑paper assessment- Can participants describe the critical attributes of an IEP required by Indicator 13? a simulation or full-scale skill demonstration- Can participants simulate a student-led IEP meeting? oral or written personal reflections examination of the portfolios participants assemble
Level 2: Participant Learning Outcomes
http://www.transitionta.org
Presenter
Presentation Notes
Brian This is an example of a SEA created evaluation. This is an example of how to gain information quickly of knowledge gained. An example of post-then-pre testing, this tool is given after the training to get participant’s self-assessment of what they knew before the training and what they know now.
http://www.nsttac.org
Policies
Support
NTACT Level 3
Procedures
Presenter
Presentation Notes
June Analysis of teacher reports regarding curriculum implementation Identification of facilitators and barriers to curriculum implementation, including administrative support Analysis of annual performance reports (APRs) to determine Change in data collection procedures Alignment of strategic plans (from institutes) with improvement activities in “determination ” areas Change in target indicators At Level 3: Organizational Policies, Procedures, and Support our focus shifts to the organization Specifically, to information on organization support and change. Organizational variables can be key to the success of any professional development effort. They also can hinder or prevent success, even when the individual aspects of professional development are done right (Sparks, 1996a). Gathering information on organization support and change is generally more complicated than previous levels. Questions focus on the organizational characteristics and attributes necessary for success. (best practices for capacity building!!) Procedures may involve analyses of district or school records, documents, or rules, examination of the minutes from follow-up meetings, compliance results, and the like. Questions may include: Was the advocated change aligned with the mission of the organization? Was change at the individual level encouraged and supported at all levels? Did the program or activity affect organizational climate and procedures? Was administrative support public and overt? Were problems addressed quickly and efficiently? Were sufficient resources made available, including time for sharing and reflection? Were successes recognized and shared? Issues such as these can be major contributing factors to the success of any professional development effort.
Level 3: Organizational Policies, Procedures, and Support
http://www.transitionta.org
Goal Attainment Scale (GAS)Used to measure the progress towards meeting
goals and objectives• Expresses distinct goals on a common scale• Periodic review (monthly/quarterly) of
obtaining anticipated outcomes• Validates anticipated outcomes or identifies
needs for adjustment
Presenter
Presentation Notes
June
We will increase the number of students responding to the post school outcomes data collection from 25% to 50%, the district will design and implement a marketing plan that targets youth from these underrepresented groups.
Presenter
Presentation Notes
June
We will increase participation of parents/families in Transition Councils, the district will collaborate with Parent Training and Information Center to develop professional development on transition for parents.
Presenter
Presentation Notes
June
Presenter
Presentation Notes
June
http://www.nsttac.org
Program Implementation
NSTTAC Level 4
Presenter
Presentation Notes
Fidelity of Implementation
Level 4: Program Implementation
http://www.transitionta.org
Presenter
Presentation Notes
June Several of our sites implemented new self-determination curriculum. This is one of the five tools used to triangulate data around implementation. Tool 1 shown here allows for an understanding of fidelity of implementation. Was the curriculum taught as intended. You can see it asks for how many lessons were completed? How many were modified? In sections not seen here it asks was this implemented as a stand-alone curriculum or was it integrated into another course? All these questions help clarify and validate why your outcomes were met or not met.
http://www.nsttac.org
In-school Outcomes
Post-school Outcomes
NSTTAC Level 5
Presenter
Presentation Notes
Tools are identified by effective practices and evaluation levels. Effective practices - these areas are replicated from the strategic planning tool. Evaluation levels - most education units are faced with evaluating learning events. Workshops Professional development Coaching Technical assistance
Level 5: In-school and Post-school Outcomes
http://www.transitionta.org
Presenter
Presentation Notes
June This is Tool 4 from the self-determination evaluation. It seeks information from the students perspective of their performance at the IEP meeting.
http://www.nsttac.org
Dissemination
Use
NSTTAC Level 6
Presenter
Presentation Notes
What stakeholders were interested in the evaluation findings? What methods of dissemination were implemented to reach all stakeholders? Did stakeholders understand the findings to the extent that they made informed decisions regarding their use? How do you know? What changes were made to improve transition education and services? (e.g. federal, state, and local agencies, school boards, administrators, teachers, and school staff, parent groups, families, and students) Finally, evaluation results need to be disseminated and applied. Level six examines (a) the dissemination of evaluation findings and (b) the extent to which findings are used to improve transition education and services. Important questions to ask at this level include: • What stakeholders were interested in the evaluation findings (e.g. federal, state, and local agencies, school boards, administrators, teachers, and school staff, parent groups, families, and students)? • What methods of dissemination were implemented to reach all stakeholders? • Did stakeholders understand the findings to the extent that they made informed decisions regarding their use? How do you know? • What changes were made to improve transition education and services? The methods used to disseminate findings affect their application. Often evaluation findings are disseminated in the form of a formal report. This manner of dissemination can narrow the intended audience to only those stakeholders with a fiscal or political interest in the findings. However, when findings are disseminated via multiple modes and venues, additional stakeholders may use those findings to assist with decision making and program improvement.
Level 5: In-school and Post-school Outcomes
http://www.transitionta.org
Presenter
Presentation Notes
June This is a tool used by a state team to assess their evaluation plans, implementation, and reporting.
Group Activity Time
http://www.transitionta.org
Presenter
Presentation Notes
Jacque Now we are going to break into groups based on the goal you’d like to focus your evaluation efforts on… Move to the section of the room with the taxonomy area of your goal (point to flipcharts with taxonomy names) – you will want to take your materials with you We will give you five minutes to get in your groups and introduce yourselves. Tell each other your name and the state you are from.
Evaluation – Is it a mountain?
http://www.transitionta.org
Presenter
Presentation Notes
Jacque Okay – time! Now we are going to focus on step one – evaluation questions Often people are intimidated by evaluation making it harder than it needs to be – today we want to help you approach and plan your evaluation in a simple and effective way
Evaluation Questions
http://www.transitionta.org
Presenter
Presentation Notes
Jacque Okay – time! Now we are going to focus on step one – evaluation questions Often people are intimidated by evaluation – today we want to help you approach and plan your evaluation in a simple and effective way
Evaluation Questions
http://www.transitionta.org
Good evaluation questions are:• easy to understand• written in plain language (no acronyms, jargons, or
abbreviations)• not too long• only ask one thing per question• don’t ask leading questions• offer mutually exclusive and exhaustive categories• can be open or close-ended• determines needs to be measured (Y/N, #, %, difference
score, range, etc.)
Presenter
Presentation Notes
Jacque
Evaluation Questions
http://www.transitionta.org
Outcome: Increased number of work-based learning opportunities provided to youth prior to school exit in the 10 pilot districts• Of the 10, how many districts increased work-based learning
opportunities? (# of districts)• How much did total opportunities for work-based learning
increase? (# of opportunities overall)and/or
• How much did opportunities for work-based learning increase by district? (# of opportunities disaggregated by district)
Presenter
Presentation Notes
Jacque – a couple of examples
Evaluation Questions
http://www.transitionta.org
Outcome: Increased number of IEPs that document student-participation across the state• How many district IEPs include documentation of student
participation in the IEP? (# of districts)• What percent of the state document student participation in
the IEP? (% of districts)• What was the level of increase in districts documenting
student participation in the IEP? (difference score between baseline and completion)
Presenter
Presentation Notes
Jacque – example two
Evaluation Questions
http://www.transitionta.org
Presenter
Presentation Notes
Jacque – in your groups take the next 10 minutes to write down some evaluation questions you are interested in answering for your focus goal Select one person to report out.
Time to Share
32
Evaluation Indicators
http://www.transitionta.org
Good evaluation indicators are measurable:• yes/no• # of• % of• difference between pre/post• range of
Presenter
Presentation Notes
Brian
Evaluation Indicators
http://www.transitionta.org
Presenter
Presentation Notes
Brian – at your tables select one or two questions you’d like to use to evaluate your outcomes and write what indicators you would use to measure them
Evaluation Data Sources and Tools
http://www.transitionta.org
Evaluation data sources and tools, remember:• you don’t need to reinvent the
wheel• you can use already existing tools• you can create your own if
needed• NTACT is here to assist you • share your experiences, tools,
