evaluating the impact of professional development: five ... · evaluating the impact of...

1

Evaluating the Impact of

Professional Development:

Five Levels of Evidence

Colorado Transition Leadership Institute

June 22, 2011June Gothberg , Western Michigan University

Agenda

Information on evaluation

Information on five levels of evaluation

Tools for evaluation

Hands-on practice

2

Intended Outcomes

Increase knowledge of five levels of

evaluation

Increase knowledge of the continuum of the

five levels of evaluation starting from

participant satisfaction and ending with

student outcomes

Intended Outcomes

Include appropriate potential indicators and

data sources to evaluate each TAP expected

outcome

Increase knowledge of evaluation tools

Demonstrate knowledge by creating an

evaluation tool

3

Purpose

Improve

student

outcomes!

Purpose of Our Work

Process

Improve

student

outcomes by

improving

what we do!

Purpose of Our Work

4

Proof

Evaluate

improvement

efforts to

determine

effect on

student

outcomes

Purpose of Our Work

Evaluation Connection

Program

planning

Program

evaluation

The best evaluation plans are created during

the strategic planning phase.

5


Evaluation compares the measured impacts

and outcomes against the strategic plan

Evaluation looks at:

the original objectives

what was accomplished

how it was accomplished.


Program

planning

Identify

outcomes

Program

evaluation

Measure

outcomes

6

Lessons Learned

What we’ve learned from local plan analysis:

If teams don’t anticipate their outcomes, they can’t

tell if they’ve achieved them

If teams don’t plan evaluation during the institute

planning phase, evaluation is not likely to happen

When evaluation doesn’t happen, teams don’t know

if they have accomplished what they planned

Our Focus: To Link

Planning and Evaluation

GoalsActivities

Outputs

Outcomes

7

Pull the Pieces Together!

GoalsActivities

Outputs

Outcomes

What is Evaluation?

A study designed and conducted to assist an

audience to assess an object’s merit and

worth.1

The identification of defensible criteria to

determine an evaluation object’s value (merit

or worth), quality, utility, effectiveness, or

significance.2

1. Stufflebeam, D. (1999)

2. Worthen, Sanders, Fitzpatrick (1997)

8

What is Evaluation?

Evaluation is judging the:

Merit

Worth

Or significance of things

near synonyms are quality/value/importance

1. Stufflebeam, D. (1999)

2. Worthen, Sanders, Fitzpatrick (1997)

What is Evaluation?

1. Scriven, M. (1990)

Value

Merit

Value to an individual consumer

Worth

Value to an institution or

collective

9

Evaluation Questions

In general most teams want to answer

these three questions:

Did we do what we said we were going to do?

How well did we do it?

What difference did it make?

In other words what were our outcomes?

Assessing your Outcomes

Outcomes

10


What do

you want to

achieve?


What do

you want to

achieve?

How will you

know you

achieved it?

11


Short-term Long-term


Short-termChanges in skills,

attitudes,

knowledge,

behavior, and

decision making

Long-termChanges in status

or life condition

12

Five Levels of Evaluation

Kirkpatrick’s four-level training evaluation

model

Reaction

Learning

Behavior

Results


Tom Guskey adapted this for his work on

professional development evaluation to

include a 5th level and what may be the most

important level for our work today:

Student learning outcomes

13


NSTTAC adapted Gusky’s model for capacity

building and technical assistance evaluation

Guskey based on evaluating a one-time linear

professional development event

NSTTAC’s model adapted for complex multi-site,

multi-level evaluation efforts

Adapted from: Guskey, T. R. (2000). Evaluating Professional Development. Thousand Oaks, CA: Corwin Press


Adapted from: Guskey, T. R. (2000). Evaluating Professional Development. Thousand Oaks, CA: Corwin Press

Level 5: Student learning outcomes

Level 4: Participant use of new knowledge and skills

Level 3: Organization support and change

Level 2: Participant learning

Level 1: Participant reactions

14

NSTTAC Examples - Level 1

Level 1 – Participant reactions

Likert-like scale evaluations of institutes,

cadre meetings, workshops

Achievement of intended outcomes

Usefulness of information

Relevance of materials

Qualitative open ended questions

What worked and what didn’t

Colorado Example - Level 1

Gothberg & Goldsby, 2010, Colorado Transition Leaders Institute Evaluation 2010

15

Level 1 – Report Table


Level 2 – Participant learning

Pre-post tests

New knowledge and skills of participants:

student, teacher, and parent instruments

Analysis of products

Development of IEPs

16


Wirth & Lowenthal, 2009, I-13 Training Pre-then-Post Evaluation


Note. Frequency (f) represents the number of participants with a correct answer on the pretest and

posttest. Dependent t test (across all items) revealed a significant difference between pretest scores and

posttest scores, t(396)=-22.06, p < .0001.

17

Level 2 – Report Graph


Level 3 – Organization support and change

Analysis of teacher reports regarding curriculum

implementation Identification of facilitators and barriers to curriculum

implementation, including administrative support

Analysis of annual performance reports (APRs) to

determine

Change in data collection procedures

Alignment of strategic plans (from institutes) with

improvement activities in “determination ” areas

Change in target indicators

18


Gothberg, 2010, Windsor Implementation Evaluation 2010


19


Level 4 – Participant use of new knowledge and skills

Analysis of state and local strategic plans (from

institutes)

To document and improve the implementation of

program content

To assess growth from year to year

Evaluation of local curriculum implementation

To assess if and how participants applied their new

knowledge at the classroom level


Kohler & Coyle, 2006, Self-determination Curriculum Evaluation, Tool 1

20



Level 5 – Student learning outcomes

Questions

What was the impact on students?

What was the impact performance or achievement?

What was the impact physical or emotional well-being?

Is student attendance improving?

Are dropouts decreasing?

What’s measured

Student learning outcomes: Cognitive, affective,

psychomotor

21

Level 5 - Student Learning

Ways to measure cognitive learning (knowledge):

Standardized achievement assessments

Standardized performance assessments (often scored

with rubrics)

Teacher-developed classroom assessments

Group tasks or activities

Portfolios and other collections of students’ work

Grades or marks

Questionnaires and interviews

School records


Ways to measure affective learning (attitudes &

dispositions) :

Questionnaires

Interviews

22


Ways to measure psychomotor learning (skills &

behaviors) :

Observations

Questionnaires

Interviews

School records



behaviors) :

Incidences in school vandalism

Student behavioral problems

Disciplinary recommendations

Suspensions

Tardiness

Absenteeism

Student participation in service organizations

Use of library, media center, or technology center

23



behaviors) :

Number of special service referrals

Dropout rates

Graduation rates

Post-school outcomes


Level 5 – Student learning

Analysis of APRs and SPP/APR Indicators

To determine school and student improvement on

federal performance and compliance indicators

To demonstrate the overall impact of capacity

building

To assess impact of capacity building model at the

state and local levels

Student portfolios and oral reports

To measure student learning outcomes

24


Level 5 – Student learning

Importance of student voice Student assessment gives us a quantitative view of what the

students learning level and yearly process may be, but when

you incorporate student voice into your evaluation work, you

may see beyond the test and into the impact happening at

the classroom level.

Please listen to the following clip from a student focus group:


Kohler, Coyle, & Gothberg, 2009, Self-determination Curriculum Evaluation, Tool 4

25


LEVEL 5 EVALUATION: STUDENT LEARNING OUTCOMES

EVALUATION Durant High School Tool 4: Student Self-Assessment of Student Involvement

Durant, OK, March 11, 2009

PURPOSE To measure extent self-determination courses have impacted student learning outcomes as seen in student involvement on the IEP

EVALUATION

QUESTIONS

Q1. Did the student attend their IEP?

Q2. How much did the student contribute in the IEP meeting?

DATA

SOURCES

Tool 4: Student Assessment of Student Involvement

OUTCOME Q1. The percentage of students that attended their IEP:

100% Students attended their IEP

Q2. The percentage of students that felt they contributed somewhat to yes in the IEP meeting:

100% Identified their post-secondary goals

100% Provided information about their strengths

100% Provided information about their limitations or problem areas

100% Provided information about their interests

100% Provided information about the courses they want to take

100% Reviewed their past goals and performance

100% Asked for feedback or information from the other participants at their IEP meeting

100% Identified the support they need

100% Summarized the decisions made at the meeting

IMPLICATION A significant increase in the number and extent of students involved in their own IEP as self-assessed

Evaluation Help

NSTTAC Evaluation Toolkit

A tool for “data-based” decision-making

Provides “real-life” examples for various states’

evaluation instruments

Samples for your use

Technical Assistance

26

Evaluation Tools

NSTTAC Evaluation Toolkit examples

The entire *updated* Toolkit may be

downloaded from:

www.nsttac.org

Evaluation Tools

NSTTAC Evaluation Toolkit

A tool for “data-based” decision-making

Provides “real-life” examples for various

states’ evaluation instruments

Samples for your use

27

Evaluation Planning Tool


28



29

Hands-on Activity

Please get into small groups

In an envelope are several section three

goal sheets

As a group, please select one for your

practice activity

Hands-on Activity

Use the handouts (Evaluation Planning, Five

Levels, and Evaluation Planning Tool) to

plan an evaluation of your selected goal

Create one evaluation product (survey,

observation guide, interview script)

Nominate someone to report out

30

Thank you!!

June Gothberg

Western Michigan University

evaluating the impact of professional development: five ... · evaluating the impact of...

Documents