Evidence for Impact on Student Learning

Tony NormanAssociate Dean, CEBS

In P. R. Denner (Chair), Evidence for Impact on Student Learning from the Renaissance Teacher Work Sample Assessment. Symposium presented at the annual meeting of the American Association for Colleges of Teacher Education, New York, NY, February 2008.

Roger PankratzAssistant to the Dean, CEBS

2

WKU Participants

• Fifty-two TWS from two semesters (fall 2006 and 2007) selected to maximize variability in holistic scores (4 – Exemplary, 3 – Proficient, 2 – Developing, 1 – Beginning)

• 73% Elementary Education Majors• 27% Middle Grades/Secondary

Education Majors

3

Teacher Work Sample Scores

• The WKU TWS uses an analytic scoring rubric with 7 targeted standards that are assessed using multiple indicators.

• The indicators are scored on a 3-point scale:1 = Indicator Not Met2 = Indicator Partially Met3 = Indicator Met

• Summing across indicators, the total TWS scores can range from 32 to 96.32 to 96.

4

Learning Impact Measures

Percent of students reaching “mastery” on each learning goal.

1. Average Percent Reaching Mastery2. Percent Mastering Learning Goal 1 3. Percent Mastering Learning Goal 2

• The candidates chose the learning goals and the difficulty of mastery level.

• Learning Goals had to fit the approved curriculum and align with Kentucky Core Content standards.

• However, the candidates set their own criteria for success on each learning goal.

5

Learning Impact Measures

Percent of students showing improvement on each learning goal.

1. Average Percent Showing Improvement2. Percent Showing Improvement on Learning Goal 13. Percent Showing Improvement on Learning Goal 2

Note: Candidates are encouraged to set Learning Goal 2 at a “higher level” in Bloom’s Taxonomy. Many of these are measured by candidate developed rubrics.

6

Reported Impacts on Student Learning

7

Regression Analyses for TWS Scores on the Reported Percent of Students Achieving Learning Goals

8

Regression Analyses for TWS Scores on the Reported Percent of Students Showing Improvement

9

A Closer Look at Key TWS Components

• 22 fall 2006 TWS were examined for quality of assessments

• 30 fall 2007 TWS were examines for quality of assessments and quality of unit learning objectives

10

Criteria for Judging Quality of TWS Assessment

• Alignment with unit objectives• Addresses depth of knowledge (DOK) level• Sufficient number of items• Rubrics provide instructions for judgment

11

Criteria for Judging Quality of TWS Learning Objectives

• Aligned with Kentucky Content Standards• Student friendly• Scope – contribution to academic year

Curriculum Map• Measurable

12

Findings (1)

Summary of Holistic Scores of Quality of Unit Objectives Compared to Holistic TWS Scores

N=32 Teachers

Unit Objectives Scored Higher

Unit Objectives Scored Same

Unit Objectives Scored Lower

4 (13%) 18 (56%) 10 (31%)

13

Findings (2)

Summary of Holistic Scores of Quality of Unit Summative Assessments Compared to

Holistic TWS ScoresN=52 Teachers

Unit Assessments Scored Higher

Unit Assessments Scored Same

Unit Assessments Scored Lower

8 (15%) 25 (48%) 19 (37%)

14

Where WKU Teacher Candidates Appear to Need Improved instruction

• Writing clear, student friendly unit objectives• Writing unit objectives for designated depth

of knowledge (DOK) levels• Developing classroom student assessments

most appropriate for unit objectives• Developing classroom student assessments

aligned with designated DOK levels

15

For a Copy of the Presentation…

http://edtech.wku.edu/rtwsc/publications-and-research.htm