Essential Practices for Effective Rubric Implementation:Tips on Training and Using Data for Improvement

Ashley Finley, Ph.D

Senior Dir. of Assessment & Research, AAC&U

National Evaluator, Bringing Theory to Practice

General Education & Assessment Conference

Portland, OR

February 27, 2014

Linda Siefert, Ed.D

Director of Assessment

University of North Carolina-Wilmington

Capturing What Matters: VALUE Rubrics Initiative

Rubric Development 16 rubrics Created primarily by

teams of faculty Inter-disciplinary, inter-

institutional Three rounds of testing

and revision on campuses with samples of student work

Intended to be modified at campus-level

Utility Assessment of students’

demonstrated performance and capacity for improvement

Faculty-owned and institutionally shared

Used for students’ self-assessment of learning

Increase transparency of what matters to institutions for student learning

VALUE Rubrics (www.aacu.org/value)

Knowledge of Human Cultures & the Physical & Natural Worlds Content Areas No Rubrics

Intellectual and Practical Skills Inquiry & Analysis Critical Thinking Creative Thinking Written Communication Oral Communication Reading Quantitative Literacy Information Literacy Teamwork Problem-solving

Personal & Social Responsibility Civic Knowledge &

Engagement Intercultural Knowledge

& Competence Ethical Reasoning Foundations & Skills for

Lifelong Learning Global Learning

Integrative & Applied Learning Integrative & Applied

Learning

Criteria

The Anatomy of a VALUE RubricThe Anatomy of a VALUE Rubric

Levels

Performance Descriptors

The Role of Calibration Essential training through collaboration

and discussion Transparency of standards (without

standardization) Building shared, inter-disciplinary

knowledge around skills assessment and application of the rubric

Shared stake in data collection and discussion for improvement

Building intentionality of assignment development

The Calibration Training Process Scoring Steps:

Review rubric to familiarize yourself with structure, language, performance levels

Ask questions about the rubric for clarification or to get input from others regarding interpretation

Read student work sample Connect specific points of evidence in work

sample with each criterion at the appropriate performance level (if applicable)

Calibration Steps: Review scores Discuss scoring rationale Opportunity to change scores

The Ground Rules We are not changing the rubric (today). This is not grading. Think globally about student work and about the

learning skill. Think beyond specific disciplinary lenses or content.

Start with 4 and work backwards. Connect evidence in work sample with language in

performance cell. Pick one performance benchmark per criterion. Do

not use “.5”. Zero and NA do exist but are distinct. Assign “0” if

work does not meet benchmark (cell one) performance level. Assign “not applicable” if the student work is not intended to meet a particular criterion.

How Have Campuses Used Rubrics to Improve Learning?

Using the VALUE Rubrics for Improvement of Learning and Authentic Assessment

12 Case Studies Frequently asked

questions

http://www.aacu.org/value/casestudies/

When presenting assessment results…

Consider the Type of Raw Data Rubrics describe categories. These categories are generally ordered. (4

reflects higher quality performance than 3.) But they are likely not equally distributed

on a number line. ( 1 is not the same distance from 2 as 2 is from 3. 4 is not twice a “good” as 2.)

Therefore, summary presentations and statistical procedures should be appropriate for ordinal numbers.

Consider the Readers

Disciplines have commonly used styles of presenting data.

Individuals have personal learning styles. Consider presenting the data in multiple

forms that will resonate with different styles. Charts Graphs Verbally, qualitatively (rubrics provide

quality descriptions for each level)

Chart

Dimension% of Work

Products Scored Two or Higher

CT 1 Explanation of Issues 83.4%

CT2 Evidence 77.0%

CT3 Influence of Context and Assumptions 58.6%

CT4 Student’s Position 70.1%

CT5 Conclusions and Related Outcomes 56.7%

The percent of students meeting the adopted benchmark for lower-division courses.

Example Combination Chart and Graph

CT1 CT2 CT3 CT4 CT5

0 0.017 0.019 0.093 0.037 0.101

1 0.149 0.211 0.321 0.262 0.332

2 0.315 0.307 0.265 0.287 0.273

3 0.416 0.372 0.275 0.339 0.263

4 0.103 0.091 0.046 0.075 0.031

10%

30%

50%

70%

90%

Example Verbal Description

CT4 Student’s Position

Less than one in twenty work products contained no statement of student position (scores of 0). One fourth of the work products provided a simplistic or obvious position (scores of 1). Three in ten work products provided a specific position that acknowledged different sides of an issue (scores of 2). One third of the work products took into account the complexities of the issue and acknowledged the points of view of others (scores of 3). And less than one in ten work products provided an imaginative position that took into account the complexities of the issue, synthesized others’ viewpoints into the position, and acknowledged the limits of the position taken (scores of 4)

Scenario

You will work in table groups to analyze and then discuss recent assessment findings and how they might be used to improve student learning.

Share Out