rubrics - university of arizonaassessment.arizona.edu/sites/default/files/rubrics.pdf · types of...

Tucson, AZAssessment Coordinating Council - February 11, 2009

Grading Rubrics

Caroline R.H. Wiley and Darrell SabersDepartment of Educational Psychology

1


Overview

Why use rubrics?Psychometric considerationsTypes of scoring systemsRubric construction and examplesThe Rubric for Rubrics (ETS, 2006)

2

2


Why Use Rubrics?

Mechanism for judging the quality of student work (Arter & Chappuis, 2006)High-stakes nature of gradesClarityAdvantageous for diagnosing learning or summarizing learning (Assessment for learning vs.Assessment of learning)Reduce subjectivity as much as possible

3

3


How Do I know If I Need a Rubric?

If you administer...Essay testsPapers (research, theoretical, etc.)Constructed response items (paragraphs, a few sentences, 1-2 words)Performance assessments (speeches, debates, presentations, skits, skills (car repair, medical procedures, scientific procedures, etc.)

4

4


Psychometric Considerations

Content validityLearning objectivesBluffing

Absence of biasHalo effect

ConsistencyWith othersWith yourself

5

5


Types of Scoring Systems

ChecklistsRating scalesAnalytic rubricsHolistic rubrics

6

6


Checklists

Gronlund & Waugh (2009)

7

7


Checklists

McMillan (2007)

8

8


Rating Scales

Reynolds, Livingtson, & Wilson (2006)

9

9


Analytic Rubric

McMillan (2007)

10

10


Analytic Rubric

McMillan (2007)

11

11


Holistic Rubric

McMillan (2007)

12

12


Generic Rubric Construction

Define learning objectivesDetermine purpose

Procedures vs. final product Determine assessment purpose

Assessment for learning vs. Assessment of learningDetermine best type to use

Depends on all of the aboveIf using analytic, determine components and weights

Determine number of levels to use3 to 7

Determine criteria for the levelsRepresentative of desired learning objectivesParallel Eliminate double-penalties

13

13


ExampleTask: Using a thermometer

14

Excelling Proficient Below Proficient(20-25 points) (15-19 points) (1-14 points)

Preparation Grasps non-bulb end Either does not grasp Is not creative10% and wipes thermometer non-bulb end OR does in grasping non-bulb

downward. not wipe thermometer end.downward.

Administration Leaves thermometer in Only does 2 of the 3 Does not leave10% mouth for 3 minutes, actions described in thermometer in mouth

removes thermometer the “Excelling” for 3 minutes.grasping non-bulb end, category.and reads temperatureto the nearest tenth of

a degree.

Clean-up Correctly records Records temperature on Doesn’t record temperature5% temperature on patient’s patient’s record, but on patient’s record AND

record and cleans does so sloppily OR does not clean thermometerthermometer with an cleans thermometer with at all.

alcohol wipe. non-alcohol wipes.

Creativity Is very creative in the A little creative. Not creative at all.40% overall process of Student has own style No originality present.

measuring temperature. of taking temperature.

14


Stevens & Levi (2005)

15

15


Stevens & Levi (2005)

16

16


Watch Out For WeightsUsing different number of points in each component leads to misleading scoresChecklist: 5/17 = 29%Persuasiveness: 3/17 = 18%Delivery: 3/17 = 18%Sensitivity: 3/17 = 18%Holistic rating: 3/17 = 18%

Double-penaltyHolistic rating?

McMillan (2007)

17

17


The Rubric For Rubrics

ETS (2006)Criterion 1: Coverage/Organization

1A: Covers the right content1B: Criteria are well organized1C: Number of levels fits targets and uses

Criterion 2: Clarity2A: Levels defined well2B: Levels parallel

18

18


Using the Rubric for Rubrics: GRE Analyze an Argument

GENERAL DESCRIPTIONS:Score 6: A cogent, well-articulated critique of the argument and conveys meaning skillfully.Score 5: A generally thoughtful, well-developed critique of the argument and conveys meaning clearly.Score 4: A competent critique of the argument and conveys meaning adequately.Score 3: Some competence in its analysis of the issue and in conveying meaning but is obviously flawed.Score 2: Serious weaknesses in analytical writing.Score 1: Fundamental deficiencies in analytical writing.

19

19


SummaryChoose the appropriate scoring method based on the purposeLook for rubrics made elsewhere and adapt as necessaryYour criteria should reflect your learning objectives

And should be weighted appropriatelyCriteria should be clearly defined and parallel across levelsCheck for double-penalty criteriaNumber of levels should be reasonably distinguishableCheck that scores reflect knowledge appropriatelyDevelop the rubric as you write the taskPhysically organize the rubric to be efficient and clearTake it for a dry run

20

20


ReferencesArter, J., & Chappuis, J. (2006). Creating & recognizing quality rubrics. Educational Testing Service. Upper Saddle River, NJ: Pearson.Gronlund, N. E., & Waugh, C. K. (2009). Assessment of student achievement. (9th ed.). Upper Saddle River, NJ: Pearson.McMillan, J. H. (2007). Classroom assessment: Principles and practice for effective standards-based instruction (4th ed.). Boston: Allyn and BaconReynolds, C. R., Livingston, R., & Wilson, V. (2006). Measurement and assessment in education. Upper Saddle River, NJ: Prentice Hall. Stevens, D. D., & Levi, J. A. (2005). Introduction to rubrics. Sterling, VA: Stylus Publishing, LLC.

21

21

rubrics - university of arizonaassessment.arizona.edu/sites/default/files/rubrics.pdf · types of...

Documents