Rubrics and Other Scoring Methods

Sue Brookhart

January 26, 2015

Introductions

• Sue Brookhart, Ph.D.

• Juliette Lyons-Thomas, Ph.D. (Fellow, Regents Research Fund)

2

Webinar Norms

• All phones will be placed on mute• If you have a question, you can type into the

chat box, and your question will be addressed during a break

The chat box icon is located at the top right hand corner of your screen (remember to direct your chat to “Everyone”)

• At the end of the webinar, you will be asked to fill out a survey based on your experience today

3

Learning Outcomes

• Participants will learn to create (or select and adapt rubrics) that match intended learning outcomes.

• Participants will learn to check for consistency among scorers.

• Participants will learn to use checklists, rating scales, and point schemes for special purposes.

4

Performance Assessment

• Task(s) or assignment must tap the student learning outcome

Knowledge Skills Level of Thinking

• Rubrics or other scoring scheme must accurately describe student performance along a continuum of quality

5

Criteria

• Important for all scoring methods Rubrics, Checklists, Rating Scales, Point Schemes

• These are what connect student performance to the learning outcomes they indicate

6

RubricsCriteria and Performance Level Descriptions

Criteria: What qualities need to be in the work?

Performance level descriptions: How well does the work exhibit these qualities?

7

Example – Choral Music Rubric

  A B C D FAccuracy Always on pitch

Rhythms correctOnly a few incorrect notesMostly accurate rhythm, stable beat

Usually on pitchSome minor rhythm problems, generally a steady beat

Rarely on pitch, many incorrect notesSome rhythmic errors and changes in beat

Not on pitchMany rhythmic errors and changes in beat

Text Fully memorizedAlways clear with precise sounding consonants

Mostly memorizedMostly clear with attempted ending consonants

Some memorizationUsually clear but missing consonants

Little memorizationUnclear or mumbled

No evidence of memorizationNot clear

Part Work Knows all partsWell balanced sound (both parts heard equally)

Knows most partsMostly balanced sound

One part not well knownAdequate balance between parts

Both parts not well knownParts not balanced

Parts mostly unknownNot able to hear two parts

Tone Quality Clear and projectedEven tone between head & chest voiceConsistent and effective breath support

Mostly clear & projectedSome difference in tone between head & chest voiceBreath support effective most of the time

Beginning to be clear & projectedMuch difference in tone between head & chest voiceMore breath support needed 

Unclear and unsupported (lack of projection), or a harsh tone in many sections

Weak (no projection)Tone harsh, unsupported, and/or uncontrolled

Interpretation Consistent and correct use of dynamics and expression

Effective use of dynamics and expression

Some dynamic variationAdequate expression

Very few dynamicsLittle expression

No dynamicsNo expression

8

A Counter-Example - What happenswhen bad rubrics are applied to good work?

9

Student A – 7th grade inclusion

10

11

Student B – 7th grade inclusion

12

13

Sometimes rubrics can be very concrete

• Example – third grade• Concepts of Multiplication poster• Present a multiplication problem, a repeated

addition problem, a picture with equal groups, and an array for the same multiplication fact

• Poster

14

15

Example

16

17

Example

18

More Complex Rubrics forMore Complex Tasks

• Several criteria Analytic rubrics – criteria considered one at a

time Holistic rubrics – criteria considered

simultaneously• Criteria should be qualities that work should

exhibit (e.g., position is supported, steps are explained with mathematical vocabulary, cites appropriate sources)

• Criteria should NOT be about directions (e.g., has 4 sentences, cites 3 sources)

19

Example

20

2 1 0

Understanding the Problem

Complete understanding of the problem

Part of the problem misunderstood or misinterpreted

Complete misunderstanding of the problem

Planning a Solution

Plan could have led to a correct solution if implemented properly

Partially correct plan based on part of the problem being interpreted correctly

No attempt, or totally inappropriate plan

Getting an Answer

Correct answer and correct label for the answer

Copying error, computational error, partial answer for a problem with multiple answers

No answer, or wrong answer based on an inappropriate plan

(NCTM, 1987, 1991)

Example

Jan's Snack Shop has 3 flavors of ice cream—vanilla, chocolate, and strawberry. The ice cream can be served in a dish, a sugar cone, or a regular cone.

There are 9 people who choose 1 dip of ice cream in a dish, or in a sugar cone, or in a regular cone, and all of their choices are different. List or show the 9 different choices.

Could another person have a choice that is different from one of these 9 choices? Why or why not?

(NAEP 2003, grade 4,

Block M10 #17)

21

Example to Score

22

Example to Score

23

Example to Score

24

Consider a Template

4 Task is complete and correct, plus it includes an extension, exceptional insight, or some other feature that is “above and beyond” what was required or intended.

3 Task is complete and correct. Response shows understanding of intended learning outcome.

2 Task is partly complete and correct. Response shows some understanding of intended learning outcome.

1 Task is not complete or correct. Response shows little/no understanding of intended learning outcome.

25

NOTE: DON’T use this “rubric” as is – it’s a template. It lacks CRITERIA. Replace the “boilerplate” language with criteria specific to your learning outcome of interest.

Example – 6th ELA – Writing a well-structured paragraph in response to a prompt

• Direct instruction and practice, four weeks• Learning outcome = can write a paragraph that

Answers the prompt question Has a focus/topic sentence Has detail sentences that relate to the focus Has an effective ending sentence

26

Example – 6th grade writing

4 Task has a focus sentence, details, and conclusion, plus it includes a particularly compelling response to the prompt, for example with especially vivid details.

3 Task is has a focus sentence, details, and conclusion.

2 Task is has some attempt at a focus sentence, details, and conclusion.

1 Task does not have a focus sentence, details, and conclusion.

27

Example – 6th grade writing

28

Example – 6th grade writing

29

For Simple Tasks

3 Task is complete and correct. Response shows understanding of intended learning outcome.

2 Task is partly complete and correct. Response shows some understanding of intended learning outcome.

1 Task is not complete or correct. Response shows little/no understanding of intended learning outcome.

30

NOTE: DON’T use this “rubric” as is – it’s a template. It lacks CRITERIA. Replace the “boilerplate” language with criteria specific to your learning outcome of interest.

Kindergarten ExampleMaking Equal Sets

3 Elements of each set are organized in a single line. There is a one-to-one correspondence between elements of each set.

2 There is an attempt at organizing elements of each set and showing correspondence between elements of each set. These attempts are not entirely successful.

1 Elements of each set are not organized. No correspondence is shown between elements of each set.

31

Kindergarten ExampleMaking Equal Sets

32

A B

Ensuring Scoring Consistency

• Use a clear scoring guide (e.g., rubric)• Score one question/task at a time• Score content separately from other factors

(e.g., grammar, spelling)• Score questions/tasks anonymously

33

Ensuring Scoring Consistency

• Select a few papers (questions, tasks) from different classes

• Double-score this set of papers (questions, tasks)

34

A Tale of 8th Grade ELA Portfolios

35

Checklists

• Most useful for self or peer assessment of following directions for the assignment

• Useful for basic skills (e.g., capital and period for each sentence) or concrete behaviors (e.g., checklist for washing hands) and procedures (e.g., making a wet mount slide).

36

Example – 3rd grade learning support

37

38

Example – Checklist for preparing a wet mount slide

Assemble materials (slides, cover slips, pipet, sample and any solutions or chemicals)

Place a drop of the sample or staining solution in the center of a clean slide

Add cells to be viewed under the microscope (unless they are already in the sample)

Angle cover slip over the sample with one edge touching the slide

Lower cover slip gently, to avoid bubbles in the solution Remove any excess solution by touching a paper towel to the

edge of the cover slip Place slide on the stage of a microscope and view at low power

to check slide contents

39

Checklists and Scoring

• Typically, checklists are NOT scored, but used for feedback.

• When a score is needed for a procedure that can be governed by a checklist, it is possible to use the checklist with 1-0 scoring (check = 1 point, no check = 0 points).

• Note that using a checklist assumes that the only measure of “quality” is that something was done – not how well – and this is why rubrics are usually preferred for scoring.

40

Rating Scales

• Rating scales are useful for behavior, learning skills, and citizenship.

Frequency scales (e.g., “always, frequently, sometimes, never”) are usually better than evaluative scales (e.g., “excellent, good, fair, poor”)

• Rating scales are useful for measuring attitudes, interests, and other survey-type questions.

• Not recommended for scoring student school work – can be useful for self-assessment

41

Example

High school

Chemistry

42

Point Schemes

• Best used to score brief, focused constructed response test questions

Essay questions Show-the-work math problems

• Need a new point scheme for every question• Cannot share with students

43

Example

Fill in the chart below with the name of each of the three branches of government and the main purpose of each branch.

Total possible points = 6

1 point each for naming Executive, Legislative, & Judicial

1 point each for main purpose, at minimum: Enforces laws (Exec), Makes laws (Legislative), Interprets laws (Judicial)

44

Brand of Government Main Purpose

Example[NAEP, Grade 8, 2005; Block S13 #10]

Some scientists think that the Earth's climate is getting warmer. If these scientists are correct and the Earth keeps getting warmer for the next 50 years, what will happen to the oceans? Explain why this would happen. [space given for answer]

If these scientists are correct, what things about the Earth's weather will change? Explain why this would happen. [space given for answer] 

If these scientists are correct, what will happen to plants? Explain why this would happen. [space given for answer] 

Scoring: Response consists of six correct parts: a prediction and an explanation for each of oceans, weather, and plants.

45

Summary

• Use rubrics to match LEARNING OUTCOMES, not directions.

• Use other scoring

methods for special

purposes.• Check consistency for

any scoring that has consequences for

students, teachers, or schools

46

Thank You

• The slides and a video of this webinar will be posted at https://www.engageny.org/resource/teaching-core-assessment-literacy-series-materials

• Please provide feedback on your webinar experience, or suggestions for future topics, by filling out this survey: https://www.surveymonkey.com/s/rubricswebinar

47