DESIGNING RUBRICS

Assessment Certificate Program

Workshop at DePaul University

March 25, 2015

By Ranfen Li, PhD

Acknowledgement

• The content of this presentation is based on

the following sources: • Myford, C. (Spring 2006), Performance Assessment, a

PowerPoint presentation

• Arter, J. and McTighe, J. (2001). Scoring rubrics in the classroom, R. Guskey & R.J. Marzano, Eds., Thousand Oaks, CA: Corwin Press

• Airasian, P and Russell, M.K（2008). Classroom assessment: concepts and application, New York, NY: McGraw-Hill

Objectives for this workshop

• Participants will be able to:

• Define what is a rubric and explain its benefits

• Identify situations in which rubrics are useful

• Gain knowledge of several types of rubrics

• Obtain skills in designing rubrics

Assessment

Gather information

• Testing – a formal, systematic procedure (assessment method)

Synthesize information

• Measurement – quantifying or assigning a number (scoring)

Interpret information

• Evaluation – judging quality or value

Assessment Methods

Major categories Selection and Supply – multiple

choice, short answer, concept map

Product – essay, paper, report, portfolio, thesis

Performance – experiment, musical performance, debate, presentation, behaviors

Scoring Methods

Major types

Assign points based on correct answers for selection or supply test items

Assign points based on a rubric for a performance-or product-based testing

A Rubric

• Is a scoring guide

•Has a set of clear expectations or

criteria, with all level score points

described and defined

Benefits of a rubric

• Increase the consistency of

judgments across teachers

and students, reducing

subjectivity of the

judgments

• Help instructors and

students focus on what is

valued in a subject, topic,

or activity

Relationship between Outcomes,

Assessment & Scoring Methods

Learning Outcomes

Assessment Methods

Scoring Methods

Situations When a Rubric Is Needed

Look for learning outcomes that:

• are skill based

• focus on actual performance of tasks in realistic settings

• would likely result in a product to be evaluated

• focus on students’ affective characteristics (i.e., feelings, values, attitudes, beliefs)

Examples of Outcomes

DePaul University undergraduate music students will be able to demonstrate the ability to perform with an acceptable tone quality, pitch and rhythmic accuracy, dynamic control, articulation, and expressiveness.

(source: one of the learning outcomes of B.M. in Music Performance at

DePaul University)

Examples of Outcomes

Apply nursing concepts, theories, and research to a family-centered and community based practice of professional nursing care and case management

(source: one of the learning outcomes of B.S. Registered Nurse at

DePaul University)

Examples of Outcomes

Students will be able to write analytical essays presenting an argument about one or more literary works and supporting it with appropriate evidence.

(source: one of the learning outcomes of B.A. in English at

DePaul University)

Examples of Outcomes

Students who engage with crisis management services at UCS will gain a new viewpoint on their problems and engage in perspective taking, thereby reducing potential for self-harm and self-sabotaging behaviors.

(source: an activity outcome of University Counseling Services at

DePaul University)

Introducing Rubrics and Other

Scoring Types

Rubrics Holistic vs. Analytic

General vs. Task Specific

Developmental

Other Types

Checklist

Rating Scales

Holistic Rubric

• A single rubric that

incorporates all the

performance criteria

• give a single score or

rating for an entire

product or performance

• rating is based on an

overall impression of a

student’s work

• Used for summative

assessment

Analytic Rubric

• Devise a separate rubric

for each performance

criterion and judge each

individually

• give a separate score

for each criterion

• Reveal a profile of

strengths and

weaknesses

• Used for formative

assessment

Example for a Holistic Rubric

Outcome: Demonstrate the ability to perform with an

acceptable tone quality, pitch and rhythmic accuracy,

dynamic control, articulation, and expressiveness.

Level 3 - Acceptable

Demonstrate excellent performance in all of the following: tone quality, pitch and rhythmic accuracy, dynamic control, articulation, and expressiveness

Level 2 - Partially Acceptable

Demonstrate good performance in most of the following: tone quality, pitch and rhythmic accuracy, dynamic control, articulation, and expressiveness

Level 1 - Not acceptable

Show poor performance in most of the following: tone quality, pitch and rhythmic accuracy, dynamic control, articulation, and expressiveness

Holistic Rubric - Advantages

• Useful for getting a “quick snapshot”

of overall quality or achievement

• More cost efficient; can be carried out

relatively quickly to score large

numbers of responses

• Useful when a single score is needed

Holistic Rubric - Disadvantages

• No detailed analysis of strengths and

weaknesses of a product or

performance

• Don’t provide students with detailed

feedback to guide their improvement

• Two students can receive the same

score for different reasons

Example for an Analytic Rubric

Outcome: Demonstrate the ability to perform with an

acceptable tone quality, pitch and rhythmic accuracy,

dynamic control, articulation, and expressiveness. Traits Not Acceptable

(Level 1)

Partially

Acceptable

(Level 2)

Acceptable

(Level 3)

Tone Quality Description for

level 1

Description for

level 2

Description for

level 3

Pitch/Rhythmic

Accuracy

Description for

level 1

Description for

level 2

Description for

level 3

Dynamic Control Description for

level 1

Description for

level 2

Description for

level 3

Articulation Description for

level 1

Description for

level 2

Description for

level 3

Expressiveness Description for

level 1

Description for

level 2

Description for

level 3

Analytic Rubric - Advantages

• Useful for judging complex performances or products involving several significant dimensions

• Provide more specific information or feedback to students about strengths and weaknesses of a performance or product

• Help students to better understand the nature of quality work

Analytic Rubric - Disadvantages

• Often slower to score (requires

giving ratings on multiple

dimensions)

• May take longer to learn to apply

• May be more expensive to use; not

as good a choice for large-scale

assessments

Generic Rubric

• Can be used for multiple tasks across similar performances or products. Useful when:

• skills being assessed are complex and generalize across tasks

• the goal is to help students see what quality looks like across similar tasks

• students will not all be doing exactly the same task

Task-specific Rubric

• Can only be used for a single task. Each task has its own rubric. Useful when you want to know whether students have learned:

• how to carry out a specific task, perform a particular skill, or produce a particular product

• particular facts, equations, methods, or procedures

Developmental Rubric

• Used to monitor development of a process, product, behavior, etc.

• Useful when the goal of evaluation is to determine level of development

• Rubric can be based on relevant developmental theory

• Usually used several times to examine changes over a period of time

Writing Rubrics

Steps Identify performance criteria

Determine the number of scoring points

Write performance level descriptions

Identify Performance Criteria

• List important aspects (elements, behaviors, components, qualities, features, characteristics) of the performance (or product) students will demonstrate. Ask yourself:

• What do you want your students to demonstrate from carrying out this process or producing this product?

• What would a high quality product look like? What are its essential, defining characteristics or features?

Determine the Number of Score Points

• Ask yourself, “How many points do I need to

adequately describe the range of performance I am

seeing in student work?”

• “Normal” range of points is 3 to 7

• If you have samples of student work (e.g., essay),

sort the samples into three or four groups based on

level of quality (e.g., well written, fair, poor).

- Write down reasons for placing pieces into piles.

- Cluster the reasons into important dimensions of

performance.

Write Performance Level Descriptions

• Start by describing the highest and lowest levels of quality, and then fill in the middle levels.

• Use clear language, defining terms for student-friendly language.

• Avoid making the rubrics too lengthy and laden with details.

Checklist

• Is a written list of performance criteria

• Scorer makes two-category judgment of each

performance criteria included in the checklist

• Examples of Two-Category Judgments

• Yes/No

• Present/Absent

• Performed/Not Performed

• Met/Not Met

• Observed/Not Observed

Example for Checklist

Outcome: Students will be able to design and carry out

chemical experiments, record data and analyze results.

• Direction: The instructor would place a checkmark before

each criterion that was met (and leave blank those criteria

that were not met).

_____ Design a chemical experiment

_____ Conduct the experiment

_____ Record experiment data

_____ Analyze the data

_____ Interpret the results

(source: one of the learning outcomes of B.S. in Chemistry at DePaul University)

Alternative Format for Checklist

Tasks Performed Not performed

Design a chemical

experiment Conduct the

experiment Record experiment

data Analyze the data

Interpret the results

• There are other formats that will work, as well.

• Use whatever format that seems to work.

Rating Scales

• Allow the observer to judge

performance along a continuum rather

than as a dichotomy

• Three commonly used types

• Numerical

• Graphic

• Descriptive graphic

Characteristics of Numerical Rating

Scales

• Each number signifies a point on the scale.

• The teacher circles a number to make a

judgment about student performance with

respect to that performance criterion.

• Each number has a verbal description that

remains constant from one performance

criterion to the next.

Example of Numerical Rating Scales

• Direction: Indicate the degree to which this student carries out

chemical experiments by circling the appropriate number. The

number represents the following rules: 3 = performs very well, 2

= performs well, and 1 = performs poorly.

Carry out chemical experiments

1 2 3

Characteristics of Graphic Rating

Scales • Each performance criterion is followed by a

horizontal line.

• The teacher makes a check on the line at

the point that best describes the student’s

product (or performance).

• A set of categories identifies specific

positions along the line, but the teacher is

free to check between these points.

Example of Graphic Rating Scales

• Direction: Indicate the degree to which this student carries out

chemical experiments by placing an X anywhere along the horizontal

line under the criterion.

To what extent does the student carry out chemical experiments?

Very well Well Poorly

Characteristics of Descriptive

Graphic Rating Scales

• Use descriptive phrases to identify the points

on a graphic rating scale.

• The descriptions are thumbnail sketches of

how students behave at different points along

the continuum (or how the quality of a

product differs).

• In some scales, only the center and end

points are defined. In others, a descriptive

phrase will appear beneath each point.

Example of Descriptive Graphic

Rating Scales • Direction: Mark your rating by placing an X anywhere along the

horizontal line under the criterion.

To what extent does the student follow the safety

procedures while carrying out chemical experiments?

Consistently follows

Generally follows but needs

improvement

Sometimes follows and needs

improvement Never follows

Time to practice

1. Write one learning outcome that would result in

a product or a performance based assessment

2. List a few observable performance criteria that

you will use to evaluate the students’

performances or products

3. Decide what type of an assessment rubric or a

rating scale you will build

4. Determine the number of scoring points

5. Sketch out descriptions of different scoring

levels