creating descriptive rubrics for educational assessment

Creating Descriptive Rubrics

for Educational Assessment

Ursula Waln, Director of Student Learning Assessment

Central New Mexico Community College

Why Create Rubrics?

To communicate learning expectations to students.

To facilitate fairness and consistency in evaluation of student learning.

To establish a common framework for assessing student learning when using diverse measures.

Choose a LayoutTo Fit Your Purpose

Two Main Types of Descriptive RubricsHolistic Provides

comprehensive descriptions of each level of performance.

Useful for quick and general assessment and feedback.

Descriptions may be organized in columns or rows.

Analytic Breaks out criteria for

distinguishing between levels of performance on each criterion.

Useful for detailed assessment and feedback.

Descriptions are organized in a matrix.

HOLISTIC RUBRIC FOR RUBRICS (in columns)

Highly Effective 3

Useful

2

Rudimentary 1

Focuses on knowledge and skills that confer life-long value through application to novel or complex situations, clearly addressing key criteria that 1) comprise the desired learning outcome, and 2) can be demonstrated through a variety of tasks. Differentiates benchmark levels inherent to development of the competency, thereby facilitating valid assessment of both formative and summative progress. Clearly and concisely communicates the nuances between differing levels of proficiency, and does so in terms of positive evidence.

Clearly addresses key criteria that comprise the desired learning outcome. Organizes performance characteristics within progressive categories that accurately reflect observable stages of development. And, clearly describes observable characteristics that demonstrate each level of performance on each criterion.

States learning outcomes in terms of broad instructional goals and/or specific learning tasks; presents performance characteristics in terms of correctness, quantity, or other grading criteria; constitutes a Likert scale.

HOLISTIC RUBRIC FOR RUBRICS (in rows)

Highly Effective (3)

Focuses on knowledge and skills that confer life-long value through application to novel or complex situations, clearly addressing key criteria that 1) comprise the desired learning outcome, and 2) can be demonstrated through a variety of tasks. Differentiates benchmark levels inherent to development of the competency, thereby facilitating valid assessment of both formative and summative progress. Clearly and concisely communicates the nuances between differing levels of proficiency, and does so in terms of positive evidence.

Useful (2) Clearly addresses key criteria that comprise the desired learning outcome. Organizes performance characteristics within progressive categories that accurately reflect observable stages of development. And, clearly describes observable characteristics that demonstrate each level of performance on each criterion.

Rudimentary (1) States learning outcomes in terms of broad instructional goals and/or specific learning tasks; presents performance characteristics in terms of correctness, quantity, or other grading criteria; constitutes a Likert scale.

ANALYTIC RUBRIC FOR RUBRICS

Criteria

Highly Effective 3

Useful

2

Rudimentary

1

Competency-Based

Focuses on knowledge and skills that confer life-long value through application to novel or complex situations, clearly addressing key criteria that 1) comprise the desired learning outcome, and 2) can be demonstrated through a variety of tasks.

Clearly addresses key criteria that comprise the desired learning outcome.

States learning outcomes in terms of broad instructional goals and/or specific learning tasks.

Representative

Differentiates benchmark levels inherent to development of the competency, thereby facilitating valid assessment of both formative and summative progress.

Organizes performance characteristics within progressive categories that accurately reflect observable stages of development.

Presents performance characteristics in terms of correctness, quantity, or other grading criteria.

Descriptive Clearly and concisely communicates the nuances between differing levels of proficiency, and does so in terms of positive evidence.

Clearly describes observable characteristics that demonstrate each level of performance on each criterion.

Constitutes a Likert scale.

Identify the Performance CriteriaWhat Will Be Assessed

First, Focus on Competencies

Whether designing a rubric for a single assignment or for assessing course or program outcomes, focus on competencies, not specific tasks. Competencies involve knowledge and skills that

confer life-long value through application to novel or complex situations.

Tasks are used to collect evidence of achievement of the competency, but a competency-based rubric can be applied to a variety of tasks.

Competencies Apply knowledge of social

systems to analyze contemporary global issues.

Frame problems within differing ethical perspectives.

Evaluate information and sources critically.

Employ contextually appropriate etiquette within various forms of electronic communication.

Tasks Discuss the role of

capitalism in the Chinese construction boom.

Critique an opinion article for political bias.

Correctly identify websites that are suspect choices as sources for academic research.

Write an e-mail to inquire about a job opening.

Examples of Competencies vs. Tasks

Design with Context In Mind

Focus on competencies that align to pertinent: Standards (educational, industry, program) Student learning outcome statements (program,

course) Instructional goals Learning objectives

Next, Derive Criteria from CompetenciesCompetencies Broad descriptions of

desired learning outcomes.

What you want the student to be able to do upon completion of the lesson, course, or program.

Criteria Standards on which

judgments about level of competency can be made.

Characterizing traits (characteristics) of the competencies.

What is needed to demonstrate the competencies.

From Competencies to Criteria

Some competencies are straight-forward enough to be used as criteria in rubrics. For example: Competency:

The student will be able to… Apply the commutative property in computation.

Criterion: To demonstrate application of the commutative

property, the student needs to… Apply the commutative property in computation.

Some competencies are overarching and/or complex and need to be broken down for analytic evaluation. For example: Competency:

The teacher will be able to… Create effective rubrics.

Criteria: To be effective, the teacher’s rubrics need to be…

Competency-based

Representative

Descriptive

Break Down Compound Competencies This competency represents a combination of

abilities: Produce audience appropriate communication that

displays consideration of ethical principles and diverse points of view.

Compound competencies are more easily evaluated when broken down into component criteria. Using the above example: Audience appropriateness Consideration of ethical principles Consideration of diverse points of view

Consider Differences in Manifestation

Some competencies call for synthesis of several related manifestations of learning. For example: The competency “Employ contextually appropriate

etiquette within various forms of electronic communication” involves:

SKILLS/ABILITIES related to language and register-based code-switching.

KNOWLEDGE/AWARENESS of etiquette associated with different contexts.

USES/APPLICATIONS of electronic communication media.

BEHAVIORS/DISPOSITIONS related to courtesy and respect.

Evaluation can be facilitated by breaking different manifestations of learning out into separate criteria. For example: Competency (from previous slide):

The student will be able to… Employ contextually appropriate etiquette within

various forms of electronic communication.

Criteria: To show proficiency, the student needs to

demonstrate… Fluency in code-switching

Familiarity with contextual expectations

Application of electronic media

Courtesy

Can One Part Be Done Without the Other?

“Demonstrate leadership and integrity.” Yes! We can find examples of strong leaders with little

integrity and people of high integrity with very poor leadership skills. Therefore, for analytic evaluation purposes, these should probably be separate criteria:

Demonstrate leadership.

Demonstrate integrity.

“Analyze and evaluate college-level texts.” This one is not so clear. Evaluation requires analysis,

and effective analysis requires a modicum of evaluative skill. It might be best to leave these together unless you intend to evaluate them as separate skills.

Is the Proficiency Multi-Faceted?

“Evaluate information and sources critically.” Yes, this consists of two separate abilities:

Evaluate information critically.

Evaluate sources critically.

And, each of these could be further broken down: Evaluate information critically → recognize manipulative

rhetorical devices, recognize bias, assess contradictory information, identify failures in logic, analyze alternative points of view, etc.

Evaluate sources critically → analyze context, infer motive, assess credibility, etc.

How Far Should You Break Things Down? Consider the purpose of the rubric.

Program-level assessment uses broadly defined criteria.

Course-level assessment uses more specific criteria.

Consider performance evaluation. Break apart criteria if you would use separate

measures to evaluate the components.

Consider complexity. A component that has nuanced and/or

distinguishable performance levels probably needs to be separated out.

How Far Is Too Far?

Rubrics usually have more than two performance levels, so use ‘all or nothing’ as a litmus test for too much specificity. For example: “Differentiate between collective and individualistic

societies.” If a student either is or is not able to identify a basic

difference (all or nothing), consider using a broader criterion, such as “Describe societal differences that contribute to diverse world views.”

On the other hand, if a student might describe a variety of characteristic differences and do so with nuanced degrees of accuracy, then the criterion should probably stand alone.

Phrasing of Criteria

Criteria are usually written as phrases and are formatted according to personal taste. Some examples of different formats:

Employs code-switching fluently

Employ code-switching fluently

Fluent code-switching

Code-switching fluency

Fluent in code-switching

For rubric clarity and cohesion, pick a format and be consistent.

Create the Headings for a MatrixAnalytic Rubrics

Criteria & Performance Levels

In analytic rubrics, the convention is to use the left-hand column for listing the criteria. You can leave the top cell blank or enter a

heading, such as “Criteria.”

The tops of the remaining columns hold the headings for the various performance levels.

Level Heading

1

Level Heading

2

Level Heading

3Criterion 1Criterion 2Criterion 3Criterion 4

Basic Layout

Determine How Many Columns You Need How many performance levels do you want to

describe? The usual is 3 to 5, but some advise using an even

number (to avoid a “central” tendency in scoring).

Do you want to include an exceptional level? Do you want to include a scoring column?

Helpful if the rubric will be used as a scoring instrument.

Do you want to include unlabeled columns with numerical values to allow for in-between scoring?

Level Heading

1 2

Level Heading

3 4

Level Heading

5

Exceptional Level Heading

6

Score

Criterion 1

Description Cr. 1 Lvl. 1




Criterion 2





Criterion 3





Criterion 4





Sample Layout with Unlabeled and Scoring Columns

Choose Ascending or Descending Order The choice of whether to arrange the levels

in ascending or descending order is a matter of preference. Does one order make more sense to you than the

other? Does your topic suggest a certain organizational

logic? Are your students accustomed to a certain

layout? Does your school or program have a preferred

model?

Exceptional Level Heading

6

Level Heading

5 4

Level Heading

3 2

Level Heading

1

Score

Criterion 1





Criterion 2





Criterion 3





Criterion 4





Sample Layout with Levels in Descending Order

Compose Performance-Level Headings Performance-level headings are usually one

or two words each, carefully selected to represent the developing levels of proficiency.

If the rubric is to be used as a scoring instrument, include a numeric score value with each heading.

You can decide upon headings early on or after writing the descriptions. Sometimes it helps to write the descriptions first. If you want to wait, just enter numbers as place

holders.

Sample Headings Beginner, Amateur, Pre-Professional, Professional Beginning, Emerging, Developing, Proficient, Exemplary Below Expectations, Satisfactory, Exemplary Benchmark, Milestone, Capstone Level 1, Level 2, Level 3, Level 4 Needs Improvement, Acceptable, Exceeds Expectations Needs Work, Acceptable, Excellent Neophyte, Learner, Artist Novice, Apprentice, Expert Novice, Intermediate, Advanced Rudimentary, Skilled, Accomplished Undeveloped, Developing, Developed, Advanced

Emerging1 2

Developing3 4

Proficient5

Exemplary6

Score

Criterion 1

Criterion 2

Criterion 3

Criterion 4

Example of Headings with Associated Values

Write DescriptionsIn the Intersecting Cells

Describe the Target Outcome

Start by describing the highest level of performance expected. Describe any level that exceeds expectations last.

Descriptions should focus on what is observable. What does it look like (how can you tell) when the

student has achieved the desired learning outcome? Try to capture the essence of the outcome, the

important manifestations. Use clear and concise language. It is okay to use sentence fragments, so long as the

meaning is clear.

What Is Observable?

Internal processes, such as understanding, appreciating, valuing, respecting, being curious, and being interested are not readily observable.

So, we have to identify behaviors/actions that evince those processes, such as explaining, analyzing, classifying, applying, appraising, defending, integrating, transforming, etc. Avoid putting the word demonstrates in front of an

internal process to make it observable (e.g., demonstrates understanding or demonstrates appreciation) unless you truly need a vaguely worded rubric for broad application and broad interpretation.

Describe the Other Levels

Seek to describe performance breakthroughs or thresholds that logically distinguish the levels of achievement.

If you include an exceptional level (“Exceeds Expectations,” “Outstanding,” etc.), make it clear the proficiency goes well above and beyond the learning expectations. Scoring in this column should be rare; otherwise,

you probably need to adjust your learning expectations.

Use Cumulative or Distinct Descriptions

Cumulative Build upon positive

indicators of achievement. What the student does

do versus does not do.

Sophistication of indicators increases with proficiency.

Lower-level abilities are assumed to be present within the more advanced levels.

Distinct Each level is described in

full. Lower levels typically

include negative as well as positive indicators.

Tend to repeat and add to lower-level positive indicators.

Can be tricky to write well because saying a student does not do something is fairly absolute and tends to restrict interpretation.

Sample Criterion: Discuss Design Concepts

Cumulative Descriptors Level 1: Identifies

fundamental design concepts.

Level 2: Applies design concepts to describe works.

Level 3: Applies design concepts to effectively critique works.

Distinct Descriptors Level 1: Identifies but does

not apply design concepts. Level 2: Identifies design

concepts and applies them to describe works but does not effectively apply them in critiques of works.

Level 3: Identifies design concepts and applies them to describe and effectively critique works.

Allow the Creative Process to Unfold

Revise Some people can clearly envision the whole and

implement it without modification, but most of us adjust a little here and there as we see the totality begin to emerge more clearly.

Entertain Ideas The process of creating rubrics often helps

teachers become much more clear in their own minds about what they want their students to learn, which can stimulate new ideas about how to promote that learning.

Rubric for Rubric Component AnalysisGood Even Better For Greatest Effect

Layout Layout presents or implies a matrix with more than one performance level.

Visual organization forms a clear basis for either holistic or analytic evaluation.

Choice of layout is appropriate to the intended use and audience.

Performance Criteria

Criteria clarify key aspects of proficiency.

Criteria are distinct, scalable, and measurable.

Criteria are clearly worded and consistently formatted.

Headings Headings represent performance levels, arranged in either ascending or descending order.

Headings accurately reflect developmental processes associated with the desired learning.

Word choices are appropriate to the context in which the rubric will be used.

Descriptions

Descriptions characterize proficiency at successive stages of development.

Higher-level descriptions relate breakthroughs in proficiency development.

Descriptions emphasize positive indicators of proficiency.

creating descriptive rubrics for educational assessment

Education

performance criteria

performance characteristics

levels of performance

desired learning outcome

rubrics criteria

key criteria

learning expectations

level of performance