clarifying fuzziness: writing rubrics for direct and ...apps.naspa.org/cfp/uploads/naspa rubric...

12
3/15/2017 1 Clarifying Fuzziness: Writing Rubrics for Direct and Indirect Assessment 2017 NASPA Annual Conference March 13, 2017 – San Antonio Dr. Erica Eckert Kent State University Dr. Brenda McKenzie Vanderbilt University (C) E. Eckert & B. McKenzie, 2017 Your Presenters ASSESSMENT, TECHNOLOGY Dr. Erica Eckert Assistant Dean for Assessment and Accreditation, Assistant Professor, Kent State University 5 years in academic assessment and accreditation 5 years in admissions operations and systems/technology STUDENT AFFAIRS, FACULTY Dr. Brenda McKenzie Senior Lecturer, Higher Education Administration, Vanderbilt University 21 Years of experience in Student Affairs (Leadership education, student activities, orientation, residence life) 2 years faculty in graduate preparation programs (C) E. Eckert & B. McKenzie, 2017

Upload: nguyenkien

Post on 12-Apr-2018

221 views

Category:

Documents


3 download

TRANSCRIPT

3/15/2017

1

Clarifying Fuzziness:

Writing Rubrics for Direct and Indirect Assessment2017 NASPA Annual Conference March 13, 2017 – San Antonio

Dr. Erica EckertKent State University

Dr. Brenda McKenzieVanderbilt University

(C) E. Eckert & B. McKenzie, 2017

Your Presenters

ASSESSMENT, TECHNOLOGY

Dr. Erica Eckert Assistant Dean for

Assessment and Accreditation, Assistant Professor, Kent State University

5 years in academic assessment and accreditation

5 years in admissions operations and systems/technology

STUDENT AFFAIRS, FACULTY

Dr. Brenda McKenzie

Senior Lecturer, Higher Education Administration, Vanderbilt University

21 Years of experience in Student Affairs (Leadership education, student activities, orientation, residence life)

2 years faculty in graduate preparation programs

(C) E. Eckert & B. McKenzie, 2017

3/15/2017

2

Our Goals for You

…identify several types of rubrics

…apply the concept of blueprinting to the

organization of checklist and performance-level

rubrics

…evaluate opportunities to deploy rubrics of

varying types

…participants will design descriptive rubrics for a

provided scenario

(C) E. Eckert & B. McKenzie, 2017

Types of Assessment Evidence

DIRECT

Evidence of student

learning is tangible or

visible, self-

explanatory;

demonstrates what

students have and

have not learned

INDIRECT

Evidence consists of

signs that students are

probably learning,

can include reflection

(C) E. Eckert & B. McKenzie, 2017

(Suskie, 2009, p. 20)

3/15/2017

3

Selecting Methods and

Approaches

Criteria for choosing methods

Content of tool

Quality/consistency of items (reliability)

Focus/truth of items (validity)

Feasibility (time, cost)

Motivation

Other Considerations (fairness, format, interpretability,

consequences to participants)

(C) E. Eckert & B. McKenzie, 2017

(Banta & Palomba, 2015, pp. 74-79)

Evaluating Values, Dispositions,

Habits of the Mind

Usually indirectly assessed

These are hard to measure… focus on how they

are expressed

2 Methods: Reflection, Behaviors

Suggested techniques: surveys, minute papers,

before-and-after reflection, journals, interviews,

rating scales… several of which can rely on

rubrics

(C) E. Eckert & B. McKenzie, 2017

(Suskie, 2009, pp. 183-201)

3/15/2017

4

Rubric Basics

Rubric – A scoring guide or grading criteria,

typically a list or chart, crafted to evaluate

assignments

Focus on elements/items you are “looking for”

The items may be accompanied by guidelines as to the

evaluation criteria

“There is no single correct way to write or format rubrics”

(p. 138) assuming one keeps in mind the rubric’s

purpose

(C) E. Eckert & B. McKenzie, 2017

(Suskie, 2009, pp. 137-138)

Activity: Blueprinting

Elements for a Rubric

How I love to evaluate thee, let me think about the ways…

(C) E. Eckert & B. McKenzie, 2017

3/15/2017

5

Formatting Options for Rubrics

Checklist Rubric Yes/No

Rating Scale Rubric Example: Poor, Fair, Good, Very Good

Holistic Rubrics Series of broad statements with levels (one rating)

Descriptive Rubric Performance levels are explained in detail

Structured Observation Guide Evaluation of focused behaviors/factors with text

(C) E. Eckert & B. McKenzie, 2017

(Suskie, 2009, pp. 138-148)

Checklist Rubric

• Contains a list of

tasks

• Evaluator notes

which elements are

present (or not)

• Pros/Cons?

(C) E. Eckert & B. McKenzie, 2017

(Suskie, 2009, pp. 138-139)

3/15/2017

6

Activity: Creating a

Checklist Rubric

Generate 5 checklist items for evaluation

(C) E. Eckert & B. McKenzie, 2017

Rating Scale

Rubric

• Contains a list of

elements or tasks

• Each element or

task is assigned a

rating (e.g., A, B, C,

D, F)

• Pros/Cons?

(C) E. Eckert & B. McKenzie, 2017

(Suskie, 2009, pp. 140-142)

3/15/2017

7

Holistic Rubric

• Complex

descriptions of a

combination of

performances/

behaviors resulting

in a single score

• Pros/Cons

(C) E. Eckert & B. McKenzie, 2017

(Suskie, 2009, pp. 145-146)

Descriptive

Rubric

• Contains a list of

elements or tasks

• Each element or

task has a series of

descriptive

performance levels

• Pros/Cons?

(C) E. Eckert & B. McKenzie, 2017

(Suskie, 2009, pp. 142-145)

3/15/2017

8

Effective Descriptive Rubrics…

Focus on the most important tasks

Are short; long rubrics can be confusing

Use concrete action verbs

Leave room for the student to display creativity

Are task/element oriented (rating scales second)

(C) E. Eckert & B. McKenzie, 2017

(Suskie, 2009, pp. 148-151)

Effective Descriptive Rubrics…

Use 3-5 performance levels

Label each level with names (not just numbers)

Use descriptive statements for performance levels

with meaningful variation

Are piloted before being used

(C) E. Eckert & B. McKenzie, 2017

(Suskie, 2009, pp. 148-151)

3/15/2017

9

Activity: Creating a

Descriptive Rubric

(C) E. Eckert & B. McKenzie, 2017

When would

you use a

checklist rubric?

(C) E. Eckert & B. McKenzie, 2017

3/15/2017

10

When would

you use a

holistic rubric?

(C) E. Eckert & B. McKenzie, 2017

When would

you use a rating

scale rubric?

(C) E. Eckert & B. McKenzie, 2017

3/15/2017

11

When would you

use a descriptive

rubric?

(C) E. Eckert & B. McKenzie, 2017

What makes a

rubric capable of

collecting directevidence?

(C) E. Eckert & B. McKenzie, 2017

3/15/2017

12

References, Questions

Banta, T. W., & Palomba, C. A. (2015). Assessment essentials: Planning, implementing, and improving assessment in higher education (2nd ed.). San Francisco, CA: Jossey-Bass.

Stevens, D.D., & Levi, A. J. (2013). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Sterling, VA: Stylus.

Suskie, L. A. (2009). Assessing student learning: A common sense guide. San Francisco, CA: Jossey-Bass.

(C) E. Eckert & B. McKenzie, 2017