Effective grading (and teaching) through the

use of rubrics & use of rubrics for

assessment

Sarah Murnen

Kenyon College

Personal Background Although a psychologist, no

particular background in this topic As Kenyon’s “Assessment

Coordinator” from 2002-2006 I learned that: Some faculty resent the assessment

process and see it as extra burden Departments with faculty who feel

burdened by assessment don’t get much out of it

Departments that find a way to integrate assessment with what they already do benefit from the process – they discuss what students are learning and change their teaching methods, assignments, and sometimes their entire curricula to help students learn

Benefits of Ohio-5 Project

To help improve student learning (and faculty teaching) by encouraging the use of rubrics (Many other benefits to the use of rubrics)

To specifically encourage a focus on creativity and critical thinking, important aspects of a liberal education

To use rubrics for assessing critical thinking and creativity

To show faculty how all of the work they put into grading can be used for assessment purposes

Background Information Presentation by Dr. Douglas Eder on

“Primary Trait Analysis” Presentation by Barbara Walvoord

on the use of rubrics for effective grading

Walvoord, B. E., P& Anderson, J. A. (1998). Effective Grading: A tool for learning and assessment. Jossey-Bass: San Francisco, CA.

Stevens & Levi (2005). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Stylus Publishing: Sterling, Virginia.

Our primary concern is student learning

Good practice in undergraduate education (Walvoord, 1998, p. 15): Encourages student-faculty contact Encourages cooperation among students Encourages active learning Gives prompt feedback Emphasizes the time that students devote

to the task Communicates high expectations Respects diverse talents and ways of

learning

Walvoord asks (1998, p. 15):

“How many of these principles of good practice in some way involve the grading system in your class, the tests and assignments on which that system is based, and your ways of communicating with students about their work and their grades?”

Walvoord’s Argument for Establishing Clear Criteria and Standards for Grading

Saves time in grading process Allows you to make the process consistent and fair Helps you explain to students what you expect Shows you what to teach Identifies essential relationship between discipline

information and processes Help students evaluate their own and each other’s

work Saves you from having to explain your criteria to

students after they have handed in their work, as a way of justifying the grades they are contesting

Helps student peers give each other constructive feedback on plans and drafts

Helps team teachers or teaching assistants grade student papers consistently

Helps teachers of sequenced courses communicate with each other about standards and criteria

Form the basis for departmental or institutional assessment

How to Establish Clear Criteria: Primary Trait Analysis Developed to score essays on the

National Assessment of Educational Progress by Lloyd-Jones in1977

Creates a scoring rubric that can be used to assess any student performance

Is assignment specific Can be used for grading For this project, we will develop

rubrics using PTA to measure the critical thinking and creativity we are encouraging in our students

Why take the time to do PTA for grading? Makes grading more consistent

and fair Saves time in the grading

process once the rubric is developed

Can diagnose students’ strengths and weaknesses very specifically in order to teach more effectively

Can track changes in student performance

Process of PTA (Eder)

• Identify the “primary traits” – essential or central components of the discipline to be learned by the student

• Build a scale for scoring the student’s performance on the trait

• Evaluate the student’s performance against those criteria

Key Stages in Constructing a Rubric (Stevens & Levi)1. Reflecting -

• What do we want from our students?• Why did we create the assignment?• What happened the last time we used the

assignment? • What are our expectations for the assignment?

2. Listing – Focus on particular details of the assignment and what specific learning objectives we hope to see completed (Sometimes helps to imagine the best and the worst performance on the assignment)

3. Grouping and Labeling the goals together - Organize the results of our reflections in Stages 1 and 2, grouping similar expectations together in what will probably become the dimensions of the rubric

4. Application – Transfer groupings to a rubric grid

Example: Research Article Critique1. Reflecting -

• Want students to learn to be critical evaluators of psychological research

• The research article critique should start them thinking critically, will follow up with class discussion on each critique, and use multiple critiques throughout the semester to help develop their skills

• I expect students will move from description to analysis

2. Listing – • I have seen students move through

Bloom’s taxonomy in a semester using this assignment: Most can show “knowledge” and “comprehension” at the beginning of the course, move to “application,” “analysis,” “synthesis,” and “evaluation” (hopefully). I want to try to capture this process.

3. Grouping and Labeling the goals together – • see the questions that follow

4. Application – Transfer groupings to a rubric grid• See the rubric that follows

Research Article Critique Assignment For each article analysis you are to answer

the following questions: What is the primary question posed by the

study? Is there a hypothesis stated? If so, what is it? What is the theoretical explanation for the

proposed hypothesis? Briefly describe the way the independent and

dependent variable(s) were manipulated or measured

How do the results of the study affect the originally posed hypothesis (or purpose of study)?

Two strengths of the study? Two weaknesses of the study? What is a logical extension of the study?

Briefly describe a study you could conduct to extend the research

Sample Rubric – Designed for AssessmentACC = Accomplished, AVG = average, DEV = developing, BEG = beginning

Understands primary purpose of research

Clearly labels the independent and dependent variable without

extraneous material provided and shows understanding of

theoretical basis of study

ACC AVG DEV BEG

Describes measurement issues clearly

Hones in on how the variables are manipulated/measured without

providing extraneous material

ACC AVG DEV BEG

Understands significance and implication of the results

E.g., does research represent a methodological improvement, novel hypothesis, important test of a theory, etc.

ACC AVG DEV BEG

Chooses relevant and important strengths of study

Discusses strengths that are very relevant to judging the internal validity of the study (or possibly external validity)

ACC AVG DEV BEG

Chooses relevant and important weaknesses of study

Discusses weaknesses that are very relevant to judging the internal validity of the study (or possibly external validity)

ACC AVG DEV BEG

Proposed study an important extension of the research

E.g., methodological improvement, advancement of theory, tests an alternative explanation, etc.

ACC AVG DEV BEG

Creativity

e.g., Unusual association of ideas in responses, proposed study is novel and represents a significant advancement

ACC AVG DEV BEG

Rubric Used for Grading – weight each component

5% Understands primary purpose of research

Clearly labels the independent and dependent variable without

extraneous material provided and shows understanding of

theoretical basis of study

ACC AVG DEV BEG

5% Describes measurement issues clearly

Hones in on how the variables are manipulated/measured without

providing extraneous material

ACC AVG DEV BEG

10% Understands significance and implication of the results

E.g., does research represent a methodological improvement, novel hypothesis, important test of a theory, etc.

ACC AVG DEV BEG

25% Chooses relevant and important strengths of study

Discusses strengths that are very relevant to judging the internal validity of the study (or possibly external validity)

ACC AVG DEV BEG

25% Chooses relevant and important weaknesses of study

Discusses weaknesses that are very relevant to judging the internal validity of the study (or possibly external validity)

ACC AVG DEV BEG

25% Proposed study an important extension of the research

E.g., methodological improvement, advancement of theory, tests an alternative explanation, etc.

ACC AVG DEV BEG

5% Creativity

e.g., Unusual association of ideas in responses, proposed study is novel and represents a significant advancement

ACC AVG DEV BEG

Describing the level of performance on the rubric Anywhere from 2 to 5+ levels of performance

on each trait Terms used to describe level of performance

might be positive and active to encourage motivation in students (suggestions from Stevens & Levi): Mastery, partial mastery, progressing,

emerging high level, middle level, beginning level Sophisticated, competent, partly competent,

not yet competent Exemplary, proficient, marginal, unacceptable Advanced, intermediate high, intermediate,

novice Distinguished, proficient, intermediate, novice Accomplished, average, developing,

beginning

Using Rubrics for Assessment

We all need to attend to the issue of assessment, and Walvoord argues that we can make use of what we already do in the grading process

American Association for Higher Education’s “Principles of Good Practice for Assessing Student Learning”

Answer questions that people care about

Lead directly to improvement in teaching and learning

Be embedded in the context of learning

Take place repeatedly over time

The rubrics we use for grading can be used for assessment

“If we as faculty do not make our learning goals, tests, criteria, and standards explicit and understandable to legislatures, boards, accrediting agencies, and other audiences in ways that meet their needs and concerns, we face the very real possibility that some of the control we currently exercise in the classrooms will be taken away from us. We must deal with assessment, but we need not construct a parallel assessment structure that ignores the assessment we already conduct.” (Walvoord, 1998, p. 5).

How to turn this into assessment

Paper 1 Paper 2 Paper 3 TotalProf ACreative1 4 1 2 7Creative2 4 1 2 7Critical1 3 1 3 7Critical2 2 1 3 6 Prof BCreative1 3 2 2 7Creative2 4 1 3 8Critical1 3 1 3 7Critical2 2 1 3 6

Prof CCreative1 4 1 2 7Creative2 3 2 3 8Critical1 2 2 3 7Critical2 2 1 3 6

Prof DCreative1 4 2 3 9Creative2 4 2 3 9Critical1 3 2 3 8Critical2 3 2 4 9

Total 50 23 45

How have faculty been influenced by this process? “I was able to clarify my expectations for the course by

articulating the purpose behind each graded exercise. This helped dramatically in easing anxieties about performance and final grades. Students seemed to appreciate understanding the logic behind the assignments and seeing that each focused on building and assessing a particular skill, rather than merely providing me with another grade.”

“Best of all, I’m looking forward to sharing the process of developing PTA-based rubrics with my students. Specifically, students will work in groups to identify their own PTA-based critical thinking rubrics before they embark on writing a particular essay. I can hardly think of a better way to develop the students’ meta-cognition., i.e., how they think about their own thinking!”

“Yes, we’ve been disappointed with some aspects of student performance (for example synthetic ability) and using the rubric has helped us communicate better with students, and we’re thinking about ways to improve courses to help teach the skills needed to succeed on this senior exercise objective.”

“I have learned a great deal, and I think that my teaching has been positively influenced by this important work. I can also say that my research and scholarship have been impacted by these foci, as well.”

Go to Website for Rubric Samples:

http://www.wooster.edu/teagle/default.html