Using Rubrics/Scoring Guides in Program Assessment
Carrie Zelna, Ph.D. Director, Office of Assessment Associate Vice Provost Division of Academic and Student Affairs Stephany Dunstan, Ph.D. Associate Director, Office of Assessment Division of Academic and Student Affairs
Outcomes
• Identify possible opportunities for using rubric/scoring guides as an assessment tool at the student level, class/course level, or program level
• Describe the types of rubrics/scoring guides that can be applied to student, class/course, or program assessment
• Identify the steps necessary to apply rubrics/scoring guides systematically (norming, sampling, analysis)
Levels of assessment
• Individual/student level
• Class/course level
• Program/Curriculum level
Four Steps of Assessment
Four Steps of Assessment Linda Suskie 2009 Assessing Student Learning: A common sense guide. 2nd edition. Jossey-Bass.
1. Establish Learning Goals (Plan)
2. Providing Learning Opportunities (Act)
3. Assess Student Learning (Observe)
4. Use the results (Reflect)
Linda Suskie email on 4/4/2008 to the Assess listserve: “….understand that assessment is
action research, not experimental research. While it is systematic, action research is context-specific, informal, and designed to inform individual practice. As such, it doesn't have the precision, rigor, or generalizability of experimental research. “
P. S
tein
ke
& C
. Z
eln
a
Rubric: Definition and Purpose
• Rubric: “a scoring tool that lays out the specific expectations for an assignment” (Stevens & Levi, 2005, p. 3)
• It is a way of organizing criteria to systematically determine if the outcome is
met based on data gathered through papers, observation, document analysis, or some other appropriate method.
• When you review the data in the aggregate, a rubric can help identify
patterns of strengths and weaknesses that might allow for enhancements to the program.
Constructed Response
• Short-Answer Essay Questions
• Concept Maps
• Identifying Themes
• Making Predictions
• Summaries
• Explain Your Solution
Course Assessment: Rubrics/Scoring Guides
http://jfmueller.faculty.noctrl.edu/toolbox/tasks.htm
Product/Performance “...reveals their understanding of certain concepts and skills and/or their ability to apply, analyze, synthesize or evaluate those concepts and skills” *
Research Paper Capstone Project Article Reviews Film Analysis Case Study Error Analysis Panel Discussion Fishbowl Discussion Oral Presentations Course Assessment: Rubrics/Scoring Guides * http://jfmueller.faculty.noctrl.edu/toolbox/tasks.htm
Types of Rubrics
1. Holistic
2. Check-list
3. Rating Scale
4. Descriptive
5. Structured Observation Guide
Examples
Holistic (Suskie, 2009)
Checklist Rubric (Suskie, 2009)
Rating Scale Rubric (Suskie, 2009)
Descriptive Rubric (Suskie, 2009)
Structured Observation Guide (Suskie, 2009)
Things to consider
• Levels of Assessment
• Individual
• Alignment from outcome to assignment to rubric
• Additional information to be measured (format, program outcomes, etc.)
• Testing the rubric
• Class/Course
• All of the above
• Sampling
• Analysis: Aggregate data
• Program
• All of the above
• Norming/multiple raters
Testing Your Rubric/Scoring Guide
• Metarubric: use a to review your work
• Peer review: ask one of your colleagues to review the rubric
and provide feedback on content
• Student review: ask several students review the rubric if
appropriate (students in a class may help you create it)
• Test with products: use student work to test the rubric once
you feel it is ready (3-5?)
Sampling: Being Systematic
• Should you sample at all or review all the products? • Mary Allen- Rule of Thumb_ 50 to 75 is sufficient for assessment
• Consider the attitudes of the faculty towards data:
• Quantitative approaches
• Qualitative approaches
**The Office of Assessment can help choose random students for your assessment.
Analysis
• Individual vs. Aggregate Scores
• Average Score (Mean) By Dimension and Total
• Total Score: Total scores may be reviewed to get a big picture
• Dimension: Dimension scores to look for patterns
• Frequency Distributions
• Scale: Frequencies by scale to get a clearer understanding of the data
Scoring the Data ID# Class Age Gender
Paper
Length Total
Separation/
Objectivity Dissonance
Understanding/
Change in
Perspective
Self-
Perception Resolution
Application/
Verification totals
A FR 19 F 5 18 3 3 3 3 3 3 18
B SR 21 M 3 17 3 3 3 3 3 2 17
C FR 18 F 7 16 3 3 3 2 2 3 16
D SR 21 M 5 16 3 3 3 2 3 2 16
E SO 19 F 9 15 2 3 3 2 2 3 15
F FR 18 M 3 14 3 3 3 2 3 0 14
G SO 20 M 3 14 3 3 3 0 3 2 14
H SO 19 M 5 13 2 2 3 2 2 2 13
I FR 18 M 8 13 3 3 3 2 2 0 13
J JR 20 F 5 13 2 2 2 2 3 2 13
K SO 20 M 5 13 3 3 2 2 2 1 13
L FR 18 M 7 13 2 3 2 2 2 2 13
M JR 20 F 3 11 3 3 3 0 2 0 11
N FR 18 F 5 10 2 2 2 2 2 0 10
O SO 22 M 4 10 2 3 2 2 2 0 11
P FR 18 F 6 10 2 3 1 2 1 1 10
Q FR 19 M 9 9 2 2 1 2 1 1 9
R FR 18 M 3 9 2 3 2 1 1 0 9
S FR 18 M 15 7 2 1 1 1 1 1 7
T SO 20 F 4 7 1 2 0 1 2 1 7
Average Score 2.5 2.8 2.4 1.8 2.2 1.4 13.1
0
2
4
6
8
10
12
14
16
Separation/
Objectivity
Dissonance Understanding/
Change in
Perspective
Self-Perception Resolution Application/
Verification
Scale: 3
Scale: 2
Scale: 1
Scale: 0
Frequencies Scale: 3 Scale:2 Scale: 1 Scale: 0
Separation/ Objectivity 9 10 1 2
Dissonance 14 5 1 2
Understanding/ Change in Perspective 10 6 3 3
Self-Perception 2 13 3 4
Resolution 6 10 4 2
Application/ Verification 3 6 5 8
Frequencies
Example: University of Virginia
Other “Scoring” Options
• Structured Observation Guide:
• Thematic approach-using qualitative coding to determine what you are seeing.
Norming with Multiple Raters • Will multiple reviewers look at each product?
• Consider time frame (when must all products be scored)
• Will raters score together in the same physical location?
• Spend time walking through the rubric/scoring guide as a group.
• Review 1 product individually, then compare responses as a group- share why scores were chosen, discuss, reach consensus.
• Repeat process above as needed until you feel comfortable that you are on the same page (5 to 10)
• This could result in additional changes to the rubric/scoring guide in some cases
• Consider doing this throughout the process to ensure that you are not drifting (“recalibrate”)
Creating a Rubric
1. What are you trying to assess? (Outcomes)
2. What does the outcome (s) ‘look‘ like?
3. How can you group the item/elements/criteria/dimensions of the outcome?
4. List these elements with brief description of the best case scenario
5. How will the student demonstrate this learning? Does the assignment align with the list of elements?
6. What else needs to be in the rubric to grade the work?
7. Do you need specific information for the curriculum?
References/Resources
• AACU Rubrics: http://www.aacu.org/value/rubrics/index_p.cfm?CFID=37317515&CFTOKEN=54026278
• Allen, Mary J. (2006). Assessing General Education Programs. Bolton, MA: Anker Publishing Co.
• Stevens and Antonia (2005). Introduction to Rubrics. Sterling, VA:Stylus Publishing, LLC.
• Suskie, L (2009). Assessing Student Learning: A common sense guide. San Francisco, CA: Jossey-Bass.