designing grading rubrics to assess student learning the diy (do it yourself) workshop mary t....
TRANSCRIPT
Designing Grading Designing Grading Rubrics to Assess Rubrics to Assess Student LearningStudent Learning
The DIY (Do It Yourself) The DIY (Do It Yourself) WorkshopWorkshop
Mary T. Blackinton PT, MS, EdDMary T. Blackinton PT, MS, EdDDirector Transition DPT ProgramDirector Transition DPT ProgramChair, PT Outcomes CommitteeChair, PT Outcomes CommitteeNova Southeastern UniversityNova Southeastern University
Dr. Mary Blackinton, 01-19-08
AgendaAgenda Introduction to GradingIntroduction to Grading
Myths & Principles of Effective GradingMyths & Principles of Effective Grading Choosing an Assignment for a RubricChoosing an Assignment for a Rubric
Analyzing Grading RubricsAnalyzing Grading Rubrics Small group discussionSmall group discussion
Developing RubricDeveloping Rubric DefinitionsDefinitions Primary Trait AnalysisPrimary Trait Analysis
Creating & Using RubricsCreating & Using Rubrics DesignDesign PitfallsPitfalls
Dr. Mary Blackinton, 01-19-08
ObjectivesObjectives Upon completion of the workshop, faculty will be Upon completion of the workshop, faculty will be
able toable to:: Describe the role of grading rubrics in: Describe the role of grading rubrics in:
Clarifying teacher expectationsClarifying teacher expectations Evaluating student learningEvaluating student learning Providing detailed feedbackProviding detailed feedback Improving instructionImproving instruction Performing outcome assessmentPerforming outcome assessment
Evaluate the strengths and weaknesses of grading rubrics Evaluate the strengths and weaknesses of grading rubrics as a student assessment tool in health professions as a student assessment tool in health professions educationeducation
Integrate the principles of primary trait analysis (PTA) into Integrate the principles of primary trait analysis (PTA) into designing a rubric for an assignment or performance skilldesigning a rubric for an assignment or performance skill
Create a grading rubric for an assignment, project, or Create a grading rubric for an assignment, project, or examexam
Dr. Mary Blackinton, 01-19-08
Grading Challenge # 214:
Student’s Perceptions
Dr. Mary Blackinton, 01-19-08
Why do we Grade?Why do we Grade? GradingGrading11
The The processprocess used by faculty to assess student used by faculty to assess student learning via assignments and exams, learning via assignments and exams, including:including:
Relating test items or assignments to learning Relating test items or assignments to learning objectivesobjectives
Establishing criteria/standards (SAFETY)Establishing criteria/standards (SAFETY) Helping students acquire needed knowledge/skillsHelping students acquire needed knowledge/skills Facilitating student motivationFacilitating student motivation Giving feedback about performanceGiving feedback about performance Communicating about what has been learned Communicating about what has been learned Using results to influence teaching and curriculumUsing results to influence teaching and curriculum
11 BE Walvoord, VJ Anderson (1998). BE Walvoord, VJ Anderson (1998). Effective grading: A Effective grading: A tool for learning and assessmenttool for learning and assessment. Jossey-Bass: San . Jossey-Bass: San FranciscoFrancisco
Dr. Mary Blackinton, 01-19-08
Student Assessment in Health Professions
Faculty charged to assess competency in patient Faculty charged to assess competency in patient carecare
Responsibility to protect publicResponsibility to protect public Other stakeholders: accrediting bodies, profession, students, Other stakeholders: accrediting bodies, profession, students,
parents…parents…
Professionals integrate 3 domains of learning Professionals integrate 3 domains of learning Cognitive-thinking/reasoningCognitive-thinking/reasoning Affective-feeling, valuing, ethicsAffective-feeling, valuing, ethics Psychomotor-doing, manipulating, performing, administeringPsychomotor-doing, manipulating, performing, administering
Assessment strategies reflect demands Assessment strategies reflect demands professional edprofessional ed
Must include more than multiple choice assessment to Must include more than multiple choice assessment to determine learningdetermine learning
Often have multiple faculty grading student performanceOften have multiple faculty grading student performance
Dr. Mary Blackinton, 01-19-08
Myths About GradingMyths About Grading
Grading:Grading: Is a necessary EVILIs a necessary EVIL Must be objectiveMust be objective Should be easyShould be easy Should not take detract from teachingShould not take detract from teaching Use knowledge of experts to evaluate novicesUse knowledge of experts to evaluate novices Is assessment only, doesn’t impact learningIs assessment only, doesn’t impact learning With clear expectations means you are “spoon-With clear expectations means you are “spoon-
feeding”feeding” Cannot be used in program assessmentCannot be used in program assessment
Dr. Mary Blackinton, 01-19-08
Rubrics as Assessment of Rubrics as Assessment of Program OutcomesProgram Outcomes
Learning Outcome
Direct Measure
Assessment Method
Students will be Students will be able to select able to select valid/ reliable valid/ reliable tests/measures tests/measures to assess to assess patient patient outcomesoutcomes
Assignment 2, Assignment 2, Neuromuscular Neuromuscular I: Given case, I: Given case, students select students select tests/measures tests/measures & provide & provide rationale based rationale based on patient on patient needs & psycho- needs & psycho- metric metric propertiesproperties
Rubric identifies Rubric identifies ability to ability to ascertain ascertain validity & validity & reliability of reliability of tests and tests and measures by measures by reading reading relevant relevant literature, and literature, and applying to applying to specific needs specific needs of a patient (see of a patient (see sample)sample)
Dr. Mary Blackinton, 01-19-08
Principles of Effective Principles of Effective GradingGrading11
1. Appreciate the complexity of grading2. Substitute judgment for objectivity3. Distribute time effectively 4. Be open to change5. Listen & observe6. Communicate & collaborate with students7. Integrate grading with other key processes8. Seize the teachable moment (feedback)9. Make student learning the primary goal10. Be a teacher FIRST, gatekeeper LAST11. Encourage learning-centered motivation12. Emphasize student involvement
11 Walvoord &Anderson (1998) Walvoord &Anderson (1998)
Dr. Mary Blackinton, 01-19-08
Types of Grading in Health Types of Grading in Health ProfessionsProfessions
Student Assessment Student Assessment MethodsMethods Multiple choice examinationMultiple choice examination Short answer questions on testsShort answer questions on tests Essay questionsEssay questions Annotated bibliographiesAnnotated bibliographies Literature reviewsLiterature reviews Case StudiesCase Studies Oral ExaminationsOral Examinations Practical/Performance Practical/Performance
ExaminationsExaminations Clinical AssessmentsClinical Assessments JournalsJournals PortfoliosPortfolios Lab ConclusionsLab Conclusions
2LC Jacobs, CI Chase (1992). Developing and using tests effectively. Jossey-Bass: San Francisco
Student-Constructed Responses
Dr. Mary Blackinton, 01-19-08
Methods of Grading Student-Methods of Grading Student-Constructed ResponsesConstructed Responses
Method Advantage DisadvantageNorm Referenced: : categorize work categorize work into A, B, C, D into A, B, C, D
Little work up Little work up front, recognizes front, recognizes faculty as faculty as ‘expert’, flexible‘expert’, flexible
-Potential for -Potential for biasbias
-Less opportunity -Less opportunity for learning, for learning, vaguevague
Checklist: list : list of criteria to of criteria to include include (introduction, (introduction, research research question…)question…)
Assignment Assignment directions match directions match checklist, not checklist, not difficult to difficult to prepareprepare
-Usually lacks -Usually lacks descriptionsdescriptions
-Lists + traits or -Lists + traits or behaviors, no behaviors, no negativenegative
Grading Rubric: criterion referenced, describe performance expectations & weighting
Explicit Explicit expectations, expectations, better feedback, better feedback, greater inter-greater inter-rater reliability, rater reliability, links to links to performanceperformance
-Takes time to -Takes time to construct, may construct, may need to modify need to modify after 1after 1stst run run
Dr. Mary Blackinton, 01-19-08
What are Assessment What are Assessment Rubrics?Rubrics?
Small Group DiscussionSmall Group Discussion In small group, look at the sample In small group, look at the sample
rubric packet. Then, answer the rubric packet. Then, answer the following:following:
Discussion QuestionsDiscussion Questions Which rubrics seem the MOST clear to Which rubrics seem the MOST clear to
you? Why?you? Why? Which rubrics are not clear? Why?Which rubrics are not clear? Why? What was common among the rubrics you What was common among the rubrics you
viewed as clear?viewed as clear? Based on samples, how do you define Based on samples, how do you define
assessment rubrics?assessment rubrics?
Dr. Mary Blackinton, 01-19-08
What is a Grading What is a Grading Rubric?Rubric?
Method of “articulating expectations for an Method of “articulating expectations for an assignment by listing the criteria, or assignment by listing the criteria, or what what countscounts, & describing , & describing levels of qualitylevels of quality from from excellent to poor”excellent to poor”33
Type of assessment that specifies Type of assessment that specifies gradations of gradations of qualityquality from excellent to poor from excellent to poor44
A A criterion-referencedcriterion-referenced method of grading using method of grading using highly specific grading criteria that are linked to highly specific grading criteria that are linked to objectivesobjectives3 HG Andrade, Y Du (2005). Students perspectives on rubric-referenced assessment. Practical Assessment, Research & Evaluation, Vol 10 (3).
4 HG Andrade ((2005). Teaching with rubrics: The good, the bad, and the ugly. Coll Teaching, 53 (1)
Dr. Mary Blackinton, 01-19-08
What is a Grading What is a Grading Rubric?Rubric?
PerformaPerformance Level nce Level 33
PerformaPerformance Level nce Level 22
PerformaPerformance Level nce Level 11
DimensioDimension 1n 1
3 pts: 3 pts: describedescribe
2 pts2 pts 1 pt1 pt
DimensioDimension 2n 2
4 pts4 pts 2.5 pts2.5 pts 1 pt1 pt
DimensioDimension 3n 3
3 pts3 pts 2 pts2 pts 1 pt1 pt
DimensioDimension 4n 4
2 pts2 pts 1 pts1 pts 0 pts0 pts
Gradation: excellent-poorCategories
important to the teacher/class
Weighted PointsDimensions also called criteria
Dr. Mary Blackinton, 01-19-08
Criteria 1Criteria 1 Criteria 2Criteria 2 Criteria 3Criteria 3
Level 1Level 1 3 pts: 3 pts: describedescribe
2 pts2 pts 1 pt1 pt
Level 2Level 2 4 pts4 pts 2.5 pts2.5 pts 1 pt1 pt
Level 3Level 3 3 pts3 pts 2 pts2 pts 1 pt1 pt
Level 4Level 4 2 pts2 pts 1 pts1 pts 0 pts0 pts
Dr. Mary Blackinton, 01-19-08
How do Students Perceive How do Students Perceive Rubrics?Rubrics?
Andrad & Yu, 2005Andrad & Yu, 2005 Investigated how students use grading rubricsInvestigated how students use grading rubrics
Focus group / qualitative designFocus group / qualitative design Students reported that they used rubrics toStudents reported that they used rubrics to
Help them determine faculty expectationsHelp them determine faculty expectations To plan an approach to assignmentTo plan an approach to assignment Check/revise work before handing inCheck/revise work before handing in Help reflect on their learning-see Help reflect on their learning-see
strengths/weaknesses clearlystrengths/weaknesses clearly Perceived results of rubric usePerceived results of rubric use
Better, fairer gradesBetter, fairer grades Improvements in quality across classesImprovements in quality across classes Less anxietyLess anxiety
Dr. Mary Blackinton, 01-19-08
Using Primary Trait Using Primary Trait AnalysisAnalysis1,41,4
To Develop a Grading To Develop a Grading RubricRubricRationaleRationale
Analyze traits / characteristics of Analyze traits / characteristics of student learning and then clearly student learning and then clearly articulate them, leading to:articulate them, leading to: Assignment specificAssignment specific Explicit criteriaExplicit criteria Adds objectivity & consistency to holistic Adds objectivity & consistency to holistic
scoringscoring Lets students know in ADVANCE how will be Lets students know in ADVANCE how will be
gradedgraded
1 1 Walvoord &Anderson (1998)Walvoord &Anderson (1998)
44R Lloyd-Jones (1977). Primary Trait Scoring in C. Cooper & L. Odell R Lloyd-Jones (1977). Primary Trait Scoring in C. Cooper & L. Odell (Eds), Evaluating Writing: Describing, measuring, judging. Urbana IL: (Eds), Evaluating Writing: Describing, measuring, judging. Urbana IL: National Council of Teachers National Council of Teachers
Dr. Mary Blackinton, 01-19-08
Constructing a Primary Trait Constructing a Primary Trait Analysis ScaleAnalysis Scale11
1. Choose assignment/test that matches course objective
2. Identify all traits that will count for scoring in assessment
1. Body Language vs Thoroughness vs Accuracy
3. Build scale for scoring performance, gradations of skill
1. Scale usually ranges from 2-5 points
2. Include what should be demonstrated and what should be avoided
3. Build a range that discriminates A from A- from B+
4. Run scale by colleague, graduate, teaching assistant
5. Weight items: content > spelling; accuracy > efficiency, etc
4. Evaluate performance against criteria1. Try scale with sample & revise as needed
Dr. Mary Blackinton, 01-19-08
Example: Designing a Example: Designing a RubricRubric
Select assignment appropriate for PTASelect assignment appropriate for PTA Taking a Patient History (clinical (clinical
performance)performance) Identify ALL traits you look for in Identify ALL traits you look for in
patient historypatient history Body language & eye contact (Body language & eye contact (X-1X-1)) Thoroughness (Thoroughness (X-2X-2)) Data gathering (Data gathering (X 1.5X 1.5))
WWeighteight them each trait them each trait
Dr. Mary Blackinton, 01-19-08
Building a Scale: Building a Scale: Body Body LanguageLanguage
Level 3-ExcellentLevel 3-Excellent Consistently had good eye contact with patientConsistently had good eye contact with patient Communicated at patient’s eye-levelCommunicated at patient’s eye-level Maintained an open posture, leaning toward patientMaintained an open posture, leaning toward patient Facial expression interested, non-judgmentalFacial expression interested, non-judgmental Confident and relaxed, not arrogant Confident and relaxed, not arrogant
Level 2-GoodLevel 2-Good At least 4 of the above behaviors notedAt least 4 of the above behaviors noted OR, all 5 qualities demonstrated but not consistentlyOR, all 5 qualities demonstrated but not consistently
Level 1-FairLevel 1-Fair Inconsistent in several (2-3) areas, for example, confident Inconsistent in several (2-3) areas, for example, confident
through some portions of history but not others; or, eye through some portions of history but not others; or, eye contact missing at endcontact missing at end
Level 0-PoorLevel 0-Poor Arrogance, avoidance, lack of confidence detracted from Arrogance, avoidance, lack of confidence detracted from
relationshiprelationship
Gives students specific behaviors to demonstrate!Identifies behaviors to avoid!
Dr. Mary Blackinton, 01-19-08
Building a Scale: Building a Scale: ThoroughnessThoroughness
44: : Extremely ThoroughExtremely Thorough History includes all the following core areas: 1. all body History includes all the following core areas: 1. all body
systems; 2. history current illness/problem; 3. family history; 4. systems; 2. history current illness/problem; 3. family history; 4. medications; 5. educational background / learning style; 6. medications; 5. educational background / learning style; 6. lifestyle (diet, exercise, habits); 7. living/ work/social lifestyle (diet, exercise, habits); 7. living/ work/social environment; 8. assistive/adaptive devices (glasses, cane, etc); environment; 8. assistive/adaptive devices (glasses, cane, etc); 9. prior level of function 9. prior level of function
3: Thorough3: Thorough All items (1-9) included but may have missed some sub-All items (1-9) included but may have missed some sub-
categories (ie missed 2 body systems); or, only missed 1-2 of categories (ie missed 2 body systems); or, only missed 1-2 of the 9 core areas abovethe 9 core areas above
2: Somewhat Thorough2: Somewhat Thorough Missing 3-4 of the 9 core areas, or, missing only 1-2 areas but Missing 3-4 of the 9 core areas, or, missing only 1-2 areas but
was also superficial in review of body systemswas also superficial in review of body systems 1: Incomplete 1: Incomplete
Less than half of the 9 core areas were covered; incomplete Less than half of the 9 core areas were covered; incomplete datadata
Dr. Mary Blackinton, 01-19-08
Building a Scale: Building a Scale: Data Data GatheringGathering
Excellent-4Excellent-4 1. Uses open-ended, broad questions; 2. encourages patient 1. Uses open-ended, broad questions; 2. encourages patient
to tell story chronologically; 3. allows patient to talk without to tell story chronologically; 3. allows patient to talk without interrupting; 4. encourages by using phrases such as ‘tell interrupting; 4. encourages by using phrases such as ‘tell me more’ or ‘what else?’; 5. summarizes what was heard; 6. me more’ or ‘what else?’; 5. summarizes what was heard; 6. Approach was organizedApproach was organized
Good-3Good-3 Demonstrates at least 4 of 6 skills noted above throughout Demonstrates at least 4 of 6 skills noted above throughout
the history, or, used all 5 skills but inconsistently. the history, or, used all 5 skills but inconsistently. Fair-2Fair-2
Demonstrates 3 of the core probing skills, or, 3-4 skills used Demonstrates 3 of the core probing skills, or, 3-4 skills used however did so inconsistently; multiple ‘missed’ however did so inconsistently; multiple ‘missed’ opportunities to probe; disorganized at timesopportunities to probe; disorganized at times
Poor-1Poor-1 Did not consistently demonstrate probing skills, questions Did not consistently demonstrate probing skills, questions
were narrow and rarely open-ended, interrupted or cut were narrow and rarely open-ended, interrupted or cut patient off, rarely encouraged more information with patient off, rarely encouraged more information with phrases such as ‘tell me more’.phrases such as ‘tell me more’.
Dr. Mary Blackinton, 01-19-08
DIY: Build Your Own PTADIY: Build Your Own PTA
TraitsTraits
GradationsGradationsTrait 1: Trait 2: Trait 3:
Excellent
Good
Fair
Poor
Assignment or Test:____________________________________________________
Related Class Objectives: ________________________________________________
Dr. Mary Blackinton, 01-19-08
Step 4: Evaluate Step 4: Evaluate Performance Performance Using CriteriaUsing Criteria Test the ScaleTest the Scale
Use it with an past paper, videotape, case Use it with an past paper, videotape, case report, etc.report, etc.
What changes are needed in the rubric? What changes are needed in the rubric? Were the criteria easy to follow?Were the criteria easy to follow? Did the overall grade reflect performance?Did the overall grade reflect performance?
Are any changes needed in description Are any changes needed in description or in the point allotment?or in the point allotment? Did an A performance score better than a B?Did an A performance score better than a B?
Is it easy? Hard? Too cumbersome?Is it easy? Hard? Too cumbersome?
Dr. Mary Blackinton, 01-19-08
Applying the RubricApplying the Rubric Distribute ( or post) rubric to students in advanceDistribute ( or post) rubric to students in advance
Have conversations about expectationsHave conversations about expectations Ask students to attach rubric to assignmentAsk students to attach rubric to assignment
Helps students pay attention!Helps students pay attention! Use rubric to gradeUse rubric to grade
Match written comments to phrases in rubricMatch written comments to phrases in rubric Revise after useRevise after use
Make changes soon after grading for next timeMake changes soon after grading for next time Answer Curriculum QuestionsAnswer Curriculum Questions
Did the students learn? To what degree were objectives Did the students learn? To what degree were objectives met?met?
If not, was it the teaching? The assignment? If not, was it the teaching? The assignment? Background skills?Background skills?
What, if any, changes should be made in the class or What, if any, changes should be made in the class or curriculum? Rubric?curriculum? Rubric?
Dr. Mary Blackinton, 01-19-08
Advantages of Grading Advantages of Grading RubricsRubrics11
Save Save timetime in grading process in grading process Makes process of grading Makes process of grading reliable/fairreliable/fair Clarifies Clarifies expectationsexpectations for students for students Reinforces key Reinforces key conceptsconcepts - help faculty relate to - help faculty relate to
objectivesobjectives Students are Students are participantsparticipants as expectations are known as expectations are known Student Student peer reviewpeer review Works well if Works well if team-teachingteam-teaching Share Share across coursesacross courses or over curriculum or over curriculum Basis for departmental/Basis for departmental/programprogram assessmentassessment Which of these advantages pertain to your teaching
in the health professions?
1 Walvoord & Anderson, 19981 Walvoord & Anderson, 1998
Dr. Mary Blackinton, 01-19-08
CAUTION:CAUTION: Common Rubric Common Rubric PitfallsPitfalls
Rubric does not correspond with class or program Rubric does not correspond with class or program outcomesoutcomes Example: Entire rubric focused on writing quality not Example: Entire rubric focused on writing quality not
contentcontent Scale does Scale does notnot have enough gradations or levels have enough gradations or levels
Not distinguishing the A’s from the B+’sNot distinguishing the A’s from the B+’s All traits are given equal weight regardless of All traits are given equal weight regardless of
complexitycomplexity Grammar = ContentGrammar = Content
Too broad, not enough content describedToo broad, not enough content described Words like ‘breadth’ and ‘depth’ used in lieu of specificsWords like ‘breadth’ and ‘depth’ used in lieu of specifics Students still not sure, other grading faculty still not clearStudents still not sure, other grading faculty still not clear
Too long/too complicatedToo long/too complicated Faculty + students get lost in the rubricFaculty + students get lost in the rubric
Dr. Mary Blackinton, 01-19-08
Classroom Assessment Classroom Assessment TechniqueTechnique
Write down ONE thing you will commit to Write down ONE thing you will commit to incorporate regarding the use of grading incorporate regarding the use of grading rubricsrubrics
Write down ONE unanswered questionWrite down ONE unanswered question
Contact: [email protected]: [email protected]