clarifying learning goals & standards: rubrics & exemplars

Clarifying learning goals & standards: rubrics & exemplars

David Carless University of Hong Kong

At EdUHK, January, 20, 2017

The University of Hong Kong

Overview

1. Key concepts2. Students’ views on rubrics & exemplars3. Using rubrics for learning 4. Analyzing exemplars 5. Summary of key implications


KEY CONCEPTS


Competing assessment functions The University of Hong Kong

Stimulating productive student learning

Satisfying accountability needs

Judging student achievement

Learning-oriented assessment

Prioritizing student learning as a key aim of all assessment



Learning-oriented assessment framework (Carless, 2015a)

Productive assessment task

design

Student engagement with

feedback

Understanding quality academic

work

Importance of learning goals

Where am I going?

How to get there?


What are rubrics?

• A.k.a. ‘assessment criteria’, ‘grade descriptors,’ ‘marking schemes’

• Rubrics combine criteria and standards in the form of a grid to provide statements of the performance needed to achieve different standards (Carless, 2015, p. 132)


What are exemplars?

Exemplars are samples of student work used to illustrate dimensions of quality

Usually assignments from a previous cohort


Two relevant chapters

Carless, D. (2017). Students’ Experiences of Assessment for Learning. In D. Carless et al., (Eds.), Scaling up Assessment for learning in Higher Education. Singapore: Springer. Jönsson, A. & Panadero, E. (2017). The Use and Design of Rubrics to Support Assessment for Learning. In D. Carless et al., (Eds.), Scaling up Assessment for learning in Higher Education. Singapore: Springer.


EVIDENCE FROM INTERVIEWS WITH STUDENTS


Research process The University of Hong Kong

Classroom observations 　

Interviews with teachers & students HistoryLaw

Business

Rubrics data (1)

“Criteria are rather vague to me …”

“I read the criteria descriptions some time ago and I’ve already forgotten them”

“The criteria were similar to previous ones so I didn’t study them carefully or consult them when preparing the assignment”.


Rubrics data (2)

“The descriptions of assessment criteria are almost the same for every subject with the same key words like critical & analytic”

“The assessment criteria are a bit lifeless. I wanted to know the teacher’s interpretation of the criteria and his personal preference”.


Rubrics data (3)

“Lists of criteria are all the same, it’s all very vague. Like for an ‘A’, it is excellent mastery and correct application and then ‘B+’ would be above average mastery and good application. You don’t know what it means so you just do what you can. Once you know how to get good academic results, you don’t need to rely on criteria any more because the criteria are always the same.”


Exemplars data (1)

“Samples are more concrete”

“It is a reference … you can imitate its presentation style”


Exemplars data (2)

“The exemplar served as a useful example of what a good report should be”

“There is a risk of copying but many of us try to add something more or use our own approach”


Inferences

Rubrics are only useful for learning if students engage with them

Exemplars are attractive to students but teacher implementation is complex


RUBRICS & STUDENT ENGAGEMENT


Dual aspects of rubrics

1. A scoring tool for teachers

2. A potential learning tool for students as aid to self-evaluation


Rubric options

• Analytic or holistic

• Generic or Task-specific

(see handout)


Writing rubrics

Standards should be represented with respect to features of work, not impenetrable terms, such as ‘good’ or ‘excellent’ (Ashwin, Boud et al., 2015, p. 259)


Rubrics activity 1

Students in groups brainstorm their own criteria for an assessment task


Rubrics activity 2

Students read the criteria for (say) a grade B and then re-write them in their own words


Rubrics activity 3

Students use rubric to evaluate work of peer (peer evaluation)

AND/OR

Students use rubric to evaluate exemplars


DISCUSSION 1: RUBRICS

• Share some of your thinking on using rubrics to enhance student learning

• What unanswered issues might you raise in relation to rubrics?


RATIONALE & USE OF EXEMPLARS


Why used?

Exemplars convey messages that nothing else can (Sadler, 2002)


Promote self-evaluation

Illustrate what good work looks like

Benchmark for student comparison with own performance


Analysis

Identify strengths, weaknesses & suggest how samples could be improved


When used?

Useful for students at transition stages

When assignments are complex, innovative or hard to explain


How used?

• Posted on LMS• Optional workshop (Rust et al., 2003)• Part of regular class• Displayed or made available• In part or as a whole


Benefits for students Gain experience in making judgments

Apply insights to own work

Notice discrepancies between present level & target level


Copying model problem

May reduce student creativity

May encourage unproductive copying


Time issues

• Time in collecting them

• Time in seeking consent for their use

• Time taken away from instruction


It’s challenging!

Difficulties for some students in evaluating exemplars accurately


Synthesis on exemplars

• Students like exemplars • They aid learning But • How to implement effectively• How to avoid ‘copying model answer

syndrome’


Key inference

The quality of the dialogue about the exemplars is a crucial factor determining student learning


DISCUSSION 2: EXEMPLARS

• Why don’t teachers make more & better use of exemplars?

• What unanswered issues, might you raise in relation to exemplars?


RECENT EXEMPLARS RESEARCH


TDG Project

Enhancing dialogic use of exemplars amongst 10 teachers in a Faculty of Education


Research question

What are the main features of a specific exemplars dialogue and how is it orchestrated?


Context of the case study

Participants: • Trainee Science teachers • BEd/BSc double degree • Year 3

Teacher-researcher: Dr Kennedy Chan


Data collection

• Classroom observation • Open-ended student survey• 2 focus group interviews• Interview with teacher-researcher• Teacher journal• Student artefacts: ‘exit slips’, assignments etc.


Exemplars provided2 high quality exemplars

– Same format (i.e., reflective essays) but different content focus

– Design aimed at reducing ‘copying’


Management of dialogue

1. Prior to class students analyzed exemplars;2. Students discussed exemplars in pairs; 3. Students elicited views from peers and made

mini-presentations; 4. Teacher-orchestrated dialogue; 5. Students submitted exit slip of reflections.


Dilemmas

Students constructing

notion of quality VS

Telling students about quality

Time for students to talk with

peers VS

Time for developing shared understandings

Exemplars as

guide VS

Exemplars as model

Using students’ voices VS

Making teacher’s voice

explicit


Good Exemplars dialogue

• Airs multiple & divergent viewpoints • Shows linkages between peer talk & whole-

class discussion • Evidences development of student views • Makes explicit some key qualities of exemplars(Carless & Chan, 2016)


CONCLUSION


Using exemplars & criteriaAssessment task

Students devise or engage with rubric & quality

Two samples read before class

Peer

discussion Teacher-led dialogue

Student ownership of

insights


Engaging with quality

Student need to engage with what quality looks like and develop capacities in making judgments


THANK YOU


References Ashwin, P., Boud, D., et al., (2015). Reflective teaching in higher education. London: Bloomsbury. Carless, D. (2015a). Exploring learning-oriented assessment processes. Higher Education, 69(6), 963-976.Carless, D. (2015b). Excellence in University Assessment: learning from award-winning teachers. London: Routledge. Carless, D. (2017). Students’ experiences of Assessment for Learning. In D. Carless, S. Bridges, C.K.W. Chan & R. Glofcheski (Eds.), Scaling up Assessment for learning in Higher Education. Singapore: Springer.Carless, D., Bridges, S., Chan, C.K.W. & R. Glofcheski, R. (Eds.), (2017). Scaling up Assessment for learning in Higher Education. Dordrecht: Springer.Carless, D. & K.K.H. Chan (2016). Managing dialogic use of exemplars. Assessment and Evaluation in Higher Education, http://dx.doi.org/10.1080/02602938.2016.1211246Hendry, G., Armstrong, S. & Bromberger, N. (2012). Implementing standards‐based assessment effectively: Incorporating discussion of exemplars into classroom teaching. Assessment and Evaluation in Higher Education, 37(2), 149-161.


References (cont.) Jonsson, A. & Panadero, E. (2017). The use and design of rubrics to support assessment for learning. In D. Carless, S. Bridges, C.K.W. Chan & R. Glofcheski (Eds.), Scaling up Assessment for learning in Higher Education. Singapore: Springer.Rust, C., Price, M. & O’Donovan, B. (2003). Improving students’ learning by developing their understanding of assessment criteria and processes. Assessment and Evaluation in Higher Education, 28(2), 147-164.Sadler, D. R. (2002). Ah! … So that’s ‘quality’. In P. Schwartz & G. Webb (Eds.), Assessment: Case Studies, Experience and Practice from Higher Education (p.130-136). London: Kogan Page.Smith, C., Worsfold, K., Davies, L., Fisher, R. & McPhail, R. (2013). Assessment literacy and student learning: The case for explicitly developing students ‘assessment literacy’. Assessment and Evaluation in Higher Education, 38(1), 44-60.
