Making assessment more efficient and effective

Ray Land, Discovery Point, Dundee, January 13th 2004

‘Streamlining’ assessment

Ashley Havinden – Design for Chrysler Motors Ltd, 1929 (detail)

• Principles and purposes of assessment (effective for what?)

• Effects of massification

• Control strategies

• Independence strategies

• Issues of risk and support –examples of adaptive initiatives

Ashley Havinden 1903-1973 Design for Chrysler Motors Ltd, 1929 (detail) Scottish National Gallery of Modern Art Archive

© The artist’s estate with thanks to DaimlerChrysler UK Limited

Construction of identities through

assessment

Paul Ramsden – ‘a serious and often tragic enterprise’

Sally Brown – ‘a nightmare’

David Boud - ‘more bad practice and ignorance of significant issues in the area of

assessment than in any other aspect of higher education’

Principles of Assessment

• Beneficial• Actively foster

learning• Fair• Diverse and varied• Valid• Reliable

• Transparent• Representative• Effective• Practicable/ cost

effective• Right of redress• Secure

Need to bear in mind that principles of assessment often contradict.

What is efficient may not be effective. What is effective may not be efficient.

What is assessment?

• Sampling evidence of performance

• Making inferences

• Estimating worth

• Representing that worth (symbolically)

I keep six honest serving-men(They taught me all I knew)Their names are What and Why and Whenand How and Where and Who

(Rudyard Kipling – Just-So Stories)

• What kinds of knowledge, skills, abilities or values are we seeking evidence of?

• Why are we seeking to gain evidence of achievement in the first place? For what purpose will this evidence be used?

• When should assessment take place – at the end of the course? at periodic intervals throughout it? continuously? at the start of the course?

• How should we gather the evidence? What is the most appropriate strategy to use for our particular course?

• Where should assessment take place – in the examination hall? during a group presentation? In the workplace? On a computer?

• Who should undertake the assessment – the teacher? Anonymous examiners? A workplace/clinical supervisor? The student? The student’s peers?

What’s assessment for?

• Diagnostic testing

• Motivating/regulating students (and staff)

• Developing student confidence

• Evaluation of teaching

• Certification /licensing

• Providing feedback to students

• Selection for subsequent courses

• Information for employment

• Predicting future performance

• Monitoring progress• Public accountability• Institutional marketing• Social stratification

Effects of massification • Over assessment• Lack of advice on improvement• Lack of knowledge of progress• Slow return of coursework• Same outcomes repeatedly assessed

(redundancy)• Credit / workload not calibrated • ‘Convergence’ of assessments leading

to stress points • Lack of variety/ repetition of method

across modules• Consumerism / commodification • ‘Performativity’

Over assessment an issue for both staff and students

Issues arising from widening participation

• Academic literacies debate / acculturation

• Traditional dominance of particular modes (eg essay)

• Pacing of learning / ‘Slow’ learning

• Assessment in the first term

• Sensitivity in feedback

IndependenceControl

(Gibbs and Jenkins, 1997)

Choice of strategies

‘control’ strategies

traditional methods retained but carried out with increasingly greater (industrial) efficiency cf ‘Performativity’ (Lyotard)

‘independence’ strategies

shift of responsibility for (usually formative) assessment to students themselves, on the grounds that, apart from economies of effort, this is in itself an educational benefit that is likely to enhance student performance

Change issues - Risk and support

• High Risk High Support

• High Risk Low Support

• Low Risk High Support

• Low Risk Low Support

control strategies

• Set fewer assignments• Set shorter assignments• Set fewer exam questions• MCQs• Use standardised feedback forms• CAA – eg computerised objective tests, word-

processed statement banks, optical mark readers• Set limits on assignment length and materials

costs; penalise those who go beyond the limit

control strategies (cont’d)

• Conduct more in-class assessments• On written assignments mark in pencil to make

alterations without time or trouble• Sort scripts into piles according to approximate standard

– helps maintain consistent standard• Scrap grades and percentages and use pass/fail instead.

Scrap degree classifications.• Refuse to accept work that is difficult to read or late

without reason. Stick to this policy• Use placement and clinical assessors – standardise

criteria and provide training

‘independence’ strategies

• ‘front-ending’ assessment• devise group tasks and assess the group – group

reach consensus on scores for individual members• Self assessment - require students to mark their

own work with comments, or submit a completed standardised feedback form with their work

• Peer assessment – ask students to use a marking scheme to mark each other’s work

What staff want are low risk, low support initiatives

1. Introductory Chemistry

‘Course puts multiple choice questions on the course’s website each week for the students to do in their own time. There are no marks for these tests, and no record is kept of who has taken them, but the students do the questions because they know from the beginning of the course that the end of module exam will include a section of MCQs and that half of these questions will have been selected from those questions used in the weekly tests’ (Rust 2003)

2. Anatomy

Students undertake regular MCQ tests but instead of simply marking their chosen response on the test sheet they also have to indicate a ‘confidence measure’ (CF), in order to minimise the element of guesswork common to many MCQs. If they indicate a high CF and get the correct answer they gain a high mark. If they indicate a high CF and get it wrong, they will be heavily penalised.

3. The Geography course• convert all examination questions from 30/40

minute to hour-long essays. • thus fewer questions, but not possible to get away

with only recall. • immediate response that quality of answers

improved and students did more reading. • number of essays also reduced throughout all

modules, with each essay requiring more work on the part of the student, and contributing more to the final grade.

• Class examinations removed from most modules and the number of exemptions given to first year students increased.

• Pass/fail examinations being used to a larger extent, and more planned.

• Project work also increased and now accounts for up to 45% of the module grade.

• prior discussion between staff and students as to what makes a good project, followed up with input in tutorials

• several options now incorporate oral assessment and are peer assessed, with half of the grade being awarded by the students.

• In feedback, some of the discussions revolve around why peer and tutor marks differ.

• Department encouraging the submission of all work in a word-processed format, though concerns have been voiced about the increasingly poor writing styles on exam scripts.

• The dissertation deadline has been changed to later in the degree programme, and the greater experience gained from options and the like has resulted in higher quality dissertations, which have been winning national prizes. (ASSHE Project 1997)

4. An Education Course

Original Course Staff Time Staff Time New Course

Lectures

(18 weeks x 2hrs)

 

Seminars

(18 wks x 9 groups of 8)

36 hrs

 

  

162hrs

36 hrs

  

 

36hrs

Workshops

(18 weeks x 2 hrs)

 

Autonomous seminars

(18 wks x 2 tutors touring 6 groups of 12)

Total teaching hours 198hrs 72hrs

Essays

(72 x 2 essays @ 20mins)

 

Examination

(72 x 2 questions @ 10 mins)

48hrs

 

 

 

24 hrs

6hrs

 

 

 

18hrs

 

 

 

36 hrs

MCQ tests

(6 x 1hrs administration)

  

Seminar presentations

(peer assessed,

1 hr/week collation)

 Portfolio (72 @ 30 mins)

Total assessment hrs 72 hrs 60 hrs

5. Using ‘Discourse’ in History

A classroom communication system (CCS)

Discourse – what the student would see

What was one effect of World War II on America? Explain.

Women worked in the factories

Discourse – Tutor’s list of responses

This is a list with the student’s name beside it. It is possible for responses to be anonymous.

The responses can be readily kept as a record.

Responses can be kept for the tutor’s view or can be shared with all in the group.

What was one effect of World War II on America? Explain.

Wu, Ken

Instead of looking at domestic affairs, America realised that it needed to be on top of world happenings

Effective uses of CCS systems

• diagnostic/ formative potential• quick-fire tests of understanding• possibilities for peer judgement• promotion of discussion • useful research tool to create an instant

teaching resource• evaluation of content / delivery • makes lectures less boring! 

Differences between ‘Discourse’ and previous generations

• used between people that are linked

• bi-directional

• web-based / doesn’t need special equipment

• questions can be asked ‘on the fly’

• open-ended questions can be asked

6. The Patchwork Text

A patchwork is not just a ‘collection’ but a ‘pattern’: in the end it does have a unity, albeit made up of separate components. The unity of the Patchwork Text has two dimensions. To begin with it is defined by academic staff, as they carefully derive a sequence of tasks from the course material. And finally it is, as it were, re-defined by individual students, who review (and perhaps edit) their separate pieces of work in order to write their final section as an interpretation of what this course materials ‘means’, to them, now…(Winter 2003)

References

Gibbs, G. & Jenkins, A. (1992 )Teaching Large Classes in Higher EducationRoutledgeFalmer, London. Lyotard, J. L. (1986) The Postmodern Condition: A Report on Knowledge, Manchester University Press, Manchester.

Rust, C. (2002) ‘The impact of assessment on stuydent learning: how can the research literature practically help to inform the development of departmental assesswmtn strategies and learner-centred assessment practices?’ Active Learning Vol3, 2 145-158 July

Winter, R. (2003) ‘Contextualizing the Patchwork Text: Addressing Problems of Coursework Assessment in Higher Education’, Innovations In Education and Teaching International, Vol 40 No 112-122

ray.land@coventry.ac.uk