Re-engineering Assessment Practices [REAP] in Higher Education

David Nicol, Project Director, REAPDeputy-Director (Research & Development in e-learning)Centre for Academic Practice and Learning Enhancement (CAPLE), University of Strathclyde

Jenny Booth, Learning Technology Adviser, REAP.(www.reap.ac.uk)

CAA Conference, Loughborough, 4th July 2006

Plan

What is the REAP project? Case study – assessment redesign in first year class Pedagogical focus Promoting transformational change in assessment

at institutional level

SFC Transformation programme

Re-engineering Assessment Practices [REAP] project is one of the six funded

E-learning transformational projects New approaches to teaching/ learning Embedding and substitution Measurable benefits to institution/sector Partnerships with other HEIs/FEIs

Rationale for the REAP proposal

Assessment – key driver of student learning

Assessment is a major cost in HE: economies of scale limited

Assessment influences a wide range of organisational, pedagogical and business processes in HE

The REAP Project

3 HEIs (Strathclyde, Glasgow Caledonian Business School, Glasgow University)

Focus is on large 1st year classes Assessment for learner self-regulation Range of technologies: online tests, simulations,

discussion boards, e-voting, e-portfolios, peer/feedack software, admin systems, VLEs, offline-online

Learning quality and efficiencies Outputs: re-design case studies: advice/tools to support

strategic change in institution (transformation), sharing

Case Example

Two recent literature reviews on assessment (synthesis)

Case study of first year assessment redesign (drawn from REAP project)

Analysis of a case study in relation to assessment principles

Discuss issues

First Year: The academic experience

What is important? Coping with transition Understanding what is required Engagement with academic programmes Support and feedback Experiences of success Agents of own learning Belief in self (ability) and motivation Social dynamics of learning (belonging)Mantz Yorke (UK) and Vincent Tinto’s (US) research

Gibbs and Simpson (2004)

Literature review:

Gibbs, G. & Simpson, C (2004) Conditions under which assessment supports students learning, Learning and Teaching in Higher Education, 1, 3-31.

See: Formative Assessment in Science Teaching (FAST) project at: http://www.open.ac.uk/science/fdtl/

Gibbs and Simpson (2004)

Assessment tasks [Conditions 1-4]

1. Capture enough study time (in and out of class)

2. Are spread out evenly across timeline of study

3. Lead to productive activity (deep vs surface)

4. Communicate clear and high expectations

Gibbs and Simpson (2004)

Feedback [Conditions 5-11]

5. Is sufficient (in frequency; detail)

6. Is provided quickly enough to be useful

7. Focuses on learning rather than on marks

8. Is linked to assessment criteria/expected outcomes

9. Makes sense to students

10. Is received by students and attended to

11. Is acted upon, to improve work and/or learning

Nicol and Macfarlane-Dick (2006)

Literature review:

Nicol, D. & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 34 (1), 199-218

See: Student Enhanced Learning through Effective

Feedback [SENLEF] project funded by HE Academy REAP project: www.reap.ac.uk

Research on Assessment in HE

Teaching/learning paradigmTransmission

Assessment paradigm

Transmission

[teacher-centred]

Constructivist [student-centred]

Some key research findings

Students are always engaged in self-assessment/self-regulation of their own learning (Black & Wiliam, 2005). Logically entailed by constructivist ideas (Winne, 2005)

The act of using teacher feedback implies that self-assessment must be present (Sadler, 1983, Nicol and Macfarlane-Dick, 2006)

Feedback in HE is being significantly reduced so how are students still learning?

The question is: how can we scaffold students learning so they become better at self-regulation? (Lajoie, 2005)

Scaffolding self regulation: 7 principles of good feedback

1. Clarify what good performance is (goals, criteria, standards).

2. Facilitate reflection and self-assessment in learning 3. Deliver high quality feedback to students: feedback

that enables students to monitor and self-correct4. Encourage peer and tutor dialogue around learning5. Encourage positive motivational beliefs & self esteem

through assessment6. Provide opportunities to close the feedback loop7. Use feedback information to shape teaching

Source: Nicol and Macfarlane-Dick (2006) Formative assessment and self-regulated learning: A model and seven principles of good feedback practice, Studies in Higher Education

Re-engineering Assessment Practices [REAP]

Psychology Case Study

Context: 560 first year students Mixture of psychology majors (130) and those

taking psychology only for one year (430) 6 topic areas, 48 lectures, 4 tutorials, 12 practicals Assessment; 2 x MCQs (25%), tutorial attendance

(4%), taking part in experiment (5%), essay exam (66%)

Psychology Case Study

Problems identified with the course: Students got no practice in writing skills but

required in the exam No feedback except on MCQs (percent correct) Didn’t want to increase staff workload Wanted to improve overall exam marks And standard of entrant to second year

Stage 1: Question 1: moderate difficulty (50 words)

Individual response – post it –discuss answer in groups (of 6)

Timed release: model answer to self-evaluate their response

Stage 2: Question 2: difficult (100 words)

Group response – discuss (online) – agree – post response

Model answer released for stage 2

Stage 3: Question 3: complex (300 word essay)

Group response – discuss (online) – agree – post

Model answer released for stage 3

Psychology re-design

Psychology

Task 1: Define and describe structural encoding, phonological encoding and semantic encoding. Provide an example of each construct. (50 words, individual)

Task 2: Describe the serial position effect and its two separate components. Discuss the specific structural components of memory that are responsible. (100 word, group response)

Task 3: Summarise the ‘stage theory’ of memory. To what extent does it provide an adequate theory of memory? (300 word, group response)

Psychology re-design: key features

Writing tasks related to current lecture delivery (on human memory).

78 volunteers – induction task Question of increasing difficulty and length of

answer (50 words, 100 words and 300 words) Moving from individual to group responses. ‘Scaffolding’ through peer discussion Cognitive growth thro conflict resolution Comparing work against model answers Use of assignment/ discussion tools in (WebCT)

Planned developments

Replace half the lectures with discussion tasks Tasks will become progressively more demanding

within and across topics as the year progresses. Participation will be compulsory and monitored Model answers selected from students reponses Once a final group response has been agreed each

student will submit a copy to VLE (WebCT) Purpose is to check on participation rates and

ensure that all students have at least ‘had sight’ of the group response.

Source: Jim Baxter, Psychology, Strathclyde University

The seven feedback principles

Standard format and model answers provide progressive clarification of expectations (clear goals, principle 1)

Students encouraged to self-assess against model answer (self-assessment, principle 2)

Online peer discussion aimed at reaching consensus about response (dialogue, principle 4)

Staged complexity and focus on learning rather marks (motivation, principle 5)

Repeated cycle of topics and tasks (closing gap, principle 6)

Tutors can monitor progress and adapt (shaping teaching, principle 7)

Gibbs & Simpson’s four assessment conditions

The individual and group responses require regular study out of class (assessments capture sufficient study time, condition 1)

Tasks are distributed across topics and weeks (are spread out evenly, condition 2)

The staged questions require progressively deeper levels of understanding (productive/ deep learning, condition 3)

There are explicit goals and a progressive increase in challenge (communicates clear and high expectations, condition 4)

What can we learn from this case study?

Use of a simple technology (discussion board) Considerable thought gone into the pedagogical

design [which is transferable]. Supported by robust assessment/learning

principles Coordinated approach within the department Evaluation showed a lot of supportive

socialisation during induction task Intention to compare year on year examination

performance.

Bigger Picture

In the early years there is a need to balance opportunities for self-regulation within organised academic structures and activities.

Transformation requires whole course redesign Design for and evaluation of pedagogical

effectiveness. Transformational change = institutional strategic

level - economic models for first year teaching, (full economic), cost to change, management of change and its evaluation (benchmarking e-learning).

Questions and discussion

The context of use of online MCQ tests

P1. Students construct MCQs (creating criteria) P2. Open book scenario with MCQs or confidence

ratings in MCQs (self-correction/ reflection)P3. Results of tests discussed in tutorials (enhanced

teacher feedback)P4. Unique MCQs per student with sharing (peer

dialogue)P5. Repeated opportunities to take MCQ tests leading

to final summative test (motivating)P6. Students repeating MCQs (closing loop)P7. MCQs before lecture (just in time teaching)

Other relevant papers

Nicol, D (accepted for publication), Laying the foundation for lifelong learning: cases studies of technology supported assessment processes in large first year classes, British Journal of Educational Technology.

Nicol, D. & Milligan, C. (2006), Rethinking technology-supported assessment in relation to the seven principles of good feedback practice. In C. Bryan and K. Clegg, Innovations in Assessment, Routledge.