E-assessment: a risk-based approach to success

Dr Chris Ricketts, Sub-Dean (Teaching Enhancement), Faculty of

Technology, University of Plymouth

and

Director of Assessment, Peninsula College of Medicine and Dentistry

Overview

history institutional strategy risk analysis examples of success discussion

History (1)

over 20,000 students 6 faculties 6 sites over 150 miles

some use of CAA on local servers

History (2)

Needs analysis in April 1999 12 staff, 5 faculties, over 3,000 students Strategic introduction of CAA

(University-wide)

Strategy (1)

Zakrzewski’s (1999) ‘Catherine-wheel’ model Module > Department > Faculty > University

We chose University-wide availability, but controlled pilot use in modules undertook risk analysis roll-out to University

Strategy (2)

What to pilot? formative assessment summative (in-course) assessment end of year examinations a variety of subject areas

Strategy (3)

Decision 1: A steering group which involved staff from all affected parts of the University

Decision 2: Academic Board support Decision 3: Staff training and support Decision 4: Easy introduction to students Decision 5: Get student feedback

- and act on it

Strategy (4)

Decision 6: the big one! Don’t treat computer aided assessments

differently

management quality processes security

General experience

bed of roses?…not we did prepare carefully and try to consider

all risks

Risk analysis (Ricketts & Zakrzewski, AEHE 2005)

Types of risk Pedagogic (P) Operational (O) Technical (T) Web-based (W) Financial (F)

Risk analysis

Define the risks Estimate likelihood Estimate severity (who is affected?) Concentrate on the severe problems How to avoid What if it happens?

Defining the risks

The literature can help, but…

… these need to be YOUR risks, not someone else’s

Estimate likelihood Estimate severity (who is affected?) Concentrate on the real problems How to avoid What if it happens?

Risk example 1

P1: Assessment method not integrated into the curriculum

Likelihood? M Who affected? Students, Academic staff How much? Module When? Before assessment

This was not a high severity risk for us.

Risk example 2

O7: Module size too large for number of workstations available

Likelihood? H Who affected? Students, Academic staff,

Support staff How much? University When? Before assessment

This was the most high severity risk for us.

Risk management example

O17: Different invigilation requirements for CAA not recognized

Liaise with examinations office Produce guidelines for invigilators Ensure support team in exam includes

technical support staff Academic staff to be present at start of on-

line examination

Issues

load testing the system for large-scalesummative assessments

link between student records and assessment databases

student computing mistakes adequate computer facilities on all sites wayward staff

Some findings (from Roy Lowry) Introduction of formative tests each week

after lecture Students are motivated enough to use the

system Although requiring some effort to set up, the

system can be re-used CAA in a formative mode has a significant

impact upon learning

Next step

Expanded to cover all material Use of MC, MR, drop-down boxes and

numeric questions

Used for the end-of-module test 103 students in Babbage open access area Students obtained their marks immediately

after they finished the test

Student use 30% of students attempted all of the tests

(average mark: 72%) 65% attempted some of the tests

(average mark: 53%) 5% did not use the system

(average mark: 45%)

Benefits

Benefits for staff... easier to give frequent feedback no marking!

Benefits for students… more frequent feedback instant marking more self-assessment

“Cost effectiveness”

Thanks to James Wisdom (ELEN Conference 2000) for the following

Cost benefit analysis

Useful when benefits can be expressed in monetary termseg.Saving in staff time (hence money)etc.

Is this why we use CAA?

Cost effectiveness

Useful when outcomes cannot be expressed in monetary terms

looks at outcomes in relation to goal “on time, on budget, to quality”

Is this why we use CAA?

Pedagogic effectiveness

Learners learning,and learning better Must be part of cost effectiveness Is this why we use CAA??

Questions

Over to you