opening up multiple choice - assessing with confidence

eST

EeM

Opening up Multiple Choice: Assessing with confidence

CALRG Conference, June 2011

Jon RosewellJ.P.Rosewell@open.ac.uk

eST

EeM

Multiple Choice Questions (MCQ)• Basic fare of e-assessment:

– robust– easy to implement– but pedagogically not ideal

• Open questions– pedagogically preferable – but difficult to provide robust and valid marking

eST

EeM

Demo• Standard MCQ• Standard CBM• Open CBM

screen dump

eST

EeM

eST

EeM

Confidence-Based Marking (CBM)• Student selects:

– answer and – level of confidence/certainty

• Incentive is scoring:– Confidently correct: Full marks– A tentative correct answer: Some credit– Cocksure & wrong: Penalty

• Pays to be honest about own knowledge

eST

EeM

CBM scoring

Confidence

Score

Correct Wrong

Low 1 0

Medium 2 -2

High 3 -6

Tentative & correct

Confidently correct

Cocksure – and wrong!

eST

EeM

CBM motivations• Rewards care and effort• Greater engagement• Encourages reflective learning• Encourages self-assessment

eST

EeM

Demo• Standard MCQ• Standard CBM• Open CBM

screen dump

eST

EeM

eST

EeM

Open CBM• Two stage question:

– Initially:• Question (but no MCQ options)• Student sets confidence level

– Next:• MCQ options revealed

• Finally:– Marking and feedback as for normal MCQ / CBM

eST

EeM

Demo• Standard MCQ• Standard CBM• Open CBM

screen dump

eST

EeM

eST

EeM

Open CBM advantages• Mechanically, question remains simple MCQ

– answer matching is trivial– easy to implement– possible to reuse existing question banks

• But transformed closed to open question– student must formulate answer to decide confidence– must decide answer without +ve or –ve clues.– cannot work backward – will not be led into misconceptions.

must answer the question as set.

eST

EeM

MCQ Open CBM• Pros

– objective marking– reliable marking– easy to implement

• Cons– distractors trivial– distractors engender

misconceptions– working backwards

• Pros– open question– reflection– ...as MCQ

• Cons– not always applicable– intimidating?– personality dependent?

eST

EeM

Possible benefits• Students

– better engagement– improved learning– more reflective learners

• Staff– easier question setting

eST

EeM

eSTEeM project plan (revised…)• Implement open CBM question type

– in OU Moodle Quiz and/or OpenMark• Trial in course assessment

– controlled experimental design– pilot (formative) on T216 CISCO networking

• Survey/interview to probe attitudinal aspects• User-lab trials• Log measures

– Assignment scores– Time spent on task

Jon RosewellJ.P.Rosewell@open.ac.uk

www.open.ac.uk