XerteXerteStandard SettingStandard Setting

ininPsychiatry Undergraduate Psychiatry Undergraduate

OSCEOSCEDr Ben di MambroDr Ben di Mambro

Clinical Lecturer in PsychiatryClinical Lecturer in Psychiatry

Psychiatry ExaminationPsychiatry Examination

80 EMIs80 EMIs 1 hour 15 minutes1 hour 15 minutes Standard setStandard set

2 OSCE stations2 OSCE stations Each station 10 minutesEach station 10 minutes 3 tasks based on competencies3 tasks based on competencies % given for each task% given for each task

May OSCEMay OSCE

StationStation MaxMax MinMin AllAll

AA60.160.1

(14.3)(14.3)46.446.4

(5.6)(5.6)54.554.5

(10.3)(10.3)

BB60.8 60.8 (6.6)(6.6)

48.248.2

(4.4)(4.4)55.555.5

(9.0)(9.0)

CC66.166.1

(8.5)(8.5)49.049.0

(9.7)(9.7)56.256.2

(9.8)(9.8)

DD67.767.7

(12.8)(12.8)47.247.2

(4.1)(4.1)54.354.3

(10.8)(10.8)

Psychiatry OSCEPsychiatry OSCE

Changes to the OSCEChanges to the OSCE Criterion marking schemeCriterion marking scheme Global clinical assessmentGlobal clinical assessment

HonoursHonours PassPass BorderlineBorderline FailFail

2 examiners were placed in each room2 examiners were placed in each room Standard set pass mark for each taskStandard set pass mark for each task

TaskTask

Develop CAL package:Develop CAL package: Train the examiners in the new Train the examiners in the new

marking schememarking scheme Standardise marking schemeStandardise marking scheme

To allow for multiple sites to To allow for multiple sites to accommodate increasing numbers of accommodate increasing numbers of studentsstudents

Improve inter-rater reliabilityImprove inter-rater reliability

Development of PackageDevelopment of Package

Scripted 10x 2-3 minute OSCE Scripted 10x 2-3 minute OSCE scenariosscenarios Borderline/PassBorderline/Pass Pass/HonoursPass/Honours

2 hours filming in my office2 hours filming in my office 2 lecturers2 lecturers 1 bullied student1 bullied student

Creation of MediaCreation of Media

wmv files converted to flv using Flash wmv files converted to flv using Flash MacromediaMacromedia

Images created utilising Flash Images created utilising Flash MacromediaMacromedia

MP3 creation using AudacityMP3 creation using Audacity

XerteXerte

Template ‘ratings’ created by Julian Template ‘ratings’ created by Julian TenneyTenney

Media was imported into Xerte Media was imported into Xerte through the Wizardthrough the Wizard

The PackageThe Package

An introduction to the new marking An introduction to the new marking schemescheme

Example mark sheetsExample mark sheets Video of example studentsVideo of example students

Rated on criteriaRated on criteria Global impression givenGlobal impression given User then offered feedbackUser then offered feedback

Using the WizardUsing the Wizard

Once Wizard create – efficient Once Wizard create – efficient method of presenting contentmethod of presenting content Define scenarioDefine scenario Define marking sheet, including number Define marking sheet, including number

of criteriaof criteria Define fail/borderline/pass/honoursDefine fail/borderline/pass/honours Import mediaImport media

Sample of PackageSample of Package

Distribution of PackageDistribution of Package Letters and emails were sent to all Letters and emails were sent to all

examining consultants with following examining consultants with following linklink

www.nottingham.ac.uk/~mczbjd/OSCEwww.nottingham.ac.uk/~mczbjd/OSCE.htm.htm

Package ran for ~20 minutesPackage ran for ~20 minutes

Result of Package Result of Package IntroductionIntroduction

30% examiners accessed the 30% examiners accessed the package prior to the OSCEpackage prior to the OSCE

Remainder underwent abridged Remainder underwent abridged version before the OSCEversion before the OSCE

AnalysisAnalysis

ANOVAs undertaken on awarded ANOVAs undertaken on awarded scores by the different examiners on scores by the different examiners on the same stationthe same station

December OSCEDecember OSCE

StationStation MaxMax MinMin AllAll

AA70.370.3

(5.0)(5.0)53.453.4

(8.6)(8.6)62.362.3

(8.6)(8.6)

BB72.672.6

(9.1)(9.1)52.152.1

(8.85)(8.85)63.763.7

(11.5)(11.5)

CC66.166.1

(5.1)(5.1)51.051.0

(7.6)(7.6)56.956.9

(8.6)(8.6)

DD66.166.1

(56.7)(56.7)56.656.6

(5.0)(5.0)60.760.7

(7.4)(7.4)

**p<0.001 *p<0.05**p<0.001 *p<0.05

MayMay DecemberDecember

OSCE 1OSCE 154.5**54.5**

(10.3)(10.3)62.3**62.3**

(8.6)(8.6)

OSCE 2OSCE 256.2**56.2**

(9.77)(9.77)63.7**63.7**

(11.5)(11.5)

OSCE 3OSCE 355.5**55.5**

(9.02)(9.02)56.9**56.9**

(8.6)(8.6)

OSCE 4OSCE 454.3**54.3**

(10.8)(10.8)60.760.7

(7.4)(7.4)

VariationVariationStationStation MayMay DecDec

AA 13.713.7 16.916.9

BB 12.612.6 20.520.5

CC 17.117.1 15.015.0

DD 20.520.5 10.510.5

LimitationsLimitations

Flash Player 7 not installed as standard Flash Player 7 not installed as standard on healthcare trusts’ computerson healthcare trusts’ computers

Minor problems with Xerte - since Minor problems with Xerte - since resolvedresolved

Delay on slow broadband connectionsDelay on slow broadband connections ?Representative videos?Representative videos Any improvement likely to be due to Any improvement likely to be due to

the new marking schemethe new marking scheme

XerteXerte

Once familiar:Once familiar: Easy to useEasy to use Multimedia packages can be create Multimedia packages can be create

in minutesin minutes Easily distributedEasily distributed Much more complicated to create Much more complicated to create

bespoke packagesbespoke packages

FutureFuture

Improve IT access in the healthcare Improve IT access in the healthcare truststrusts

Reduce bitrateReduce bitrate Produce improved examples of Produce improved examples of

borderline studentsborderline students Extend the package to attempt to Extend the package to attempt to

standardise the Formative assessmentsstandardise the Formative assessments ?Make available to students?Make available to students