XerteXerteStandard SettingStandard Setting

ininPsychiatry Undergraduate Psychiatry Undergraduate

OSCEOSCEDr Ben di MambroDr Ben di Mambro

Clinical Lecturer in PsychiatryClinical Lecturer in Psychiatry

Psychiatry ExaminationPsychiatry Examination 80 EMIs80 EMIs

1 hour 15 minutes1 hour 15 minutes Standard setStandard set

2 OSCE stations2 OSCE stations Each station 10 minutesEach station 10 minutes 3 tasks based on competencies3 tasks based on competencies % given for each task% given for each task

May OSCEMay OSCEStationStation MaxMax MinMin AllAll

AA 60.160.1(14.3)(14.3)

46.446.4(5.6)(5.6)

54.554.5(10.3)(10.3)

BB 60.8 60.8 (6.6)(6.6)

48.248.2(4.4)(4.4)

55.555.5(9.0)(9.0)

CC 66.166.1(8.5)(8.5)

49.049.0(9.7)(9.7)

56.256.2(9.8)(9.8)

DD 67.767.7(12.8)(12.8)

47.247.2(4.1)(4.1)

54.354.3(10.8)(10.8)

Psychiatry OSCEPsychiatry OSCE Changes to the OSCEChanges to the OSCE

Criterion marking schemeCriterion marking scheme Global clinical assessmentGlobal clinical assessment

HonoursHonours PassPass BorderlineBorderline FailFail

2 examiners were placed in each room2 examiners were placed in each room Standard set pass mark for each taskStandard set pass mark for each task

TaskTaskDevelop CAL package:Develop CAL package: Train the examiners in the new Train the examiners in the new

marking schememarking scheme Standardise marking schemeStandardise marking scheme

To allow for multiple sites to To allow for multiple sites to accommodate increasing numbers of accommodate increasing numbers of studentsstudents

Improve inter-rater reliabilityImprove inter-rater reliability

Development of PackageDevelopment of Package Scripted 10x 2-3 minute OSCE Scripted 10x 2-3 minute OSCE

scenariosscenarios Borderline/PassBorderline/Pass Pass/HonoursPass/Honours

2 hours filming in my office2 hours filming in my office 2 lecturers2 lecturers 1 bullied student1 bullied student

Creation of MediaCreation of Media wmv files converted to flv using Flash wmv files converted to flv using Flash

MacromediaMacromedia Images created utilising Flash Images created utilising Flash

MacromediaMacromedia MP3 creation using AudacityMP3 creation using Audacity

XerteXerte Template ‘ratings’ created by Julian Template ‘ratings’ created by Julian

TenneyTenney Media was imported into Xerte Media was imported into Xerte

through the Wizardthrough the Wizard

The PackageThe Package An introduction to the new marking An introduction to the new marking

schemescheme Example mark sheetsExample mark sheets Video of example studentsVideo of example students

Rated on criteriaRated on criteria Global impression givenGlobal impression given User then offered feedbackUser then offered feedback

Using the WizardUsing the Wizard Once Wizard create – efficient Once Wizard create – efficient

method of presenting contentmethod of presenting content Define scenarioDefine scenario Define marking sheet, including number Define marking sheet, including number

of criteriaof criteria Define fail/borderline/pass/honoursDefine fail/borderline/pass/honours Import mediaImport media

Sample of PackageSample of Package

Distribution of PackageDistribution of Package Letters and emails were sent to all Letters and emails were sent to all

examining consultants with following examining consultants with following linklink

www.nottingham.ac.uk/~mczbjd/OSCEwww.nottingham.ac.uk/~mczbjd/OSCE.htm.htm

Package ran for ~20 minutesPackage ran for ~20 minutes

Result of Package Result of Package IntroductionIntroduction

30% examiners accessed the 30% examiners accessed the package prior to the OSCEpackage prior to the OSCE

Remainder underwent abridged Remainder underwent abridged version before the OSCEversion before the OSCE

AnalysisAnalysis ANOVAs undertaken on awarded ANOVAs undertaken on awarded

scores by the different examiners on scores by the different examiners on the same stationthe same station

December OSCEDecember OSCEStationStation MaxMax MinMin AllAll

AA 70.370.3(5.0)(5.0)

53.453.4(8.6)(8.6)

62.362.3(8.6)(8.6)

BB 72.672.6(9.1)(9.1)

52.152.1(8.85)(8.85)

63.763.7(11.5)(11.5)

CC 66.166.1(5.1)(5.1)

51.051.0(7.6)(7.6)

56.956.9(8.6)(8.6)

DD 66.166.1(56.7)(56.7)

56.656.6(5.0)(5.0)

60.760.7(7.4)(7.4)

**p<0.001 *p<0.05**p<0.001 *p<0.05

MayMay DecemberDecember

OSCE 1OSCE 1 54.5**54.5**(10.3)(10.3)

62.3**62.3**(8.6)(8.6)

OSCE 2OSCE 2 56.2**56.2**(9.77)(9.77)

63.7**63.7**(11.5)(11.5)

OSCE 3OSCE 3 55.5**55.5**(9.02)(9.02)

56.9**56.9**(8.6)(8.6)

OSCE 4OSCE 4 54.3**54.3**(10.8)(10.8)

60.760.7(7.4)(7.4)

VariationVariationStationStation MayMay DecDec

AA 13.713.7 16.916.9

BB 12.612.6 20.520.5

CC 17.117.1 15.015.0

DD 20.520.5 10.510.5

LimitationsLimitations Flash Player 7 not installed as standard Flash Player 7 not installed as standard

on healthcare trusts’ computerson healthcare trusts’ computers Minor problems with Xerte - since Minor problems with Xerte - since

resolvedresolved Delay on slow broadband connectionsDelay on slow broadband connections ?Representative videos?Representative videos Any improvement likely to be due to the Any improvement likely to be due to the

new marking schemenew marking scheme

XerteXerteOnce familiar:Once familiar: Easy to useEasy to use Multimedia packages can be create Multimedia packages can be create

in minutesin minutes Easily distributedEasily distributed Much more complicated to create Much more complicated to create

bespoke packagesbespoke packages

FutureFuture Improve IT access in the healthcare Improve IT access in the healthcare

truststrusts Reduce bitrateReduce bitrate Produce improved examples of Produce improved examples of

borderline studentsborderline students Extend the package to attempt to Extend the package to attempt to

standardise the Formative assessmentsstandardise the Formative assessments ?Make available to students?Make available to students