running an exam from beginning to end - university of bristol
TRANSCRIPT
Running an Exam From Beginning to End
Andrew Blythe
Autumn 2011
BLUEPRINTING
What are we assessing?
“The outcomes in Tomorrows Doctors (2009) provide a structure against which everything else can be planned & evaluated. Some outcomes cover more ground than others so will require more extensive testing.”
Paragraph 29, Assessment in Undergraduate Medical Education – Advice
Supplementary to Tomorrow’s Doctors (2009)
Hierarchy of Learning Outcomes
Unit Intended Learning Outcome Approved by MEC & Programme Approval
MB ChB Programme Specification University of Bristol Under revision
GMC 16 high level outcomes 32 practical procedures
Learning Outcomes for Preparing for Professional Practice (Year 5)
Examples Tomorrows
Doctors Brief description topic/learning objective
Method of Assessment
14a Practitioner
Synthesises history, examination, mental state & patient factors
Formative & summative long cases Portfolios Case-based discussion within tutorial
17a 17b Practitioner
Takes a drug history including OTC medications & complementary therapies & plan appropriate therapy for common indications, including pain & distress
Examples of appropriate drug histories in clerking portfolio with copies of original drug charts Student script countersigned Oncology/palliative care prescribing tutorial signed off
Community Orientated Medical Practice 2 (COMP2)
Blueprint for Primary Care
ROLE OF EXTERNAL EXAMINER
External Examiner from Report Received
Human Basis of Medicine Oxford Yes
Molecular & Cellular Basis of Med
Cardiff, Kings College + 2 others
Yes (2 out of 4)
Systems of the Body 1 & 2 Kings College Yes
Junior Medicine & Surgery
Cardiff + 1 other Yes (1 out of 2)
Psychiatry & Ethics Southampton & Hull-York Yes (both)
MDEMO 1 No
COMP1 Highlands + 2 Yes (2 out of 3)
Applied Clinical Sciences 2 No
Reproductive Health Care Swansea & Penninsula Yes (both)
COMP2 Kings College & Sheffield Yes (both)
Senior Surgery Oxford No
Oncology & Palliative Care No one in place
Senior Medicine Birmingham Yes
SSCs (years 3 & 5) 1 No (2nd year running)
University Guidelines
“The role of the external examiner is not confined to consideration of examination results and attendance at examination boards”.
“External examiners may, and are encouraged, to comment and advise on the content, balance and structure of the programmes and units, the development and review of programmes and/or units and on assessment processes.”
The External Examiner’s Annual Cycle
Review content & assessment of unit. Help set priorities for coming year
Comment on proposed
examination questions
Contribute to standard setting
exercise
Attend examination & scrutinize
student performance
Submit report
Report viewed by Unit Assessment Group APR Response sent to External Examiner
Planning Changes
Reviewing Questions
Standard Setting
Attends Exam
Reviews marks of failing students
HBoM
MCBoM
Systems 1
Systems 2
Junior Med & Surg OSCE
Psych & Ethics Presentations
MDEMO
COMP1 Prize viva
ACS
RHCN SSC Pres’n
COMP2 OSCEs & SSC presentation
Senior Surgery
Senior Medicine OSCE & resit long cases
Problems
Long delays in
• getting some reports
• reviewing & responding to some reports
Not involving external examiners enough in
• reviewing some units
• looking at performance of students
Examination Format: written
• Replace essays with more short answers
• Replace true/false with best-of-5
• Standardise Extended Matching Questions
• Base majority of questions on clinical topics
Examination Format: clinical
• More OSCE stations where student has to examine the patient/actor
• Standardise assessments
• Set standard for portfolios
Communication with External Examiner
• Discuss course details with him/her
• Discuss student performance with him/her
• Inform him/her of student progress
Other comments
• Concern about method of scaling marks
• Some unit leads need more support
TRAINING EXAMINERS
Doves & Hawks
STANDARD SETTING
Written Clinical
Year 1 HBoM: Team of trained examiners mark essays. Check-lists created
MCBoM: Trial standard setting group spanning all 5 academic departments in Faculty MVS
Year 2 Ebel method used for written papers Mock standard setting exercise for spot anatomy
Year 3 Ebel for Junior Medicine & Surgery Borderline Group Method for OSCE in Junior Med & Surgery
Year 4 Full implementation of Ebel in all 4 units Borderline Group Method for OSCEs.
Year 5 Full implementation of Angoff in EMQs for Finals
Angoff for DOSCE stations Borderline Regression Method for OSCE stations
SSCs
Competent
Pass
Incompetent
Pass
Competent
Fail
Incompetent
Fail
4 components to COMP2 exam
• OSCE 10 stations (each out of 10)
• MCQ (negatively marked)
• Student selected component (essay & group presentation)
• Satisfactory professional behaviour
4 components
• OSCE 10 stations (each out of 10)
• MCQ (negatively marked)
• Student selected component (essay & group presentation)
• Satisfactory professional behaviour
50%
50%
45%
Pass/fail
Marks in an OSCE
10 20 30 40 50 60 70 80 90 100
No. of students
Pass mark
Total mark in OSCE (%)
The marks of a poorly performing student
Station
A B C D E F G H I J
4 4 6 5 5 4 4.5 7 6 5 50.5
New rule:
“Students should get more than 50% overall & should not fail more than 2 stations”
Station
A B C D E F G H I J
4 4 6 5 5 4 4.5 7 6 5 50.5
FAIL
EXAMINER FORM STICK STUDENT ID LABEL HERE
Check that you have correct
student’s name
and number on the marking sheet.
This station is testing the ability of the student to
Manage a patient with mechanical low back pain Elicit the patient’s expectation of a consultation
COMPETENCE TASK SCORE
Competence task (Maximum mark) Mark ( ½ marks allowed) Comment (only if required)
Student takes a good history including the effect of the
symptoms on the patient’s work (2)
Student checks for red flag symptoms: bladder, bowels & saddle
numbness (1 ½)
Student explores patients concerns & finds out that patient was
expecting an X-ray; student explains why this is not a good
idea (2 )
Student explains natural history of low back pain (1)
Student negotiates management plan with patient. This should
using a stronger analgesic drug, encouraging patient to mobilise
and possibility of self certification/ Med3 (2)
Student gives advice on safe lifting (1 ½ )
TOTAL (out of 10)
OVERALL IMPRESSION How do YOU feel the student has performed? Please circle the description that best describes the performance of this student:
FAIL BORDERLINE
PASS
CLEAR PASS
EXAMINER FORM STICK STUDENT ID LABEL HERE
Check that you have correct
student’s name
and number on the marking sheet.
This station is testing the ability of the student to
Manage a patient with mechanical low back pain Elicit the patient’s expectation of a consultation
COMPETENCE TASK SCORE
Competence task (Maximum mark) Mark ( ½ marks allowed) Comment (only if required)
Student takes a good history including the effect of the
symptoms on the patient’s work (2) 1
Student checks for red flag symptoms: bladder, bowels & saddle
numbness (1 ½) 1
Student explores patients concerns & finds out that patient was
expecting an X-ray; student explains why this is not a good
idea (2 )
½
Student explains natural history of low back pain (1) ½
Student negotiates management plan with patient. This should
using a stronger analgesic drug, encouraging patient to mobilise
and possibility of self certification/ Med3 (2)
1½
Student gives advice on safe lifting (1 ½ ) ½
TOTAL (out of 10) 5
OVERALL IMPRESSION How do YOU feel the student has performed? Please circle the description that best describes the performance of this student:
FAIL BORDERLINE
PASS
CLEAR PASS
Borderline Group Method C
om
pet
en
ce
task
sco
re
10
6.1
0
Clear Borderline Clear Very Outstanding
Fail Pass Good
EXAMINER FORM STICK STUDENT ID LABEL HERE
Check that you have correct
student’s name
and number on the marking sheet.
This station is testing the ability of the student to
Manage a patient with mechanical low back pain Elicit the patient’s expectation of a consultation
COMPETENCE TASK SCORE
Competence task (Maximum mark) Mark ( ½ marks allowed) Comment (only if required)
Student takes a good history including the effect of the
symptoms on the patient’s work (2)
Student checks for red flag symptoms: bladder, bowels & saddle
numbness (1 ½)
Student explores patients concerns & finds out that patient was
expecting an X-ray; student explains why this is not a good
idea (2 )
Student explains natural history of low back pain (1)
Student negotiates management plan with patient. This should
using a stronger analgesic drug, encouraging patient to mobilise
and possibility of self certification/ Med3 (2)
Student gives advice on safe lifting (1 ½ )
TOTAL (out of 10)
OVERALL IMPRESSION How do YOU feel the student has performed? Please circle the description that best describes the performance of this student:
FAIL BORDERLINE
PASS
CLEAR PASS VERY GOOD OUTSTANDING
EXAMINER FORM STICK STUDENT ID LABEL HERE
Check that you have correct
student’s name
and number on the marking sheet.
This station is testing the ability of the student to
Manage a patient with mechanical low back pain Elicit the patient’s expectation of a consultation
COMPETENCE TASK SCORE
Competence task (Maximum mark) Mark ( ½ marks allowed) Comment (only if required)
Student takes a good history including the effect of the
symptoms on the patient’s work (2) 1
Student checks for red flag symptoms: bladder, bowels & saddle
numbness (1 ½) 1
Student explores patients concerns & finds out that patient was
expecting an X-ray; student explains why this is not a good
idea (2 )
½
Student explains natural history of low back pain (1) ½
Student negotiates management plan with patient. This should
using a stronger analgesic drug, encouraging patient to mobilise
and possibility of self certification/ Med3 (2)
1½
Student gives advice on safe lifting (1 ½ ) ½
TOTAL (out of 10) 5
OVERALL IMPRESSION How do YOU feel the student has performed? Please circle the description that best describes the performance of this student:
FAIL BORDERLINE
PASS
CLEAR PASS VERY GOOD OUTSTANDING
Borderline Regression Method
Clear Borderline Clear Very Outstanding
Fail Pass Good
Co
mp
ete
nce
ta
sk s
core
10
6.1
0
EXAMINER FORM STICK STUDENT ID LABEL HERE
Check that you have correct
student’s name
and number on the marking sheet.
This station is testing the ability of the student to
Manage a patient with mechanical low back pain Elicit the patient’s expectation of a consultation
COMPETENCE TASK SCORE
Competence task (Maximum mark) Mark ( ½ marks allowed) Comment (only if required)
Student takes a good history including the effect of the
symptoms on the patient’s work (2) 1
Student checks for red flag symptoms: bladder, bowels & saddle
numbness (1 ½) 1
Student explores patients concerns & finds out that patient was
expecting an X-ray; student explains why this is not a good
idea (2 )
½
Student explains natural history of low back pain (1) ½
Student negotiates management plan with patient. This should
using a stronger analgesic drug, encouraging patient to mobilise
and possibility of self certification/ Med3 (2)
1½
Student gives advice on safe lifting (1 ½ ) ½
TOTAL (out of 10) 5
OVERALL IMPRESSION How do YOU feel the student has performed? Please circle the description that best describes the performance of this student:
FAIL BORDERLINE
PASS
CLEAR PASS VERY GOOD OUTSTANDING
Competence Task Score of borderline students at one station
5½ 6 6½ 7
Competence Score out of 10
No. of students
7
6
5
4
3
2
1
Mean = 6.14
Extract of OSCE Spreadsheet showing 10 students’ marks & the pass mark calculated by Borderline
Group Method
StudentStation
A
Station
B
Station
C
Station
D
Station
E
Station
F
Station
G
Station
H
Station
I
Station
J
TOTAL
(out of 100)
a 9.5 5 7 6.5 5.5 6 8 7.5 8.5 7 70.5
b 5 6 8 7.5 2.5 6 6.5 5.5 6.5 9 62.5
c 7.5 7.5 8 6.5 4 7.5 5.5 6.5 6 8 67
d 5.5 6 4.5 5.5 1 6 8 5.5 7 9 58
e 5.5 5 5.5 6.5 5 5 6.5 8 4.5 6 57.5
f 9.5 7 7 5.5 6 6 8 7 6.5 4 66.5
g 9.5 6 7 7.5 8 8 6 6.5 8 8.5 75
h 7 6.5 7 7 6 5.5 5 8 6.5 5.5 64
I 10 5.5 8 8.5 4 9.5 7 6.5 7 7.5 73.5
j 9 7 6 6 5.5 7 8 7 6.5 7 69
Pass
Mark 5.68 6.4 5.81 7 5 6.14 5.82 6.13 6 5.82 59.8
Training package for OSCE examiners
A DVD showing students perform to different levels at same station
Marks from a written exam
10 20 30 40 50 60 70 80 90 100
No. of students
50% Pass mark
“Moderating” (lowering the pass mark to 45%)
10 20 30 40 50 60 70 80 90 100
No. of students
45% Pass mark
Ebel Table
Easy Medium Hard
Essential
Important
Acceptable
Ebel Table
Easy Medium Hard
Essential 90% 80% 70%
Important 60% 50% 40%
Acceptable 30% 20% 10%
Ebel Table
Easy Medium Hard
Essential 1a, 1b 1c 1d
Important 1e
Acceptable
Ebel Table for Extended Matching Questions that have 10 options
Easy Medium Hard
Essential 75% 55% 45%
Important 60% 50% 30%
Acceptable 40% 25% 10%
Ebel Method
Category Number of
Questions in
this category
Proportion of time
minimally competent
student will get this
question correct (%)
Score expected
from minimally
competent
student
Essential Easy 4 75 3.00
Medium 3 55 1.65
Hard 1 45 0.45
Important Easy 4 60 2.40
Medium 5 50 2.50
Hard 2 30 0.6
Acceptable Easy 1 40 0.40
Medium 35 0
Hard 10 0
Total 20 11
Ebel Method
Category Number of
Questions in
this category
Proportion of time
minimally competent
student will get this
question correct (%)
Score expected
from minimally
competent student
Essential Easy 26 75 19.5
Medium 31 55 17.05
Hard 2 45 0.90
Important Easy 32 60 19.20
Medium 33 50 16.5
Hard 11 30 3.30
Acceptable Easy 1 40 0.40
Medium 3 25 0.75
Hard 1 10 0.10
Total 140 77.4
Divided by number of judges (7)
Total 20 11.1
(55.5%)
Summary
Borderline Group Method for OSCE
Ebel Method for written (EMQ or best of 5)
Both
Built on concept of minimally competent student
Use panel of judges to assess each question
The final marks S
tud
en
t
SS
C /20
MC
Q /130
EM
Q/5
0
EM
Q /100
Written
/230
Written a
s
% Wri
tten
/35
OS
CE
as
% OS
CE
/45
To
tal
a 14 79 40 80 159 69.1 24.2 60.7 27.3 65.5
b 17 82 32 64 146 63.5 22.2 52.7 23.7 62.9
c 15.4 69 32 64 133 57.8 20.2 57.2 25.7 61.4
d 12 54 31 62 116 50.4 17.7 46.7 21.0 50.7 Fail
e 14 68 32 64 132 57.4 20.1 47.7 21.5 55.6 Fail
f 18.4 87 41 82 169 73.5 25.7 56.7 25.5 69.6 Merit
g 14.2 91 40 80 171 74.3 26.0 65.2 29.3 69.6 Merit
h 15.6 69 35 70 139 60.4 21.2 54.2 24.4 61.1
I 18.6 79 36 72 151 65.7 23.0 63.7 28.7 70.2 Merit
j 14.2 80 37 74 154 67.0 23.4 59.2 26.6 64.3
HANDLING MARKS
Spreadsheets & Scaling
EXAM BOARDS
GIVING FEEDBACK TO STUDENTS ON THEIR PERFORMANCE