does the mandatory postgraduate uk surgical exam predict ... · basic surgical examinations (icbse)...
TRANSCRIPT
![Page 1: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/1.jpg)
Does the mandatory postgraduate UK
surgical exam predict selection into higher
speciality training?
Duncan Scrimgeour, J Cleland, A Lee, G Griffiths, A McKinley, C Marx and P Brennan
Intercollegiate Research Fellow
General surgery StR, Aberdeen Royal Infirmary and University of Aberdeen
![Page 2: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/2.jpg)
Membership of the Royal College
of Surgeons (MRCS) exam
One of the largest postgraduate surgical exams in the world
Designed to safeguard patients and ensure high standards
Prerequisite for completion of core surgical training and entry into higher speciality training
![Page 3: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/3.jpg)
Membership of the Royal College
of Surgeons (MRCS) exam
One of the largest postgraduate surgical exams in the world
Designed to safeguard patients and ensure high standards
Prerequisite for completion of core surgical training and entry into higher speciality training
![Page 4: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/4.jpg)
Membership of the Royal College
of Surgeons (MRCS) exam
One of the largest postgraduate surgical exams in the world
Designed to safeguard patients and ensure high standards
Prerequisite for completion of core surgical training and entry into higher speciality training
![Page 5: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/5.jpg)
MRCS
Part A – written papers
Part B – OSCE
Both can be sat from FY1 onwards
![Page 6: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/6.jpg)
MRCS (Part A)
Two 2-hour papers (135 marks each)
Paper 1 Basic sciences
MCQs
Paper 2 Principles of surgery in general
SBAs
![Page 7: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/7.jpg)
MRCS (Part B)
18 manned stations (9-minutes each) Total marks 360
Applied skills
Knowledge
Two broad content areas
![Page 8: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/8.jpg)
Reliable - reproducible
Valid - how well an assessment measures what it claims to assess
High-stakes examinations
![Page 9: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/9.jpg)
High-stakes examinations
Reliable - reproducible
Valid - how well an assessment measures what it claims to assess
![Page 10: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/10.jpg)
Predictive validity
“able to predict future performance
in the specified domain”
(P.D Van Hove et al 2010)
![Page 11: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/11.jpg)
Intercollegiate Committee for
Basic Surgical Examinations
(ICBSE)
Responsible for continued development, quality assurance and standards of MRCS in the UK
Produces an annual report highlighting the exam’s reliability
Yet, unlike other postgraduate UK exams (e.g. MRCP, MRCGP) there has been no analysis of the predictive validity of MRCS
![Page 12: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/12.jpg)
Intercollegiate Committee for
Basic Surgical Examinations
(ICBSE)
Responsible for continued development, quality assurance and standards of MRCS in the UK
Produces an annual report highlighting the exam’s reliability
Yet, unlike other postgraduate UK exams (e.g. MRCP, MRCGP) there has been no analysis of the validity or predictive validity of MRCS
![Page 13: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/13.jpg)
National selection
Given that MRCS is a perquisite for progression to higher speciality training in the UK
Determining the relationship between MRCS scores and outcomes in national selection is crucial
One of the largest surgical specialities is general surgery
![Page 14: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/14.jpg)
National selection
Given that MRCS is a perquisite for progression to higher speciality training in the UK
Determining the relationship between MRCS scores and outcomes in national selection is crucial
One of the largest surgical specialities is general surgery
![Page 15: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/15.jpg)
Aim
To examine the predictive validity of MRCS in relation to
outcomes in the general and vascular surgery national
selection process
![Page 16: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/16.jpg)
National selection process General and vascular surgery (ST3)
Candidates are assessed each year on eight components
Academic, leadership and team working
Communication skills and clinical skills p
Clinical management, technical skills and teaching p
Audit and portfolio
![Page 17: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/17.jpg)
National selection process General and vascular surgery (ST3)
Candidates are assessed each year on eight components
• Academic, leadership and team working
Communication skills and clinical skills p
Clinical management, technical skills and teaching p
Audit and portfolio
![Page 18: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/18.jpg)
National selection process General and vascular surgery (ST3)
Candidates are assessed each year on eight components
• Academic, leadership and team working
• Communication skills and clinical skills p
Clinical management, technical skills and teaching p
Audit and portfolio
![Page 19: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/19.jpg)
National selection process General and vascular surgery (ST3)
Candidates are assessed each year on eight components
• Academic, leadership and team working
• Communication skills and clinical skills p
• Clinical management and technical skills teaching p
Audit and portfolio
![Page 20: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/20.jpg)
National selection process General and vascular surgery (ST3)
Candidates are assessed each year on eight components
• Academic, leadership and team working
• Communication skills and clinical skills p
• Clinical management and technical skills teaching p
• Audit and portfolio
![Page 21: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/21.jpg)
Methods
We included all UK graduates who attempted the selection process from 2011-2015
Cross-linked selection score with MRCS database
- Part A & B scores, number of attempts, date of graduation, date of exam
- Self-declared socio-demographics (gender, ethnicity, DOB, first language)
![Page 22: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/22.jpg)
Methods
Pearson correlation coefficients to examine the linear
relationship between each assessment
Multiple linear regression analysis to identify potential independent predictors of national selection ST3 score
![Page 23: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/23.jpg)
1458 interviewed from May 2011 to May 2015
For the general and vascular surgery national selection process
![Page 24: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/24.jpg)
1458 interviewed from May 2011 to May 2015
For the general and vascular surgery national selection process
1231
227 could not be matched
![Page 25: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/25.jpg)
1458 interviewed from May 2011 to May 2015
For the general and vascular surgery national selection process
1231 305 non-UK medical graduates 9 had no GMC number 4 were duplicate names
136 Old MRCS (Parts 1, 2 and 3)
457 excluded
913
777 New MRCS
(Parts A and B)
3 missing Part A
= excluded
227 could not be matched
![Page 26: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/26.jpg)
1458 interviewed from May 2011 to May 2015
For the general and vascular surgery national selection process
1231
227 could not be matched
305 non-UK medical graduates 9 had no GMC number 4 were duplicate names
136 Old MRCS (Parts 1, 2 and 3)
457 excluded
913
777 New MRCS
(Parts A and B)
3 missing Part A
= excluded
774 matched
with national selection scores
![Page 27: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/27.jpg)
Part A score and selection first attempt score
r=0.21, n=774, p<0.001
Nat
ion
al s
ele
ctio
n s
core
(%
)
Part A passing score (%)
![Page 28: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/28.jpg)
Part B score and selection first attempt score
Part B passing score (%)
Nat
ion
al s
ele
ctio
n s
core
(%
) r=0.32, n=774, p<0.001
![Page 29: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/29.jpg)
Variable
Unstandardised Coefficients
P value
B Std. Error
CONSTANT 69.67 0.97 <0.001
Part B MRCS - % above the pass mark
Score 0.28 0.04 <0.001
Number of Part B MRCS attempts Reference - One attempt Two attempts -2.93 0.79 <0.001
≥3 attempts -8.11 1.31 <0.001
Ethnicity Reference - White British
Asian -1.77 0.75 0.019
Other -1.30 1.02 0.201
Black -1.08 1.74 0.536
Model R2 =0.18, n= 598
Multiple linear regression analysis – ST3 score
![Page 30: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/30.jpg)
Model R2 =0.18, n= 598
Variable
Unstandardised Coefficients
P value
B Std. Error
CONSTANT 69.67 0.97 <0.001
Part B MRCS - % above the pass mark
Score 0.28 0.04 <0.001
Number of Part B MRCS attempts Reference - One attempt Two attempts -2.93 0.79 <0.001
≥3 attempts -8.11 1.31 <0.001
Ethnicity Reference - White British
Asian -1.77 0.75 0.019
Other -1.30 1.02 0.201
Black -1.08 1.74 0.536
Multiple linear regression analysis – ST3 score
![Page 31: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/31.jpg)
1% of variance
Model R2 =0.18, n= 598
Variable
Unstandardised Coefficients
P value
B Std. Error
CONSTANT 69.67 0.97 <0.001
Part B MRCS - % above the pass mark
Score 0.28 0.04 <0.001
Number of Part B MRCS attempts Reference - One attempt Two attempts -2.93 0.79 <0.001
≥3 attempts -8.11 1.31 <0.001
Ethnicity Reference - White British
Asian -1.77 0.75 0.019
Other -1.30 1.02 0.201
Black -1.08 1.74 0.536
Multiple linear regression analysis – ST3 score
![Page 32: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/32.jpg)
7%
10% Model R2 =0.18, n= 598
of variance
17%
Variable
Unstandardised Coefficients
P value
B Std. Error
CONSTANT 69.67 0.97 <0.001
Part B MRCS - % above the pass mark
Score 0.28 0.04 <0.001
Number of Part B MRCS attempts Reference - One attempt Two attempts -2.93 0.79 <0.001
≥3 attempts -8.11 1.31 <0.001
Ethnicity Reference - White British
Asian -1.77 0.75 0.019
Other -1.30 1.02 0.201
Black -1.08 1.74 0.536
Multiple linear regression analysis – ST3 score
![Page 33: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/33.jpg)
58
60
62
64
66
68
70
72
Two One ≥Three
Firs
t at
tem
pt
at n
atio
nal
se
lect
ion
(
%)
Number of attempts at Part B MRCS
![Page 34: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/34.jpg)
Previous minimal appointable scores
58
60
62
64
66
68
70
72
Two One ≥Three
Firs
t at
tem
pt
at n
atio
nal
se
lect
ion
(
%)
Number of attempts at Part B MRCS
![Page 35: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/35.jpg)
Conclusions
Supports validity and predictive validity of MRCS
17% of variance in national selection score is explained by Part B MRCS
Should Part B MRCS be used as one of the potential selection criteria for entering higher surgical training?
![Page 36: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/36.jpg)
Conclusions
Supports validity and predictive validity of MRCS
17% of variance in national selection score is explained by Part B MRCS
Should Part B MRCS be used as one of the potential selection criteria for entering higher surgical training?
![Page 37: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/37.jpg)
Future work
The relationship between MRCS and…
Performance in clinical practice e.g. ARCP outcomes
High-stakes exit exam, FRCS
![Page 38: Does the mandatory postgraduate UK surgical exam predict ... · Basic Surgical Examinations (ICBSE) Responsible for continued development, ... Yet, unlike other postgraduate UK exams](https://reader034.vdocuments.net/reader034/viewer/2022051802/5aee9aac7f8b9a572b8cf3d1/html5/thumbnails/38.jpg)
Acknowledgments
The 4 Surgical Colleges
ICBSE
Iain Targett - lead database administrator, Royal College of Surgeons