moving towards programmatic assessment
DESCRIPTION
Presented by Dame Lesley Southgate at the 2014 Annual MeetingTRANSCRIPT
Moving towards programmatic assessment
challenges and opportunitiesLesley Southgate
St Georges Medical School University of London
Background
• Over the past 30 years there has been significant change in approaches to assessment for the UK medical profession and wider.
• From entry to medical school through postgraduate training and on to established practice, regular assessment has become a fact of life.
09/04/2023 [email protected]
Background
• The days are gone when progress to the next stage was determined largely by exit examinations comprising elements that were sometimes poorly constructed, unrelated to the taught curriculum and which could not be challenged.
09/04/2023 [email protected]
Background
• During the period of change the focus has principally been on improving the quality of methods of assessment and developing approaches to assessing performance in practice
09/04/2023 [email protected]
Background
• And more recently, it has become usual to combine various assessment instruments into a programme of assessment which may support a period of education or training over several years
09/04/2023 [email protected]
Principles for good assessment design
• Modern assessment programmes typically contain a range of assessment methods combined in a programme of assessment developed as part of a curriculum. They are selected in the light of the purpose and content of the assessment
• Van der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Med Edu 2005 Mar;39(3):309-17.. [email protected]
09/04/2023 [email protected]
Changes have come…..
One of the cornerstones of the reform of assessment for UK undergraduate and postgraduate training, is the acceptance that a student/trainee will be assessed both by examinations, and in the workplace, within a programme of assessment methods which, taken together, ensures assessment of each domain of Good Medical Practice. (UK regulator guidance)
Changes have come
This enables a rich picture of the performance of the student/trainee to be assembled and considered in the light of the type of decision that is to be made about career progression, learning needs, and professional development.
• http://www.gmc-uk.org/guidance/good_medical_practice/contents.asp
09/04/2023 [email protected]
The purpose of assessing in the workplace
In educational settings assessment for learning should take priority over assessment of learning.
A programme of assessment should aim at building n:n relationships: each competency domain should be informed from various assessment sources and each assessment source should be used to inform about several competency domains.’
The GMC and other national bodies are currently grappling with this issue
Programmatic assessment: From assessment of learning to assessment for learning. Schuwirth & Van der Vleuten Medical Teacher 2011:33:478-485
Introduction and context
• From the beginning briefly– The UK scene from 1995………
• GMC Good Medical Practice effective from 1995• Latest edition 2013
– The GMC performance procedures• A form of programmatic assessment introduced 1997
– PMETB 2005-2010• Principles for assessment
• The UK foundation programme
UK GMC standards: Good Medical Practice
• Good clinical care• Maintaining good medical practice• Teaching and training, appraising and
assessing• Relationships with patients• Working with colleagues• Probity• http://www.gmc-uk.org/guidance/good_medical_practice/duties_of_a_doctor.asp
(for example) Good medical practice
• Good clinical care must include:– adequately assessing the patient's conditions,
taking account of the history (including the symptoms, and psychological and social factors), the patient's views, and where necessary examining the patient
– providing or arranging advice, investigations or treatment where necessary
– referring a patient to another practitioner, when this is in the patient's best interests
UK GMC performance procedures
Late 1997/2004• Assessment of poorly performing doctors
– At risk of losing licence to practice.• Questions we asked
– What are the standards all practising physicians must reach whether they are in training or fully registered
– What evidence must we collect to demonstrate that the standards have been reached
– And where could we collect the evidence about performance
Which standards, what evidence, where…….
• The conclusions we reached, and the design for the performance procedures, now well established in law, informed thinking in the establishment of the workplace assessments now in use in UK postgraduate training.
Southgate, L., Cox, J., McAvoy, P., McCrorie, P.,et al. (2001) The General Medical Council’s Performance Procedures: peer review of
performance in the workplace. Medical Education, vol.35 , Issue Supplement s1 Pages 1–78
When enough is enough: a conceptual basis for fair and defensible practice performance assessment. Schuwirth LW1, Southgate L, et al. Med Educ. 2002 Oct;36(10):925-30.
PMETB principles for assessmentA working paper from the Postgraduate Medical Educationand Training Board 14 September 2004Lesley Southgate and Janet Grant
In this paper, an assessment system refers to an integrated set of assessments which is in place for the entire postgraduate training programme and which supports the curriculum. It may comprise different methods, and be implemented either as national examinations, or as assessments in the workplace. The balance between these two approaches principally relates to the relationship between competence and performance. Competence (can do) is necessary but not sufficient for performance (does do), and as experience increases so performance based assessment in the workplace becomes more important
http://evavalpa.org/modulos/modulo_04/principles__assessment.pdf
PMETB principles for assessment• Principle 1 The assessment system must be fit for a
range of purposes• Principle 2 The content of the assessment will be
based on curricula for postgraduate training which themselves are referenced to all of the areas of Good Medical Practice The blueprint from which assessments in the workplace or national examinations are drawn will be available to trainees and educators in addition to assessors/examiners
• Principle 3 The methods used within the programme will be selected in the light of the purpose and content of that component of the assessment framework.
PMETB principles for assessment• Principle 4 The methods used to set standards for
classification of trainee’s performance/competence must be transparent and in the public domain
• Principle 5 Assessments must provide relevant feedback
• Principle 6 Assessors/examiners will be recruited against criteria for performing the tasks they will undertake
PMETB principles for assessment
• Principle 7• There will be Lay input in the development of
assessment• Lay opinion will be sought in relation to appropriate
aspects of the development, implementation and use of assessments for classification of candidates.
• Lay people may act as assessors/examiners for areas of competence they are capable of assessing.
• Principle 8• Documentation will be standardised and accessible
nationally
PMETB principles for assessment
• Principle 9• There will be resources sufficient to support
assessment• Resources will be made available for the
proper training of assessors
Principles for good assessment design
• Modern assessment programmes typically contain a range of assessment methods combined in a programme of assessment developed as part of a curriculum. They are selected in the light of the purpose and content of the assessment
• Van der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Med Edu 2005 Mar;39(3):309-17.. [email protected]
09/04/2023 [email protected]
And: the workplace
• The workplace enables the trainee to encounter and resolve common and important clinical problems in real time.
• The level of performance expected will depend on stage of training and feedback about progress
• It gives an opportunity to observe a trainee demonstrate understanding of what it means to adopt Good Medical Practice as the basis for all aspects of professional life.
From the GMC…to summarise……
One of the cornerstones of the reform of assessment for UK postgraduate training, is the acceptance that a trainee will be assessed both by examinations, and in the workplace, within a programme of assessment methods which, taken together, ensures assessment of each domain of Good Medical Practice. This enables a rich picture of the performance of the trainee to be assembled and considered in the light of the type of decision that is to be made about career progression, learning needs, and professional development.
• http://www.gmc-uk.org/guidance/good_medical_practice/contents.asp
The purpose of assessing in the workplace
In educational settings assessment for learning should take priority over assessment of learning.
A programme of assessment should aim at building n:n relationships: each competency domain should be informed from various assessment sources and each assessment source should be used to inform about several competency domains.’
The GMC are currently grappling with this issue
Programmatic assessment: From assessment of learning to assessment for learning. Schuwirth & Van der Vleuten Medical Teacher 2011:33:478-485
UK FOUNDATION PROGRAMMEEarly days
The UK Foundation programme from 2007 onwards
• The Foundation Programme is a two-year generic training programme which forms the bridge between medical school and specialist/general practice training.
Early days: Assessment in the Programme
• Purpose– Determine fitness to progress to next stage of training– Identify trainees in difficulty– Provide focused feedback consistent with CQI– Meet needs for accountability
• Four Methods – mini-Clinical Evaluation Exercise (mCEX) – Directly Observed Procedures (DOP)– Case-Based Discussion (CbD)– Peer Assessment (mini-PAT)
• Refined versions of traditional measures
•
04/09/2023 Prof Dame Lesley Southgate [email protected]
The toolkit for workplace assessmentDOPS MiniCex CbD MSF
Purpose ObservationObserve and assess the conduct of a practical procedure
Observation Observe and assess a clinical encounter
Conversation/discussionDiscuss an outcome/ output from workplace activity using a record the trainees has made a contribution to
Review by others/colleaguesProfessionalismInterpersonal skills/Team workingCommunication
Takes place
Process Reviewed and documented with feedback in the moment/ as it is happening
ProcessReviewed and documented with feedback in the moment/ as it is happening
Outcome/outputDiscussing, explaining, justifying aspects of the report/record/result. Including aspects of professionalism
Reflecting on comments of others within the framework of constructive feedback
Assessment: early days
• Foundation Programme Year 1 – 3640 trainees had at least one of instruments
completed– 2929 submitted at least one encounter for all four
tools– There were 7 to 11 questions per instrument
• All used a 6-point scale where 4 is “meets expectations”
CbD: Basic data
CbD:
CbD
Encounters: Complexity
0
10
20
30
40
50
60
70
mCEX CbD DOPs
LowAverageHigh
Early days: References• Med Educ. 2009 Jan;43(1):74-81. doi: 10.1111/j.1365-2923.2008.03249.x.• Initial evaluation of the first year of the Foundation Assessment Programme.• Davies H1, Archer J, Southgate L, Norcini J.
• Adv Health Sci Educ Theory Pract. 2008 May;13(2):181-92. Epub 2006 Oct 12.• mini-PAT (Peer Assessment Tool): a valid component of a national assessment
programme in the UK?• Archer J1, Norcini J, Southgate L, Heard S, Davies H.
• Med Educ. 2008 Oct;42(10):1014-20. doi: 10.1111/j.1365-2923.2008.03162.x.• Specialty-specific multi-source feedback: assuring validity, informing training.• Davies H1, Archer J, Bateman A, Dewar S, Crossley J, Grant J, Southgate L.
Early warnings 1
• The FAP was implemented in a very short time-frame in response to a central mandate and there was understandably significant concern about feasibility and the time it would require. Despite this, a mean of 16.6 case-focused assessments were submitted by each F1 trainee, although 40% of these were submitted in the last 6 weeks. It is likely that this reflects anxiety about achieving low scores early on in the year. Although the programme explicitly states that some scores of < 4 would be expected early in the year, this represents a major cultural shift in assessment.
• Med Educ. 2009 Jan;43(1):74-81. doi: 10.1111/j.1365-2923.2008.03249.x.• Initial evaluation of the first year of the Foundation Assessment Programme.• Davies H1, Archer J, Southgate L, Norcini J.
Early warnings 2
• It is also important that training is directed at all the health professionals involved in assessments and that it includes senior trainees and nurse specialists. In order to fully meet the PMETB principles, not only will assessors need to be trained, but there will need to be systematic processes in place to provide them with feedback on their performance
Feedback, judgement and training assessors
• ‘What was striking during these discussions was the expectation among trainees that WPBA should be about helping them to become better doctors and their corresponding openness to feedback. However, expectations are dashed by a system that is seen to be open to bias and corruption, with assessors who are untrained and too busy, and which is thus failing to deliver high quality, honest feedback. In turn, the enormous potential benefit of helping trainee doctors learn from their performance is being lost’.
• Abigail Sabey, Centre for Learning and Workforce Research University of the West of England, Bristol
The UK Foundation programme fast forward to 2013
• Foundation year 1 (F1) enables medical graduates to begin to take supervised responsibility for patient care and consolidate the skills that they have learned at medical school. Satisfactory completion of F1 allows the relevant university (or their designated representative in a postgraduate deanery or foundation school) to recommend to the GMC that the foundation doctor can be granted full registration.
• http://www.foundationprogramme.nhs.uk/pages/home/curriculum-and-assessment/curriculum2012
The UK Foundation programme 2013
• Foundation year 2 (F2) doctors remain under clinical supervision (as do all doctors in training) but take on increasing responsibility for patient care. In particular they begin to make management decisions as part of their progress towards independent practice. F2 doctors further develop their core generic skills and contribute more to the education and training of the wider healthcare workforce e.g. nurses, medical students and less experienced doctors.
• http://www.foundationprogramme.nhs.uk/pages/home/curriculum-and-assessment/curriculum2012
The UK Foundation programme 2013
• At the end of F2 they will have begun to demonstrate clinical effectiveness, leadership and the decision-making responsibilities that are essential for hospital and general practice specialty training. Satisfactory completion of F2 will lead to the award of a foundation achievement of competence document (FACD) which indicates that the foundation doctor is ready to enter a core, specialty or general practice training programme.
• http://www.foundationprogramme.nhs.uk/pages/home/curriculum-and-assessment/curriculum2012
And what happened to the workplace assessments?
• SLEs (structured learning events) will continue to use the established set of tools of mini-clinical evaluation exercise (mini-CEX), acute care assessment tool (ACAT) and case-based discussion (CbD) and the forms will focus on constructive feedback and action plans.
• Trainees may link SLEs and other evidence to curriculum competencies in order to demonstrate engagement with and exploration of the curriculum. The trainee has to make a judgement as to the evidence needed
• Supervisors should sample the evidence linked to competencies in the ePortfolio. It is not necessary to examine all the competencies to determine a trainee’s engagement with the curriculum and to make a judgement on the trainee’s progress
• http://www.jrcptb.org.uk/assessment/Documents/STAR%20report%20final%2029%20April%202014.pdf
A happy ending? 2014 • Supervised learning event Recommended minimum number* Direct
observation of doctor/patient interaction:• Mini-CEX• DOPS• 3 or more per placement* • (minimum of nine observations; • at least six must be mini-CEX)• Case-based discussion (CBD)• 2 or more per placement*• Developing the clinical teacher• 1or more per year