gordon page.pdf
TRANSCRIPT
Gordon PageEmeritus Professor, Faculty of Medicine
University of British ColumbiaVancouver Canada
Outline
The Rationale – Why developnational examinations for thehealth professions?
How is ‘competence’ defined?
What are the desired features ofnational examinations for healthprofessionals
Outline
The Rationale – Why developnational examinations for thehealth professions?
How is ‘competence’ defined?
What are the desired features ofnational examinations for healthprofessionals
Why develop nationalexaminations in the health
professions?
Why develop nationalexaminations in the health
professions?
A Quality AssuranceStrategy
‘Assurance’ of theQuality of What?
1.The competence of the graduates ofthe Health Professions SchoolsDo the graduates possess at least the minimallevels of competence (a) to fulfill the roles thatsociety has assigned to their professions, or (b) toadvance to the next level of training in theirprofession?
Summative Formative
‘Assurance’ of theQuality of What?
1.The competence of the graduates ofthe Health Professions Schools
2. The quality of the training programsthat prepare students for these roles
Summative Formative
Why develop nationalexaminations in the healthprofessions?
The educational rationale
To convey to students what is important to learn,and to motivate them to learn it (‘Assessmentdrives learning!’)
To influence the curricula of health professionsschools by providing an operational definition ofthe competencies expected of graduatingstudents.
To provide a mechanism for evaluating the qualityof and inequities between the schools in eachhealth profession, identifying S&W in each
Why develop nationalexaminations in the healthprofessions?
The health care rationale
To ensure that the graduates of healthprofessional programs are competent – toprovide a basis for decisions on theirlicensure/registration
To define a national standard – thoseabilities that together define thecompetencies and levels of performanceexpected of all graduates
Why develop national healthprofessions examinations?
In summary …
To provide guidance and motivation tostudents and schools on what isimportant to learn
To pass judgement the competence ofgraduating students, and the educationaleffectiveness of schools
To define a national standard – thoseabilities that define the competencies andlevels of performance expected of allgraduates
Outline
The Rationale – Why developnational examinations for thehealth professions?
How is ‘competence’ defined?
What are the desired features ofnational examinations for healthprofessionals
What is ‘competence’?
Competence is the “ability to do something”
In the context of health professionseducation and practice, Epstein (2002)defined competence as, “the habitual andjudicious use of communication, knowledge,technical skills, clinical reasoning, emotions,values, and reflections in daily practice toimprove the health of the individual patientand community”
A national examination of‘competence’ must therefore testthe ability to use/apply knowledgeand skills in the context of caringfor patients.
A test of the recall of knowledge is not a testof ‘competence’. While possessing and beingable to recall knowledge is ‘the cornerstone’ ofbecoming competent, it is not sufficient.Students who memorize and can recallknowledge are often not able to effectivelyapply or use that knowledge.
Possessing Knowledgeversus
Possessing Competence
There is a bigdifference!
Non VignetteWhat is the most likely renalabnormality in children with nephroticsyndrome and normal renal function?
acute poststreptococcal glomerulonephritis
hemolytic-uremic syndrome
minimal change nephrotic syndrome
nephrotic syndrome due to focal andsegmental glomerulosclerosis
Schönlein-Henoch purpura with nephritis
(A)
(B)
*(C)
(D)
(E)
Case SM, Swanson DB. Constructing Written Test Questions for the Basicand Clinical Sciences. Page 58-9. www.nbme.com
Short Vignette
A 2-year-old boy has a 1-weekhistory of edema. Blood pressureis 100/60 mm Hg, and there isgeneralized edema and ascites.Serum concentrations are:creatinine 0.4 mg/dL, albumin 1.4g/dL, and cholesterol 569 mg/dL.Urinalysis shows 4+ protein andno blood. What is the most likelydiagnosis?
Long Vignette
A 2-year-old black child developedswelling of his eyes and ankles overthe past week. Blood pressure is100/60 mm Hg, pulse 110/min, andrespirations 28/min. In addition toswelling of his eyes and 2+ pittingedema of his ankles, he has abdominaldistension with a positive fluid wave.Serum concentrations are: creatinine0.4 mg/dL, albumin 1.4 g/dL, andcholesterol 569 mg/dL. Urinalysisshows 4+ protein and no blood.
What is the most likely renal abnormality in children withnephrotic syndrome and normal renal function?
Non Vignette
acute poststreptococcal glomerulonephritishemolytic-uremic syndrome
minimal change nephrotic syndrome
nephrotic syndrome due to focal and segmentalglomerulosclerosisSchönlein-Henoch purpura with nephritis
(A)(B)
*(C)
(D)
(E)
Short VignetteA 2-year-old boy has a 1-week history of edema. Blood pressure is
100/60 mm Hg, and there is generalized edema andascites. Serum concentrations are: creatinine 0.4 mg/dL,
albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows4+ protein and no blood. What is the most likely diagnosis?
Long VignetteA 2-year-old black child developed swelling of his eyes andankles over the past week. Blood pressure is 100/60 mm
Hg, pulse 110/min, and respirations 28/min. In addition toswelling of his eyes and 2+ pitting edema of hisankles, he has abdominal distension with a positivefluid wave. Serum concentrations are: creatinine 0.4 mg/dL,albumin 1.4 g/dL, and cholesterol 569 mg/dL. Urinalysis shows4+ protein and no blood.
What is the most likely renal abnormality in children withnephrotic syndrome and normal renal function?
A 2-year-old boy has a 1-week history of edema. Blood pressureis 100/60 mm Hg, and there is generalized edema and ascites.Serum concentrations are: creatinine 0.4 mg/dL, albumin 1.4g/dL, and cholesterol 569 mg/dL. Urinalysis shows 4+ proteinand no blood. What is the most likely diagnosis?
A 2-year-old black child developed swelling of his eyes andankles over the past week. Blood pressure is 100/60 mm Hg,pulse 110/min, and respirations 28/min. In addition to swellingof his eyes and 2+ pitting edema of his ankles, he has abdominaldistension with a positive fluid wave. Serum concentrations are:creatinine 0.4 mg/dL, albumin 1.4 g/dL, and cholesterol 569mg/dL. Urinalysis shows 4+ protein and no blood.
A B *C* D E1 0 99 0 0
8 1 90 1 0
0 0 98 2 0
5 2 82 8 1
0 1 98 1 0
10 9 66 10 5
Overall P-Value
94
88
84
Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences, 1996. Page 58-9.
Does testing knowledge recall vs applicationof knowledge affect test scores?
Possessing Knowledgeversus
Possessing Competence
There is a bigdifference!
National examinations must testcompetence, not knowledge recall!
But whatcompetencies should
be tested?
Standar Kompetensi DokterKonsil Kedokteran Indonesia
(Standards of Medical Competencies)(Indonesian Medical Council)
2006
Describes the competencies expected ofstudent doctor graduating from an
Indonesian medical schools.
Areas of Competency(Standar Kompetensi Dokter 2006)
1. Effective communication
2. Clinical skills
3. Scientific basis of medical knowledge
4. Management of health problems
5. Management of information
6. Self awareness and self development
7. Ethics, morals, medico-legal aspectsand professionalism, and patient safety
Medical Council of Canada
Objectives for the QualifyingExamination
The ‘Objectives’ define thecompetencies expected of medicalgraduates entering supervised andindependent practice
http://www.mcc.ca/Objectives_Online/
Competency Frameworkfor Doctors in Canada
Summary …‘How is competence defined?’
The ability to use/applyknowledge and skills in thecontext of caring for patients.
The ‘competencies’ comprising‘competence’ should be definedby each health profession
Outline
The Rationale – Why developnational examinations for thehealth professions?
How is ‘competence’ defined?
What are the desired features ofnational examinations for healthprofessionals
Utility (U) of anAssessment Strategy
V = Validity
R = Reliability
E = Educational impact
A = Acceptability
C = Cost
U = f(V, R, E, A, C)
Cees van der Vleuten, 1990s
What are the desired featuresof national examinations forhealth professionals?
V = Validity
R = Reliability
E = Educational impact
A = Acceptability
C = Cost
What are the desired featuresof national examinations forhealth professionals?
V = ValidityR = Reliability
E = Educational impact
A = Acceptability
C = Cost
Validity – An Essential Featureof National Examinations
The question posed by validity is,“Does this examination permit usto make correct inferencesabout the competence ofexaminees?”
Validity – An Essential Featureof National Examinations
The two main factors influencingvalidity are:
Examination Content – what istested?
Examination Methods – how is thecontent tested?
Validity – An Essential Featureof National Examinations
The two factors influencingvalidity are:
Examination Content – what istested?
Examination Methods – how is thecontent tested?
International Trends in Medicine –Testing Methods used in National
Examinations in Indonesia, Canada, theUSA and Australia
An MCQ ‘written’ test Questions are ‘case-based’ Tests are computer-based 150 or more questions Use A-type MCQ questions Allow 3 or more hours of testing time
An ‘Objective Structured Clinical Examination’(OSCE) Use standardized patients (SPs) Use 3 or more hours of testing time Use 10 to 20 ‘stations’, each from 8 to 25 minutes in
length
Is there evidence supportingthe ‘validity’ of theseexamination methods?
Norcini (ASME, 2003) – Of all assessmenttechniques, written test (MCQs) scores haveshown to be the best predictors of futureclinical performance and of clinical outcomes.
Tamblyn et al (JAMA, 2002,1998) – Canadianlicensing examination (clinical decision-making) scores show sustained relationshipswith effectiveness indices of preventive care
and acute and chronic disease management
The Relationship of ExaminationScores to Concurrent ClinicalPerformance for Practice-readydoctors (Canadian study, 2007)
Correlations between:
Clinical Performance (Mini-CEX scores)
and MCQ scores .335
Clinical Performance (Mini-CEX scores)
and OSCE scores .372
Is there evidence supportingthe ‘validity’ of theseexamination methods?
Yes, it isreassuring but not
strong!
Validity – An Essential Featureof National Examinations
The two factors influencingvalidity are:
Examination Content – whatis tested?
Examination Methods – how is thecontent tested?
An Examination Blueprint
Sample Blueprint forIndonesia
Competency areas MCQ OSCE
Communication skills 10%
Clinical Skills 20% 70%
Scientific basis … 10%
Management of healthproblems
60%
Management ofinformation
10%
Self awareness &development
10%
Ethics, morals,professionalism
10%
Blueprint for MCQ Items Testing‘Management of Health Problems’
Clinical Tasks ChildHealth
Maternalhealth
Adulthealth
Mentalhealth
Populationhealth
Data gathering 5 3 13 5 4
Datainterpretationand synthesis
6 3 17 6 4
Drugmanagement
2 2 10 3 0
Non-drugmanagement
2 1 9 3 0
Georges Bordage
Clinical Decision-Making
Reduced (“empty mind”)
Compiled (“recall/recognition”)
Elaborated (“deductive thinker”)
Dispersed (“cluttered mind”)
(Bordage, G. Academic Medicine, 1994, 1999)
Clinical Decision-Making
Reduced (“empty mind”)
Compiled (“recall/recognition”)
Elaborated (“deductive thinker”)
Dispersed (“cluttered mind”)
(Bordage, G. Academic Medicine, 1994, 1999)
****************************************
Summary – ‘What to test?’
National examinations should:
Test content defined relative to anexamination blueprint
Test the application/use of knowledgeand skills (i.e., ‘competencies’)
XNot test recall of knowledge
XNot test thoroughness of datagathering
XTest clinical decision making, notclinical reasoning
What are the desired featuresof national examinations forhealth professionals?
V = Validity
R = Reliability
E = Educational impact
A = Acceptability
C = Cost
Reliability
Is a measure of the accuracy of testscores
For MCQ examinations, accuracy islargely a function of the number of itemson the test (i.e., sampling)
For OSCEs, accuracy is also a function ofthe consistency of examiners and SPs
Reliability is expressed as a reliabilitycoefficient ‘r’, where 0<r<1
‘Rule of thumb’: r > .80 for makingdecisions about individual examinees
Reliability as a function oftesting time
TestingTime inHours
1
2
4
8
MCQ
0.62
0.76
0.93
0.93
OSCE
0.43
0.60
0.76
0.86
What are the desired featuresof national examinations forhealth professionals?
V = Validity*
R = Reliability*
E = Educational Impact
A = Acceptability
C = Cost
Educational Impact
Students will learn, and schools willteach what is tested on nationalexaminations
With a view to ‘assessment drivinglearning’, is the national examination‘driving’ teaching and learning in theright direction? It will if it testscompetencies that are important tothe practice of the health profession inquestion.
Acceptability What strategies encourage acceptance of
national examinations by key stakeholders –i.e., the schools, the professional licensing andregistration bodies, the profession, and thegovernment?
The key strategy is INVOLVEMENT – indefining policies related to the role of theexamination in the profession, in defining anddeveloping examination content, in definingthe standards of acceptable performance(passing scores), in quality assurance, …
Cost
National examinations with acceptablemeasurement qualities are costly todevelop, administer, score and maintain
The major cost is peoples’ time
In the USA, Australia and Canada thecosts of the MCQ and OSCE nationalexaminations exceed $2000 per candidatefor each examination – a cost which issupported primarily by examinee fees
Cost Major cost items include:
developing test blueprints
Developing/peer review of examinationquestions/cases
developing supportive computer systems for testdevelopment, test administration, and test analysis
pilot testing and refining questions/cases
standard setting
recruiting/training/paying examiners
examination scoring
quality assurance reviews (using examinationstatistics – e.g., reliability of test scoresdiscrimination indices of test items)
maintaining test security
…
In summary …
The Rationale – Why developnational examinations for thehealth professions?
How is ‘competence’ defined?
What are the desired features ofnational examinations for healthprofessionals
Developing the Ideal Modelfor National HealthProfessional Examinations Is an MCQ examination and an OSCE an ideal model for a
national examination? My answer: No!
Why? Option lists in MCQs provide too much cuing to weaker
candidates, and
OSCEs lack reliability (accuracy) and hence validity.
What is the ideal model? Short answer questions in place of MCQs, and
Highly structured assessments of clinical skills in real clinicalsettings in place of OSCEs
Why not use the ideal model? Too expensive, and quality control is even more challenging!
Terima kasih