ABMS Board Certification: What Will Career-long Assessment Look Like?

Considering the High Stakes Exam

Consortium of the American College of Surgeons Accredited Education Institutes

13 March 2015

Lois Margaret Nora, MD, JD, MBA

Disclosures

» Salaried employee, American Board of Medical Specialties • ABMS Board Certification and Maintenance of Certification are

programs of the ABMS and its 24 Member Boards.

» Member, Council on Graduate Medial Education (COGME) » Advisory Board - Sam’s Club Healthy Living Made Simple

Magazine » Advisory Board, National Center for Interprofessional

Practices and Education

2

Outline

» Professional Self-regulation » ABMS Board Certification » Program for Maintenance of Certification - 2015 » The High Stakes Exam » Innovations: Pilot, In Development, In Consideration

3

Characteristics of a Profession » Special knowledge and skills acquired by certain members of

society and not others; often with the substantial investment of society in the educational process

» The commitment of the profession’s members to, and the trust by the other members of society that, the special knowledge and skills will be used – not in the professional’s self-interest – for the good of society and its members

» Trusting in that commitment, Society grants the profession substantial autonomy to determine educational standards, self-assess, and to self-regulate

4

Professional Self-regulation

» The community of physicians (profession) regulates the profession • Educational, Behavioral, Assessment standards • Directed toward program: LCME, ACGME, ACCME • Directed toward individual: State Medical Boards, ABMS

Member Boards • Other engaged parties: Medical Schools, Specialty

organizations, AMA, AAMC, CMSS, others

» This responsibility is related to, and distinct from, the responsibility of an individual physician to behave professionally 5

American Board of Medical Specialties

» ABMS Board Certification originated early 1900s » Evolved from concerns about quality of patient care in an

environment of concern about medicine/medical education » ABMS Founding Boards: Ophthalmology, Otolaryngology,

Obstetrics-Gynecology, Dermatology and Syphilology » ABMS is a federated organization of 24 Member Boards » The mission of the American Board of Medical Specialties

is to serve the public and the medical profession by improving the quality of health care through setting professional standards for lifelong certification in partnership with Member Boards.

6

ABMS Member Boards

• Allergy and Immunology • Anesthesiology • Colon and Rectal Surgery • Dermatology • Emergency Medicine • Family Medicine • Internal Medicine • Medical Genetics and Genomics • Neurological Surgery • Nuclear Medicine • Obstetrics and Gynecology • Ophthalmology

• Orthopaedic Surgery • Otolaryngology • Pathology • Pediatrics • Physical Medicine and

Rehabilitation • Plastic Surgery • Preventive Medicine • Psychiatry and Neurology • Radiology • Surgery • Thoracic Surgery • Urology

24 ABMS Member Boards 37 specialties and 123 Sub-specialties

7

ABMS Member Board Executives - Surgical Section

Daniel J. Cole, MD Co-Executive Director

Mary E. Post MBA, CAE Co-Executive Director

Larry C. Gilstrap III, MD American Board of

Obstetrics and Gynecology

John G. Clarkson, MD American Board of

Ophthalmology

Shepard R. Hurwitz, MD American Board of Orthopaedic Surgery

Robert H. Miller, MD American Board of

Otolaryngology

R. Barrett Noone, MD American Board of

Plastic Surgery

Frank R. Lewis Jr., MD American Board of

Surgery

William A. Baumgartner, MD American Board of Thoracic Surgery

Gerald H. Jordan, MD American Board of

Urology

David J. Schoetz, Jr., MD American Board of

Colon and Rectal Surgery

Fredric B. Meyer, MD American Board of

Neurological Surgery (June 2015)

American Board of Anesthesiology

8

ABMS Member Board Certification

» Two components: initial and continuing (MOC) » Initial Board Certification

• Meet all requirements for licensure • Requires completion of an extended high-quality period

of training and assessment in knowledge, skills, and professionalism (usually ACGME-approved residency/fellowship)

• Complete additional assessments of knowledge, clinical skills, and professionalism specific to the discipline

9

Why Transition from “Diploma-style” Board Certification to ABMS Continuing Certification? » Public expectations of “board certification”; fulfilling the public

trust » Profound change in knowledge and practice and social

construct over time » Skills can diminish over time » People don’t self-assess particularly well » Safety concerns in healthcare settings » Programs of on-going assessment and learning can improve

knowledge, skills, judgment, and practice » Practice in other high-consequence industries

10

Changes Over Time: • Science (HIV, Genomics); • Practice (Peptic Ulcer, Acute MI, Laparoscopic surgery); • Setting (In-patient to Ambulatory); • Dangers (From limited efficacy & safe to substantial efficacy & dangerous); • Construct (From Captain of the ship to Team member/sometimes leader); • Social culture (Beneficent paternalism to Patient autonomy); • Pace of change (Rapid to super-rapid); • Physician (Roles, Wisdom, Age, Abilities, Life stressors)

Why Transition from “Diploma-style” Board Certification to ABMS Continuing Certification?

Medical School

Residency & Fellowship Career-long Medical Practice

15 10 20 25 30 35 40 45 Years

11

Movement Toward Continuous Professional Self-regulation is Widespread » Renewal of licensure

• FSMB and State medical boards considering expanded expectations for maintenance of licensure

» Continuing Certification • ABMS Programs for MOC • AOA Programs for OCC

» International • UK – Revalidation Program • Canada – RCPSC MOC Program • Australia – Revalidation

12

ABMS - Movement to Continuing Certification

» 1970-1990: Periodic recertification via high stakes examination

» 2006: Entire ABMS Boards Community commit to program for continuing certification (Maintenance of Certification; MOC); phased implementation

» 2012-13: In-depth review of MOC; development of new program standards

» 2015: New Program Standards in place, January 1

13

Goals of Continuing Certification - 2015

» Improve the quality of care provided to patients and communities; Improve health outcomes

» Meaningful, rigorous professional self-regulation that merits and reinforces public confidence in the profession

» Respond to valid concerns about MOC » Provide a systematic, rigorous, relevant method for on-going

professional learning and assessment in knowledge, skills, judgment, and professionalism of the profession and the specialty that has meaning and value for diplomates

» Assessment for learning as well as assessment of learning

14

ABMS Maintenance of Certification - 2015

» Must have achieved Initial ABMS Board Certification » Career-long process » Based in the six ABMS/ACGME Competencies

• Practice-based Learning and Improvement; Patient Care and Procedural Skills; Systems-based Practice; Medical Knowledge; Interpersonal and Communication Skills; and Professionalism

» MOC Program is grounded in Assessment, Educational, and Quality Science research

15

ABMS Program for MOC - 2015 Four Integrated Elements » Professionalism and Professional Standing » Lifelong Learning and Self Assessment

• Relevant to specialty and area of practice • Emphasis on Patient Safety • Emphasis on Accredited CME

» Assessment of Knowledge, Judgment and Skills » Improvement in Medical Practice

• At individual and/or system levels

» Standards for each element at abms.org

16

ABMS Program for MOC - 2015

» General Program Standards include: • ABMS Member Boards will work to increase the

Program’s value, relevance, and meaning; and be sensitive to the time, administrative burden, and cost associated with participation.

• ABMS Member Boards will engage in CQI of their Program for MOC and engage in ABMS-wide review process. − Diplomate and Public engagement in CQI and review processes

17

MOC - Multiple Potential Roles for ACS-Accredited Education Institutes » Training and assessment when the Expert is/becomes a

Novice • Re-entry process for physicians • Identification of new/newly articulated competencies

» Training and assessment activities related to specialty-specific, cross-specialty, interprofessional competencies

» Linking Educational Institute activities to information about practice gaps noted via registries and other sources

» Linking local health system needs to MOC » Research » Expertise

18

Considering the High-Stakes Examination

Assessments Serve Multiple Purposes

» Providing a reliable, valid and fair basis for grading in individual educational experiences

» Making promotion and other “status” decisions

» Promoting safe and effective patient care; protecting patient safety

» Communicating to those tested what material is important and motivating them to learn it

» Encouraging integration of information across instructional experiences and application of knowledge in patient care

» Monitoring and encouraging learning and retention

» Providing information to evaluate and improve the educational program

» Providing information to aid in selection

Assessment for learning, not just assessment of learning 20

The High Stakes Exam

» Used in initial and maintenance of certification; assurance of the public

» Evaluate the knowledge base, diagnostic reasoning, and clinical judgment in the domain of the discipline

» Foster learning through preparation for, and feedback from, the examination

» Provide a psychometrically valid and reliable mechanism to contribute to a summative decision

» Usually requires recognition of the common and the uncommon

» Heavy use of clinical vignettes and management

21

High-Stakes Examinations - Similarities and Differences Across the ABMS Member Boards » Computer-based (written) examination » Oral exams and other assessments » Similarity between initial certification and MOC

exams » Process for preparation » Use of modular (focused) exams » Periodicity of the exam » Exam as the summative element vs. exam as

contributing to a summative decision

22

The High Stakes Exam - Issues Identified in 2012-13 Review of MOC » Required/Not required of all » Costs and other burdens of preparation/taking exam » Assistance in preparing » How focused, how general? Relevance of questions to the

diplomate’s specific practice areas » How often? » The exam center experience » Access to materials; examine similar to practice » Receipt of feedback » Differences among the Member Boards

23

MOC Standards for Part III: Assessment of Knowledge, Judgment, and Skills

» Examination of diplomates’ knowledge of core content, judgment, and skills no less often than every ten years • Assessments meeting standards for test design/development,

administration, reliability (particularly reproducibility of pass/fail results), standard setting, scoring and score reporting

• Secure defined as identified person; no materials other than what is allowed by Board; no transmission of information by anyone

• Transparency about the above with diplomates » Provision of meaningful performance feedback aiding diplomates

in identifying strengths and weaknesses » Encourage innovation in methods

24

Enhancements to Part III Assessments (Already in Place or Being Piloted by Some Boards)

» Modular examination design » Use of test material describing patient care situations and

requiring diplomates to indicate clinical decisions; use of imaging study results, patient videos, and other multimedia to present patient findings more realistically

» Access to reference material during test » Improved linkage of updates/practice guidelines/other materials

available during lifelong-learning and self-assessment activities to the MOC exam

» Expanded performance reports » Remote proctoring

25

Two Early Projects

Project I: The MOCA Redesign Project American Board of Anesthesiology:

Web-based learning platform that supports personalized knowledge acquisition, assessment, and demonstration of proficiencies.

» Identifies knowledge strengths and gaps among diplomates as a group and individually using a longitudinal assessment approach.

» Optimizes diplomate learning experiences by providing customized feedback and learning recommendations.

» Connects diplomate to a universe of relevant continuing education content and learning opportunities.

» The “Big Data” created better informs CME providers about diplomate needs and future product development.

27

ABA MOCA 2.0

28

29

ABA MOCA 2.0

30

ABA MOCA 2.0: Personalized Learning

31

ABA MOCA 2.0: Search for Content

32

ABA MOCA 2.0: Online Simulation

33

ABA MOCA 2.0: Online Simulation

34

ABA MOCA 2.0: MOCA Minute Application

35

ABA MOCA 2.0: MOCA Minute Application

36

My MOCA: MOCA Minute Application

To Date and Going Forward

» Vision for Testing: Anesthesia will employ an intensive longitudinal assessment model that will foster on-going professional learning and allow a summative decision across time.

37

Project II: A Question Being Considered by Several Boards » Can demonstrated excellence in patient outcomes

satisfactorily substitute for a high stakes exam?

38

What Will Professional Self-regulation, and assessment of knowledge, skills, judgment as

part of that self-regulation, look like?

39