ExaminerNational Board of Medical Examiners3750 Market StreetPhiladelphia, PA 19104-3102

www.nbme.org

Fall/Winter 2010 Volume 57, Number 2

In late September 2010, the NBME,along with the EducationalCommission for Foreign MedicalGraduates (ECFMG®) and theFederation of State Medical Boards(FSMB), co-hosted the 9th biennialconference of the InternationalAssociation of Medical RegulatoryAuthorities (IAMRA). Held inPhiladelphia at the Society HillSheraton, this three-day meeting wasattended by 216 participants fromover 30 countries, including, amongothers, Albania, Australia, Egypt,India, Iraq, Nepal, the Netherlands,Pakistan, Singapore, South Korea, theUnited Kingdom, and Zimbabwe. The9th biennial conference is the first in aseries of planned IAMRA conferencesthat will bring world experts togetherin an effort to develop global best

In This Issue:

1 NBME Co-hosts 9th Biennial IAMRA Conference

1 New Item Formats UnderDevelopment: Interpreting Medical Literature

4 Sharing Isn’t Always Caring

5 Assessment of Communication Skills within Clinical Encounters

practices for use in licensing and regulating physicians.

In a welcoming message to atten-dees, the CEOs of the host organiza-tions noted, “We believe that in atime of increasing global interconnec-tion in healthcare, it is important forgeographically diverse organizationsto stay in contact and communicatewith one another. Our borders maybe distinct politically, but from aglobal health perspective—especiallyin terms of public health—ournations are interconnected andaligned. Every healthcare and

Following the Comprehensive Review of USMLE, the Committee to Evaluatethe USMLE Program (CEUP) recommended that the USMLE program intro-duce, as soon as possible, a testing format designed to assess an examinee’sability to recognize and define a clinical problem; to access appropriate refer-ence resources in order to find the scientific and clinical information neededto address the problem; and to interpret and apply that information in aneffective manner. As a result, the NBME was asked to develop assessmentsfocusing on a doctor’s ability to access relevant information, evaluate itsquality, and apply it to solving clinical problems. If successful, these new

(continued on page 7)

(continued on page 3)

© 2010 NBME

NBME Co-hosts 9th Biennial IAMRA Conference

New Item Formats Under Development:Interpreting Medical Literature

IAMRA members meet during the conference.

INSIDE THE NBME

1 Conference attendees learn about item formats at the NBME offices 2 Attendees participate in facilitated small group discussion3 Presentation by IAMRA Chair-Elect Fleur-Ange Lefebvre 4 Conference attendees enjoy the Mummers5 Small groups in session 6 Fife & drum corps welcomes visitors to the NBME7 The Mummers’ Strut!

Scenes from the 9th Biennial IAMRA Conference

1

2

3

4

5 6 7

2

medical regulatory organizationneeds to focus on its strengths andits role in protecting the public. Therole our organizations can andshould play is helping ensure thatpatients can trust the veracity andthe integrity of the professionals theyentrust their lives with—the world’sphysicians and health professionals.”

Staff members from each host organi-zation participated in three workinggroups: program planning (led byECFMG), logistics (led by NBME), andmarketing (led by FSMB). The programplanning committee adopted a uniqueformat, minimizing plenary presenta-tions and engaging the participants infocused, facilitated small group workintended to create draft “best prac-tice” principles in three relevantdomains: initial licensure/registration,complaints/discipline, and maintenance

of licensure. Three progressive smallgroup sessions gave all attendees anopportunity to share their own experi-ences, identify common themes, andexplore the real world challenges of

establishing best practices. Followingthe small group sessions, a closing ple-nary session included summaries of theresults of each content area and a dis-cussion of next steps.

The conference also included a numberof social events and entertainment. TheOpening Reception featured a stirringperformance by the world famousPhiladelphia Boys Choir & Chorale. Thefollowing evening, attendees wereinvited to the NBME headquarters forthe Hosts’ Reception and Exhibition.Conference attendees and their guestswere welcomed into the lobby of theNBME by a fife and drum corps andBen Franklin, and a buffet dinner fea-tured US regional fare. This event alsooffered participants the opportunity tovisit a number of exhibits by the hostorganizations and other associatedorganizations and to interact with

3

9

(IAMRA, continued from page 1)

(continued on page 4)

The Philadelphia Boys Choir & Chorale practice for their performance.

Plenary session during the conference.

4

FOR STUDENTS

When you take NBME subject exams,USMLE Steps and other assessments,you sign a statement agreeing to main-tain the confidentiality of the test con-tent. You agree not to memorize, copy,reconstruct, or disseminate what youwill see on your exam. Take a momentto think about why such confidentialityis important. While the term “copyrightinfringement” might first cross yourmind, we would like to remind you ofsome other reasons why maintainingexam confidentiality is important to you,to the public, and to the profession.

When you engage in activities involv-ing discussion of actual exam content,you put your test scores (and possiblyyour career) in jeopardy. Formal andinformal test preparation sources,Internet forums, social media and thelike may appear to be fruitful sourcesof information about examinations, butyou should be aware that engaging inthese activities is risky. Inadvertentaccess to exam materials prior to test-ing could result in the invalidation ofyour scores, and knowingly engaging inthis activity could result in a bar fromexams and possibly the end of yourmedical practice career. The Internet isreplete with prep courses and tutorspromising guaranteed passes and highscores. Some of these enterprises maybe legitimate and some may not. Weurge you to be careful and cautious ifyou chose to engage in these forms oftest preparation.

Sharing Isn’t Always Caring

While you may view your exams as hur-dles to overcome to achieve your careergoals, think a minute about the public,your future patients. Healthcare con-sumers rely on the knowledge, skills,and professionalism of the individualswho care for them. Think about yourfamily and your loved ones, all of whomare or will be healthcare consumers.They rely on the systems in place toensure their healthcare providers havethe requisite knowledge and skills toprovide safe, effective care. The exami-nations you take as a student or resi-dent play an important role in publicprotection, and they should be takenwith honesty and integrity.

Medicine is a profession that has theprivilege of self-regulation. The publicexpects that the profession, with itscommitment to helping and healing,can be trusted to establish its ownstandards in education, training,

licensure, and practice and thatphysicians themselves are best suitedto define and uphold these stan-dards. Thus, the profession and theindividuals who belong to it are obli-gated to ensure that practitioners(and future practitioners) meet mini-mum standards of knowledge andskills necessary for the safe andeffective practice of medicine. Thepublic must be assured that physi-cians value professionalism andintegrity. Your commitment to profes-sionalism includes maintaining theintegrity of the systems designed toprotect the public.

Maintaining the confidentiality of yourown examinations and reporting anyunauthorized sharing of examinationcontent or other unethical behaviorsaround professional assessment willbenefit you, your family, the public,and the profession.

staff. Tours of the Clinical Skills TestCenter and mock encounters withstandardized patients were alsooffered. Toward the end of theevening, attendees were treated tomore Philly flavor—a performance bythe Mummers. For their final evening,conference participants were invited tothe National Constitution Center, the

most interactive history museum in theUnited States. This evening featuredexhibits on important events in US history, a multimedia presentation featuring a live actor, and “A Taste ofPhiladelphia” buffet dinner.

When available, presentations and pro-ceedings from the conference will beposted at www.iamra.com.

(IAMRA, continued from page 3)

cient for modern clinical encounters.Finally, recent data demonstrate thatnew residents have significantresponsibility for encounters withpatients and their families and thatthey engage in much more independ-ent, sensitive, and complex communi-cations than has been previouslythought.3 Such data suggest a needfor assessment focused on the tasksperformed in supervised practice, ie,residency training.

A new team has been formed todevelop and test revised assessmentapproaches. The team represents acollaboration between NBME and theClinical Skills EvaluationCollaboration (CSEC) staff and twoexternal consultants. The team pullstogether expertise in communicationskills assessment, the development ofsimulated encounters utilizing stan-dardized patients (SPs), instrumentdesign, and research methodology.An extensive literature review provid-ed an evidence-based approach todeveloping an assessment construct.It also facilitated identification ofacademically based communicationskills experts who assisted the teamwith refinement of the construct,based on the “six function model” ofthe clinical encounter (Table 1).4

Table 1: Six Function Construct 1. Fostering the Relationship 2. Information Gathering 3. Information Provision 4. Making Decisions 5. Supporting Emotions 6. Enabling Patient Behaviors

The team has divided this model intobasic and advanced components. Theimmediate focus is to develop anenhanced assessment scale based onthe basic functions, along with modi-fying case development and SP train-ing processes to better embrace themodel. While there is similaritybetween these basic functions and theoriginal Step 2 CS construct, there areimportant differences (Table 2). Later,the team will turn to the task of incor-porating the more advanced functionsinto the assessment system.

The team has used a unique approachto move from construct developmentto instrument design. Multiple videorecordings of encounters have beenreviewed to identify specific behav-ioral expressions of the assessmentconstruct. These behavioral expres-sions are being transformed into thedesign of a new instrument that will

5

INSIDE THE NBME

Over two million simulated encountershave been administered since the roll-out of Step 2 CS In June 2004. Muchhas been learned from this experi-ence, generating a rich opportunity forenhancement of our methods forassessing communication skills.Below we describe work directed atsuch evolution, including adoption ofa literature-based model to guide casedevelopment and scoring; identifica-tion of behavioral descriptionsgrounded in empiric methods; andresultant changes anticipated for Step 2 CS.

Over the past 10 years, the evidencebase for the importance of communi-cation skills has grown muchstronger, and good communicationskills have been shown to producedesirable and efficient clinical out-comes, as well as satisfied patients.1

Such data tend to upgrade the impor-tance of communication skills in theassessment continuum. Further, con-tinued evolution of the doctor-patientrelationship is calling for morepatient-centered approaches, onesthat involve patients in decision mak-ing and support them in adoptingbehaviors conducive to better health.2

This means that highly interrogatoryand controlled encounters are insuffi-

Assessment of Communication Skills within Clinical Encounters

(continued on page 6)

CIS OriginalConstruct

Professional Manner and Rapport

Information Gathering

Information Sharing

Enhanced ConstructBasic

Fostering the RelationshipSupporting Emotions: Basic

Gathering Information

Providing InformationMaking Decisions: Basic

Enhanced ConstructAdvanced

Supporting Emotions:Advanced

Making Decisions: AdvancedEnabling Patient Behaviors

cover the range of functions from theconstruct and guide SPs in recordingtheir observations. Early pilot workwill allow for testing of revised train-ing methods and case developmentprocesses, as well as the new instru-ment. It is expected that the earlypilots will inform revisions of process-es as well as the instrument. Therewill be iterative research leading upto SPs fully adopting both portrayaland assessment tasks under condi-tions that simulate a high-stakesexamination.

By early 2012, it is expected that thiscomprehensive consideration of allsteps in the assessment system willproduce enhancement to our assess-ment of communication skills withinclinical encounters. These enhance-ments will include:

• clarification for the examinees oftheir role within the encounters,based on our improved under-standing of the tasks they performas residents;

• improved scenario-developmentprocesses that stimulate a broaderand deeper range of essentialcommunication skills;

• an improved assessment scalebased on the basic functions thatwill focus on that which SPs dobest: observe examinee behaviors;

• modified SP training processes toenrich portrayals and makeassessment observations moreprecise; and

• maintenance of the standardiza-tion that is needed for high-stakesexaminations across a far-flungassessment network.

Longer term, the more advanced func-tions will be incorporated. As theComprehensive Review of USMLE(CRU)-driven process pushes us toconsider additional methods of assess-ing communication skills, the assess-ment construct that has been devel-oped will serve as an important foun-dation and guide.

Table 2: Comparison of Original and Enhanced Construct for Step 2 CS

Bibliography

1 King, A. & Hoppe, R. (2009). Assessing

communication skills in the doctor-patient

encounter: Where are we? A “best prac-

tices” answer. Unpublished manuscript.

2 National Alliance for Physician Competence.

(2009). Good Medical Practice USA.

http://gmpusa.org/ (last accessed

November 2010).

3 Raymond, M., et al. (2010). The clinical

responsibilities of first-year residents

during their initial months of graduate

medical education. NBME Research Report

2010-01.

4 de Haes, H., & Bensing, J. (2009). Endpoints

in medical communication research, pro-

posing a framework of functions and out-

comes. Patient Education and Counseling,

74(3), 287-94.

6

(Communication Skills, continued from page 5)

CIS = communication and interpersonal skills.

7

test item formats will not onlyaddress the specific recommendationrelated to interpreting medical litera-ture but would also touch on anoth-er CEUP recommendation, ie, thatfoundational science (and the appli-cation of scientific reasoning andevidence-based decision making) bereflected across all three Steps.

Two new assessment formats areunder development in support of thisproject. These new items may usethe F-type format (ie, sequential itemsets) or the standard multiple-choicequestion (MCQ) format of one bestanswer:

• Drug Ad Format – MCQ-baseditem sets that assess examinees’ability to apply their knowledgeof biostatistics, epidemiology,pharmacology and/or publichealth in appraising informationprovided in drug advertisements.Some sets will begin with a clini-cal scenario describing a patientcare situation related to theabstract.

• Abstract Format – MCQ-baseditem sets that assess examinees’ability to apply their knowledgeof biostatistics, epidemiology,and/or public health in apprais-

ing information provided in anarticle abstract. Some sets willbegin with a clinical scenariodescribing a patient care situa-tion related to the abstract.

Interpreting drug advertisementsand article abstracts and appraisingthe medical literature are commonclinical tasks that provide a naturalcontext for authentic assessment ofthese skills. In 2011, the NBMEwork will focus on three majorareas: 1) development of test mate-rial; 2) development of software fortest delivery; and 3) validityresearch.

(New Item Formats, continued from page 1)

The Wind Helix by artist Roy Wilson was commis-sioned in 1994 for the NBME’s then-new headquar-ters. The artwork, a kinetic fence of steel panels alongthe perimeter of the NBME site, has 600 welded arcspositioned to describe the double-helix design of theDNA molecule. Aluminum vanes within the designchange position and color with the wind. In 2010, thisdouble-helix design was incorporated into the walk-ways of the NBME garden. Photos © Jeffrey Totaro