narrative review cognitive interventions to reduce ... · pdf filecognitive interventions to...

Cognitive interventions to reducediagnostic error: a narrative review

Mark L Graber,1,2,3 Stephanie Kissam,3 Velma L Payne,4,5 Ashley N D Meyer,6,7

Asta Sorensen,3 Nancy Lenfestey,3 Elizabeth Tant,3 Kerm Henriksen,8

Kenneth LaBresh,3 Hardeep Singh6,7

ABSTRACTBackground: Errors in clinical reasoning occur in most

cases in which the diagnosis is missed, delayed or

wrong. The goal of this review was to identify

interventions that might reduce the likelihood of these

cognitive errors.

Design: We searched PubMed and other medical and

non-medical databases and identified additional

literature through references from the initial data set

and suggestions from subject matter experts. Articles

were included if they either suggested a possible

intervention or formally evaluated an intervention and

excluded if they focused solely on improving

diagnostic tests or provider satisfaction.

Results: We identified 141 articles for full review, 42

reporting tested interventions to reduce the likelihood

of cognitive errors, 100 containing suggestions, and

one article with both suggested and tested

interventions. Articles were classified into three

categories: (1) Interventions to improve knowledge

and experience, such as simulation-based training,

improved feedback and education focused on a single

disease; (2) Interventions to improve clinical reasoning

and decision-making skills, such as reflective practice

and active metacognitive review; and (3) Interventions

that provide cognitive ‘help’ that included use of

electronic records and integrated decision support,

informaticians and facilitating access to information,

second opinions and specialists.

Conclusions: We identified a wide range of possible

approaches to reduce cognitive errors in diagnosis.

Not all the suggestions have been tested, and of those

that have, the evaluations typically involved trainees in

artificial settings, making it difficult to extrapolate the

results to actual practice. Future progress in this area

will require methodological refinements in outcome

evaluation and rigorously evaluating interventions

already suggested, many of which are well

conceptualised and widely endorsed.

INTRODUCTION

Although the rate of diagnostic error inpractice is unknown, experts estimate it to bein the range of 10%e15%.1 Diagnostic errors

are of great concern in all specialties andthose characterised by high levels of stress,workload and distractions are particularlyvulnerable. Errors are more likely when thelevel of uncertainty is high, if clinicians areunfamiliar with the patient, and when thereare atypical or non-specific presentations ofa common disease or ‘distracting’ comorbidconditions.2

Diagnostic errors reflect the complexinterplay of system-related and cognitivefactors, typically with multiple root causesidentifiable in a single case.3e6 Cognitiveerrors can be found in the majority ofcases.4 7 Given the dominant role thatcognitive shortcomings play in contributingto diagnostic error, it is appropriate to beginconsidering what could be done to helpminimise the likelihood of these errors. Wetherefore conducted an analytic review of theliterature to identify interventions to reducethe likelihood of cognitive errors or error-related harm in healthcare. Interventionsrelating to system-related factors werediscussed in a companion publication.8

METHODS

Our search strategy has been previouslydescribed.8 Briefly, we sought articles, booksand conference presentations relating to theprevention, reduction or mitigation of diag-nostic errors in PubMed and several othermedical and non-medical databases. Wepursued references from these sources andasked authorities in the field of appliedcognition and decision-making to recom-mend additional readings. Articles and bookswere included in this analysis if theycontained results from an intervention trialor suggested an intervention to reducecognitive-related diagnostic error. Publica-tions that focused on development or

< Additional appendices arepublished online only. Toview these files please visitthe journal online (http://qualitysafety.bmj.com/content/21/7.toc).

For numbered affiliations seeend of article.

Correspondence toDr Mark L Graber, RTIInternational, c\o 1 BreezyHollow, St James, NY 11780,USA; [email protected]

The authors of this paper aresolely responsible for itscontent, and disclosed nocompeting interests. Thefindings and interpretationsin the paper do not representthe opinions orrecommendations of theinstitutions with which theauthors are affiliated, theAgency for HealthcareResearch and Quality, or theUS Department of Health andHuman Services, Departmentof Veterans Affairs.

Accepted 20 February 2012Published Online First27 April 2012

BMJ Qual Saf 2012;21:535e557. doi:10.1136/bmjqs-2011-000149 535

Narrative review

group.bmj.com on October 31, 2012 - Published by qualitysafety.bmj.comDownloaded from

http://qualitysafety.bmj.com/

http://group.bmj.com/

refinement of specific diagnostic tests or technologies,or solely on the aetiology or epidemiology of error, ordealt primarily with provider satisfaction or preferenceswere excluded.A full-text review using an approach described by

Gordon and Findley9 was performed on the 42 empiricalstudies that tested an intervention. Nineteen quality-based criteria were independently extracted from eacharticle using a data extraction form (online appendix A).Items answered with ‘yes’ or ‘no’ included literaturereview described, clear objectives reported, study designreported, appropriate design to address objectives,control group used, subjects randomised, blinding used,intervention clearly described, resources described,outcomes match objectives, statistical tests used, statis-tical tests appropriate, data collection replicable, studyreplication possible and limitations discussed. Addi-tional items assessed were the study design, subjectcharacteristics and number of subjects. Based on theseitems, we assigned an ‘Outcomes Rating’ and ‘Strengthof Conclusions’ rating to each article (detailed instru-ments in online appendix B). The Outcomes Rating wasbased on Kirkpatrick’s hierarchy9 10 that we slightlymodified for use in assessing diagnostic errors. Thishierarchy demonstrates the level of impact of eachintervention on diagnostic errors (eg, Level 2b refers toan intervention in which an acquisition of conceptsmight impact diagnostic error, whereas Level 4b refers toan intervention that directly reduces diagnostic error).The Strength of Conclusions of each study was rated ona numerical scale (1e5) in accordance with BestEvidence in Medical Education guidelines.9 11 Thisrating is not an assessment of the overall methodologicalquality, but is a measure of how well the conclusionsmade are supported by the data presented.Two reviewers with expertise in cognitive psychology

(ANDM and VLP) assessed each of the interventionstudies independently. We assessed agreement betweenthe reviewers for the Outcomes Rating and the Strengthof Conclusions with Cohen’s k statistic. Differences wereresolved by discussion between the two reviewers and incases of disagreement, another investigator (SK)reviewed and rated the article. In these cases, we usedconsensus among these three reviewers to determine thefinal ratings.Based on a prior classification scheme,1 all articles

were assigned to one of three natural categories: (1)Interventions that increase medical knowledge andexperience; (2) Interventions that improve clinicalreasoning; and (3) Interventions that involve gettinghelp. Articles were further subdivided into more specifictypes of interventions (such as ‘focused training onspecific content areas’, ‘develop simulation exercise toexpose clinicians to a greater number and variety of

cases presentations’, etc.) to facilitate the synthesis of thefindings (tables 1e4).

RESULTS

We identified 141 sources (articles, books and confer-ence papers) for full review. Of these, 42 sources (tables1e4) reported empirical studies of an intervention toreduce cognitive-based diagnostic error (and sometimesalso additional suggestions for interventions), 100sources contained only suggestions (table 5) and onehad both. Some sources reported more than onesuggestion.During the full-text review of the empirical studies

assessing cognitive interventions, agreement betweenreviewers on the Outcomes Rating was substantial(k¼0.70). Similar agreement was obtained for theStrength of Conclusions (k¼0.70). There were threearticles with disagreements that were resolved bydiscussions with a third reviewer. We categorised theintervention studies into one of three mutually exclusivecategories: (1) Interventions to increase clinicians’knowledge and experience, (2) Interventions to improveclinical reasoning and decision-making skills or (3) ‘Gethelp’, interventions that assist clinicians with tools oraccess to other clinicians or experts. For each of thesesections, we use the suggested intervention and back-ground literature to first provide context, followingwhich we discuss the tested interventions. The OutcomesRatings and Strength of Conclusions ratings for eachintervention article are included in tables 1e4.

1) Increase knowledge and experienceDiagnostic error could potentially be reduced byincreasing physician’s structured knowledge and expe-rience, the essential basis of expertise.143 By definition,experts will tend to make the fewest errors, have thebest degrees of calibration and excel in efficientdiagnosis.50 144 145 Medical educators similarly agree withthe concept of increasing experience as the key todeveloping expertise.143 146 147 The interventions in thisdomain are summarised in table 1 and are organisedinto the following three categories.

Training focused on specific content areas

An effect of training on diagnostic reliability is illustratedin radiology, where certain certification programmes arebased on demonstrating competency. For instance,radiologists in the UK must review 5000 mammogramsa year for certification, as opposed to 480 in the USA,which may in part account for the large difference indiagnostic accuracy noted between the two countries.54

In certain programmes, radiologists also receive

536 BMJ Qual Saf 2012;21:535e557. doi:10.1136/bmjqs-2011-000149

Narrative review




Table

1Interventionsto

increasemedicalknowledgeandexperience

Author

(year)

Studytype

andparticipants

Intervention

Outcomemeasures

Results

Conclusions

Outcomes

Rating

Strength

of

Conclusions

(1e5)

Training:focusedtrainingonspecificcontentarea

Fridriksson

etal12

(2001)

Grouprandomised

withnursesand

physicians

Trainingonidentifying

subarachnoid

haemorrhages.

Diagnosticerrors

basedon

subarachnoid

haemorrhagerupture

ratesin

actualp

atients.

Decreasein

diagnosticerrors.

Trainingcanhavea

profoundim

pacton

diagnosticerror

outcomeatalow

cost.

4b

4

Rezvyy

etal13

(2008)

Before/afterwith

psychiatrists

Additionofdiagnostic

codingtrainingto

generalpsychiatric

educationprogramme

requiredforpsychiatric

recertification.

Diagnosticaccuracy

ofclinicalcase

vignettesafter

training.

Diagnosticaccuracy

improvedmore

than

controlgroupwhose

educationprogramme

focusedonly

on

generalpsychiatry.

Trainingfocusedon

diagnosticcodingcan

positively

affect

diagnosticaccuracy

ofcasevignettes.

Furtherresearch

neededto

determ

ine

ifim

provementleads

tochangesin

routine

diagnosticpractice.

4b

4

Gutm

ark

etal14

(2007)

Surveywith

physicians(alllevels)

Useofteaching

databaseofradiology

casesdiagnosed

accurately

or

inaccurately.

Physician’s

opinion

ofthecollectionof

casesavailable

indatabase.

Databasemostused

byresidents

and

fellowsforlearning

andreference.

Attendingsusedit

forteaching.

Teachingdatabases

canbeusefulfor

trainingandreference.

Researchneededto

determ

ineeffects

of

toolondiagnostic

accuracy.

32

Sim

ulation:developsim

ulationexercisesto

exposecliniciansto

agreaternumberandvariety

ofcasepresentations

Bond

etal15

(2004)

Caseserieswith

2ndand3rd

year

emergency

medicineresidents

Embedcognitive

errortrapsin

cases

within

sim

ulationlab

tointroducecognitive

forcingstrategiesto

dealwitherrors.

Debriefingusedto

review

errors.

Self-reportof

residents’

perceptionof

benefitof

intervention.

Residents

said

sim

ulationwas

beneficial(though

lessthandirect

patientcare).

Increasedknowledge

ofcognitiveerrors

andforcingstrategies.

Knowledgewasnot

actually

assessed.

Educatingresidents

on

cognitiveerrors

and

forcingstrategiesis

promising,butwarrants

furtherstudyand

quantificationofthe

effectivenessof

educationaltechniques.

2b

1

Continued


Narrative review




Table

1Continued

Author

(year)

Studytype

andparticipants

Intervention

Outcomemeasures

Results

Conclusions

Outcomes

Rating

Strength

of

Conclusions

(1e5)

Carlson

etal16

(2011)

Before/afterwith

4th

yearmedical

students

Useofdiagnostic

remindersystem

(ISABEL)to

reduce

diagnosticerrorwhile

cliniciansassessed

livepeople

trainedas

patients

oramanikin

inasim

ulationlab.

Changein

diagnostic

accuracyafter

intervention.View

of

diagnosticreminder

system

asan

educationtooland

resourceforpractice.

Diagnosticaccuracy

improved.Toolwas

viewedasbeneficial

asatrainingtooland

asusefulin

practice.

Diagnosticreminder

systemscanpotentially

reducecognitive-based

diagnosticerrors.The

optimaltimewithin

trainingto

introducethe

useofsuchtools

remainsunclear.Itis

unclearwhether

diagnosticaccuracyis

increasingsim

ply

becauseofadditional

timespenton

diagnosingproblem.

2b

2

Feedbackandcalibration:provideintensive,detailed,specificfeedback

Tudorand

Finlay17

(2001)

Before/afterwith

expertradiologists

Review

feedback

regardingdiagnostic

errors

made

diagnosingradiographs.

Diagnosticaccuracy

rate

4e5monthsafter

reviewingfeedback.

Diagnosticerrorrate

insamecases

decreasednominally

(nostatisticsreported).

Allowingradiologists

toreview

theirerrors

may

decreasediagnostic

errorrate.More

researchneededto

determ

ineifeffects

generaliseto

new

cases/domain.

32

Woodand

Tracey18

(2009)

Grouprandomised

controltrialwith

doctoralstudents

Receivingcontent-based

or

principle-based

feedbackto

reduce

diagnosticerrorof

overshadowing.

Reductionof

overshadowing

acrossfeedback

conditions.

Sim

ilardecreasein

overshadowingfor

both

feedback

interventions.Less

reductionin

control

group.Generalisation

effectforcontent-

basedfeedback.

Givingclinicians

feedbackregarding

specificattributesofa

caseorcognitive

diagnosticreasoning

strategiesmayreduce

diagnosticerror

involving

overshadowing.

4b

4

OutcomesRatin

gsreflectthelevelo

fim

pactforeachinterventio

nonreducingdiagnostic

errors.9

10Strength

ofConclusionswasratedonanumericalscale

(1e5)in

accordancewithBestEvidence

inMedicalEducatio

nguidelines(5¼s

trongest).9

11


Narrative review




Table

2Interventionsto

improveintuitive(system

1)anddeliberate

(system

2)processesin

decision-m

aking

Author(Year)

Studytype

andparticipants

Intervention

Outcome

measures

Results

Conclusions

Outcomes

Rating

Strength

of

Conclusions

(1e5)

Improvesystem

1processing:im

provetrainingonintuitiveprocessinganditsshortcomings

Sherbinoetal19

(2011)

Caseserieswith

4th

yearresidents

Teachingcognitive

forcingstrategiesto

introducediagnostic

errorrisk,identifyand

counterm

andbiases.

Diagnosticaccuracy

invariousdomains

aftervaryingdelays.

Aftertrainingabout

halfthesubjects

stillcommittedbias.

Perform

ance

sufferedeven

furtherafteradelay.

Teachingcognitive

forcingstrategies

mayreducediagnostic

errors

inatransferable

way(thoughitis

difficultto

tellwithout

perform

ance

measurespriorto

training).Also,the

effects

were

short-lived.

32

Evaetal20

(2007)

Randomisedcontrol

trialw

ithundergraduate

psychologystudents

Usecombined

approachto

reasoning

includingpattern

recognitionand

carefulconsideration

ofpresentingfeatures

whendiagnosing

ECGswithbiasing

inform

ation.

Diagnosticaccuracy

ofECG

readings.

Diagnosticaccuracy

improvedforsubjects

usingcombined

reasoningapproach

evenin

presenceof

biasinginform

ation.

Traineescanbenefit

from

explicitguidance

ofacombined

approachto

clinical

reasoning,

suggestingavalue

toboth

analyticand

non-analytical

reasoningtendencies.

4b

4

Improvesystem

1processing:im

provemetacognition,intuitionandreflectivepractice

Mamedeetal21

(2010)

Before/afterwith

1stand2ndyear

residents

Incorporationof

reflectivereasoning

toreducediagnostic

errors.

Diagnosticerrors

producedbefore

andafterreflective

reasoning.

Significantnumber

ordiagnoseswere

correctedafter

reflectivereasoning,

presumably

reducingbias.

Diagnosticaccuracy

maybeim

proved

withreflective

reasoning.More

researchneededto

disentangle

reflective

reasoningfrom

additionaltime

assessingacaseto

determ

ineifreflection

generalisesto

experts.

32

Continued


Narrative review




Table

2Continued

Author(Year)

Studytype

andparticipants

Intervention

Outcome

measures

Results

Conclusions

Outcomes

Rating

Strength

of

Conclusions

(1e5)

Coderreetal22

(2010)

Before/afterwith1st

yearmedicalstudents

Havestudents

reflect

oninitialhypothesis

establishedpriorto

reviewingallclinical

evidence.Then

presentadditional

concordantor

discordantevidence.

Diagnostic

accuracy.

Withdiscordant

data,significant

increasein

diagnostic

accuracy.No

differencein

accuracywith

concordantdata.

Overall,

no

significant

differencein

diagnosticaccuracy.

Providingstudents

withadditional

inform

ationafteran

initialdiagnosis

affects

diagnostic

accuracydifferently

dependingonthe

typeofinform

ation

received.More

researchneeded

todeterm

inevalidity

oftheintervention.

31

Improvesystem

1processing:consideralternatives,considertheopposite,useprospectivehindsight,thinklikeanoutsider

Wolpaw

etal23

(2009)

Randomisedcontrol

trialwith3rd

year

medicalstudents

Incorporationof

SNAPPStechnique

duringcase

presentationto

facilitate

learning.

Diagnostic

reasoningmeasures

includingnumberof

minutesto

present

findings,was

summary

concise

andthorough,etc.

SNAPPSgroup

showedmore

diagnostic

reasoningthan

afeedback

comparisonand

acontrolgroup.

UsingtheSNAPPS

techniquemayresult

incorrectionof

flawedreasoning,

reductionof

diagnosticand

therapeuticerrors.

Studyonly

measured

amountofreasoning

andnotaccuracy

thereof,soitis

unclearhow

this

interventionwould

improvediagnostic

accuracy.

2b

3

OutcomesRatingsreflectthelevelo

fim

pactforeachinterventiononreducingdiagnosticerrors.9

10Strength


(1e5)in

accordancewithBestEvidence

inMedicalEduca

tionguidelines(5¼s

trongest).9

11SNAPPS:Summarizehistory

andfindings,Narrow

thedifferential,Analyze

thedifferential,Probepreceptoraboutuncertainties,Plan

management,Selectcase-relatedissuesforself-study.


Narrative review




Table

3Interventions:help

from

otherpeople

Author(year)

Studytypeand

participants

Intervention

Outcomemeasures

Results

Conclusions

Outcomes

Rating

Strength

of

conclusions

(1e5)

Secondopinionsin

pathology

Raabetal24

(2008)

Before/afterwith

expertcytologists

Useofsecond

readingspre

sign-out

atthreeinstutions,

comparingrandom

reviewsto

organ-

targetedreviews.

Proportionof

diagnosticerrors

detected.

Few

diagnosticerrors

detected;no

significantdifferences

amongsites;Tissue-

specificreviews

yieldedhighererror

ratesthanrandom

reviews.

Tissue-specific

reviewsyieldedhigher

errorratesthanrandom

reviews.

2b

2

Raabetal25

(2006)

Before/afterwith

expertpathologists

Secondreadingof

pathologycases.

Random

review

of5%

ofcasesandfocused

review

ofallcases.

Percentofdiagnostic

errors.Im

pactof

differenceonpatient

care.

Focusedreview

detected

approxim

ately

four

timesmore

diagnostic

errors

than5%

random

review.The

majority

oferrors

inboth

groupsdid

not

leadto

patientharm

or

resultedin

low-grade

harm

.

Secondopinionreviews

canbeamethodto

standardisediagnostic

practice.

4a

3

Manionetal26

(2008)

Before/afterwith

expertpathologists

Secondreadingof

pathologyslides

receivedfrom

an

externalorganisation.

Rate

ofdiagnostic

variationandchange

inpatient

managementdueto

secondreading.

Nodisagreementin

majority;minor

disagreementin

small

%;major

disagreementin

very

small%

ofcases.

Changein

managementplanin

halfofcaseswith

majordisagreement.

Mandatory

second

opinionofsurgical

pathologymaybe

abeneficialpatient

care

practice.However,

upondisagreement,

itis

notclearhow

often

thesecondopinion

wascorrectdueto

inconclusivechart

reviews.

4a

2

Nordrum

etal27

(2004)

Before/afterwith

expertpathologists

Useofstillim

agesin

secondopinionof

pathologycases.

Diagnosticaccuracy

rate

(glassslidesvs

stillim

ages).

Nearlythesame

diagnosticaccuracy

rate

withstillim

ages

andglassslides.

Usingstillim

agesto

diagnosecasesappears

tobecomparable

tousingglassslides,thus

increasingeaseof

obtainingsecond

opinions.

33

Continued


Narrative review




Table

3Continued

Author(year)

Studytypeand

participants

Intervention

Outcomemeasures

Results

Conclusions

Outcomes

Rating

Strength

of

conclusions

(1e5)

Hamadyetal28

(2005)

Before/afterwithan

expertsurgeon,

oncologistand

pathologist

Useofsecond

opinionsof

amultidisciplinary

team

ofclinicians.

Percentageofsecond

opinionsresultingin

differentdiagnosis.

Complete

agreement

inmajority

ofcases.

Disagreementasmall

%ofthetime.

Diagnosticand

therapeutic

discrepanciescan

occurwhenmultiple

expertsreview

thesame

patientcase.Itisunclear

ifthesecondopinion

leadsto

better

outcomes.

4a

4

Secondopinionsin

radiology

Bengerand

Lyburn

29

(2003)

Before/afterwithER

andradiologystaff

Secondreadingof

radiographsbythe

radiologystaffforx-

raysprocessedby

ER.

Rate

ofdiagnostic

agreement.Clinical

impactofdiagnostic

discrepancy.

Very

smallamountof

discrepanciesthat

requiredminuscule

changein

management.

Thelow

rate

of

significantmisread

radiographssuggests

incorporationof

selectivesecond

readingsmaybe

warranted.

4a

3

Espinosa

andNolan30

(2000)

Before/afterwithER

physiciansand

radiologists

x-R

aysreadbyER

physicianand

radiologist.

Radiograph

interpretationerror

andnumberof

potentialadverse

events.

Interpretationerror

rate

andpotential

adverseeffects

decreased(basedon

reliability

modelnot

raw

data).

Proceduresfor

interpretingradiographs

designedto

mitigate

errors

canreducethe

adverseevents.Without

acontrolgroupitis

difficultto

know

ifim

provementis

from

intervention.

4a

2

Duijm

etal31

(2007)

Before/afterwith

mammography

techniciansand

radiologists

Secondreadingof

mammogramsby

technologists,along

withstandard

double

readingby

radiologists.

Breastcancer

detectionandpositive

predictivevalue(PPV)

ofreferral.

Modestincreasein

cancerdetectionand

modestdecreasein

PPV.

Addingsecondreading

bytechnologists

may

beeffectivein

detecting

more

breastcancer

cases.Readingsshould

beconsideredfor

referraldueto

high

prevalenceofbreast

cancer.

4a

4

Continued


Narrative review




Table

3Continued

Author(year)

Studytypeand

participants

Intervention

Outcomemeasures

Results

Conclusions

Outcomes

Rating

Strength

of

conclusions

(1e5)

Kweketal32

(2003)

Before/afterwith

expertpathologists

Blindedsecond

readingsin

mammography.

Rate

ofcancer

detection,patient

recall,

rate

ofbiopsy

andmeansecond

screenercontribution.

Low

increaseof

cancerdetection.

Recallrate

increased

modestly.Biopsyrate

slightlyincreased.

Efficiencyofsecond

readerminim

al.

Secondreadingof

mammogramsis

recommendedfor

breastcancer

screeningif

resourcesare

available.

4a

4

Canonetal33

(2003)

Before/afterwith

expertradiologists

Secondreview

of

barium

enematests.

Detectionofpolyps.

Secondreadingfailed

toim

provedetection

ofpolyps.

Routinesecondreading

isnotwarrantedfor

barium

enema

examination.

4a

4

Help

from

groupsandlibrarians

Christensen

etal34(2000)

Non-randomised

controlledtrialwith

clinicalteams

Team

diagnostic

decision-m

aking

where

members

were

givensharedor

private

inform

ation

thatthegroupneeded

toshare

forcorrect

diagnoses.

Diagnosticerrorrate.

Diagnosticerrors

increasedwhenteam

members

held

private

inform

ation.

Lackofsharingdata

maybedetrim

entalto

diagnosticaccuracy.

Clinicaldecisions

relyingonprivately

held

inform

ationare

susceptible

toerrors.

2b

4

Mulvaney

etal35(2008)

Randomisedcontrol

trialw

ithclinicalteams

Useofevidence

basedinform

aticstool

thatprovidesresearch

evidenceto

inform

cliniciansofpatient

care

practices.

Impactonpatientcare

practicesandclinical

actions,articlesread,

satisfactionofsearch

results,consultations,

timeto

obtain

evidence,clinician

searches.

Toolhadsignificant

impactonusers’

reportoffuture

patient

care,satisfactionof

articlesreturnedand

amountoftimespent

receivingevidence.

Nosignificantim

pact

onotheritems.

Inform

aticstools

may

facilitate

useof

researchevidenceand

influenceclinicalactions.

However,data

regarding

effects

onpatients

are

unknownasaresultof

this

study.

34

OutcomeRatingsreflectthelevelo

fim

pactforeachinterventiononreducingdiagnosticerrors.9

10Strength


(1e5)in

acco

rdancewithBestEvidence

inMedicalEducatio

nguidelines(5¼s

trongest).9

11ER;EmergencyRoom.


Narrative review




Table

4Interventions:help

from

decisionsupport

Author(year)

Studytypeand

participants

Intervention

Outcomemeasures

Results

Conclusions

Outcomes

Rating

Strength

of

Conclusions

(1e5)

Focuseddecisionsupport

usingdecisionsupport

tools

onaspecificcondition

Pozenetal36

(1984)

Non-randomised

controltrialwithER

physicians

Incorporationof

decisionsupporttool

thatcalculatesthe

probability

ofpatient

havingischaemic

heartdisease.

Diagnosticaccuracy,

sensitivity,

specificity,FP

diagnosis

rate,FN

diagnosis

rate,CCU

admissionrate,ER

dischargerate.

Useoftoolincreased

diagnosticaccuracy

andspecificity;

sensitivityremained

unchanged.FPsand

CCU

admissions

decreased.FNs

remainedunchanged.

Useoftoolhas

potentialto

increase

diagnosticaccuracy

andreduceadmission

rate.

4b

4

Selkeretal37

(1998)

Non-randomised

controltrialwith

attendingER

physicianand

residents

Computerised

predictiontoolthat

calculated

probability

ofacute

ischaemia

thatwas

printedonECG.

Accuracyoftriage

decisions.

Toolreduced

unnecessary

CCU

admissionsanddid

notchange

appropriate

admission

rates.

Useofthetoolhas

thepotentialto

significantlyreduce

unnecessary

hospitalisations.

4b

4

Bogusevicius

etal38(2002)

Randomisedcontrol

trialwithradiologists

Useofacomputer-

aideddiagnosis

tool

todiagnoseacute

smallbowel

obstruction.

Specificity,

sensitivity,FP

predictivevalue,FN

predictivevalue,

timeto

diagnose,

mortalityand

morbidity.

Nosignificant

advantageover

contrastradiography

fordiagnostic

accuracy.Significantly

lesstimerequiredto

diagnosis.

Computer-aided

diagnosis

improved

amountoftimeto

diagnose,butno

otherindices.

4b

3

DeSim

one

etal39(2007)

Before/afterwith

medicalpersonnel

Useofcomputer-

assisteddiagnosis

databasefor

diagnosis

and

patientmanagement

ofheadaches

comparedwith

standard

clinical

method.

Diagnosticaccuracy,

meanvisitduration,

operators’subjective

opinionoftooluser

friendliness,

patients’subjective

opinionofcomputer-

assistedinterview

acceptability.

Slightincreasein

diagnosticaccuracy.

Durationofclinical

visitcomparable.

Subjects

felttoolwas

easyto

useand

patients

feltthetool

usewasacceptable.

Thetoolim

proves

diagnosticaccuracy

withoutincreasingvisit

duration.

4a

3

Continued


Narrative review




Table

4Continued

Author(year)

Studytypeand

participants

Intervention

Outcomemeasures

Results

Conclusions

Outcomes

Rating

Strength

of

Conclusions

(1e5)

Focuseddecisionsupport

usingembeddeddecisionsupport

tools

(Infobuttons)

Cim

ino40(2006)

Surveywithnurses,

attendings,

residents,medical

students

Useofonline

resourcesthrough

anInfobutton

Manager(IM)

providingdirect,

context-specific

inform

ation.

Usersatisfaction

regarding

inform

ationreturned

from

online

resources.

Satisfactionwiththe

IMvaried.Overall

users

feltIM

had

apositiveeffecton

patientcare

decisions.

Context-specific

accessto

health

knowledgeresources

canbeseenasuseful.

More

researchis

neededontheim

pact

onpatientoutcomes.

13

Generaldecisionsupport

usingweb-enableddifferentialdiagnosis

generators

Ramnarayan

etal41(2006)

Before/afterwith

paediatric

interns

andresidents

Useofaninternet

diagnosticdecision

supporttool

(ISABEL)during

diagnosis.

Changein

proportion

ofunsafe

diagnostic

investigationsand

meanqualityscore

ofdiagnosis

followingtool

consultation.

Significantproportion

ofunsafe

investigationsreduced

withtooluse.Mean

diagnosticquality

score

increased.

Useofstand-alone

diagnosticsystem

toim

provediagnostic

decision-m

akingfor

juniorphysiciansis

beneficial.However,

severalbarriers

must

beovercomein

order

forsuchtools

tobe

mosteffective.

33

Ramnarayan

etal42(2007)

Auditofsystem

Useofdiagnostic

decisionsupporttool

(ISABEL)when

diagnosingpatients

withresuscitation

issuesin

emergency

room.

Percentageoftime

accurate

diagnosis

wasin

listprovided

bytooland

proportionoftime

toolincludedmust-

not-missdiagnoses.

Toolcontainedcorrect

diagnosis

andmust-

not-missdiagnosis

innearlyevery

case.

Diagnosticaid

perform

san

acceptable

degreeof

clinicalaccuracyin

ED.Furtherresearch

isneededto

determ

inerole

oftool

inclinicalpractice.

2b

3

Ramnarayan

etal43(2003)

Auditwithclinicians

ofvaryinglevels

of

expertiseand

system

Useofinternet

diagnosticdecision

supporttool

(ISABEL)as

areminderof

possible

diagnoses

toconsider.

Proportionofcases

withexpected

diagnosis

inresults

generatedbytool.

Usinghypothetical

andrealpatient

cases,toolreturned

thecorrectdiagnosis

innearlyevery

case.

Toolshowed

acceptable

clinical

accuracybyproviding

correctdiagnosis

for

realandhypothetical

cases.Itis

anticipated

theuseoftoolis

effectivein

assisting

physiciansto

accurately

diagnosechildren.

2b

4

Continued


Narrative review




Table

4Continued

Author(year)

Studytypeand

participants

Intervention

Outcomemeasures

Results

Conclusions

Outcomes

Rating

Strength

of

Conclusions

(1e5)

Ramnarayan

etal44(2006)

Before/afterwith

cliniciansofvarying

levels

ofexpertise

Useofadiagnostic

system

(ISABEL)to

determ

inediagnosis

afterdiagnosing

casewithoutsystem.

Rate

ofdiagnostic

errors,diagnostic

qualityscore

and

timeusingsystem.

Declinein

diagnostic

errors.Increasein

diagnosticquality

score.Nosignificant

outcomebylevelof

experience.Median

timeforsystem

use

was1min.

Studysuggests

promisingrole

for

reminder-based

diagnosticdecision

supporttoolto

reduce

diagnosticerrors.

4b

4

Graberand

Matthew45

(2008)

Before/afterwith

physiciansand

system

Useofdiagnostic

decisionsupporttool

(ISABEL)to

determ

inecorrect

diagnosis

uponentry

ofpatientkey

findingsofcomplex

medicalcases.

Percentageofcases

where

toolreturned

thesamediagnosis

aslistedin

the

NEJM.Amountof

timeusingthetool.

Whenentering

evidencemanually

toolreturnedcorrect

diagnosis

innearlyall

cases.Whenpasting

casetextaslistedin

NEJM,toolcontained

correctdiagnosis

three-fourthsofcases.

Both

entry

approacheswere

fast.

Toolperform

ed

quickly

andaccurately

insuggestingcorrect

diagnosesandshould

beevaluatedin

naturalenvironments

todeterm

ineits

potentialto

support

clinicaldiagnosis

and

reducetherate

of

diagnosticerrors.

2b

3

TangandNg46

(2006)

Auditwithphysician

andrheumatologist

SearchingGoogle.

com

todeterm

ine

correctdiagnosis

for

casepresentedin

NEJM.

Percentageof

diagnosesfrom

Google

that

correspondedwith

NEJM

diagnosis.

Google

searches

revealedcorrect

diagnosis

inslightly

more

thanhalfof

cases.

Astheinternet

becomesmore

available

inclinical

settings,useofweb-

basedsearchingtools

mayhelp

physicians

diagnosedifficult

cases.

2b

2

OutcomeRatingsreflectthelevelo

fim

pactforeachinterventiononreducingdiagnostic

errors.9

10Strength


(1e5)in

acco

rdancewithBestEvidence

inMedicalEduca

tionguidelines(5¼s

trongest).9

11CCU,Coronary-care

unit;ER,EmergencyRoom;FN,FalseNagetive;FP,FalsePositive;NEJM

,New

EnglandJournalofMedicine.


Narrative review




additional training in cancer detection where theyattend disease-related meetings, receive feedback oncancer detection rates and attend a 2-week course led byspecialists at high-volume mammography screeningsites.55 Similar measures, including regular peer reviewand participation in the American College of Radiol-

ogy’s RADPEER� system, have been proposed for theUSA.148

Interventions to increase the knowledge base ofpracticing clinicians through continuing medicaleducation activities have generally not led to substantialimprovement in measured performance.56 57

Table 5 Intervention suggestions

Intervention idea Suggested in article authored by

INCREASE MEDICAL KNOWLEDGE AND EXPERIENCEGeneralIncrease expertise and experience Kassirer et al47; Gigerenzer48; Elstein49; Klein50 51; Berner

and Graber1; Bazerman and Moore52; Norman53

TrainingRoutinely test competency in diagnostic accuracy;provide training to increase competency

Smith-Bindman et al54; Singh et al55

Continuing education Bowen56; Davis et al57

Focused training on specific content area Freidlander and Phillips58; Gentner59; Hershberger et al60;Parmley61

Improve learning skills, per se Hogarth62

SimulationDevelop simulation exercises to expose clinicians toa greater number and variety of case presentations

Bond et al63

Feedback and calibrationProvide intensive, detailed, specific feedback Smith-Bindman et al54; Schiff64; Jamtvedt et al65; Papa

et al66; Stone and Opel67; Alpert and Hillman68; Arkes69;Humble et al70; Pulford and Colman71; Subbotin72

Learn from errors Fischer et al73; Hogarth62; Eva74

IMPROVE INTUITIVE AND DELIBERATE PROCESSES IN DECISION-MAKINGGeneralImprove general training on clinical reasoning & the dualprocess model

Berner and Graber1; Kassirer et al47; Elstein49; Eva74 75;Croskerry76e78; Norman53; Wolpaw et al23

Improve system 1 processingImprove training on intuitive processing and itsshortcomings

Berner and Graber1; Croskerry76e78; Wedding and Faust79

Groves et al80; Pines81; Moulton et al82, Trowbridge83;Kuhn84

Debias your own intuitive decisions Croskerry 76e78; Norman53; Fischhoff85; Milkman86 Larrick87;Gentner et al59; Koriat et al88; Renner and Renner89; Estradaet al90 Scott91; Slovic and Fischoff92 Arkes et al69 154;Lichtenstein et al93

Improve metacognition, nurture intuition and use ofreflective practice

Schon94; Greenhalgh95; Mamede et al96; Singh et al97;Brawn98; Gregory99; Noddings and Shore100; Quirk101;Klein51; Hogarth62; Hamm and Zubialde102; Moulton et al82;Trowbridge83; Croskerry76e78 103

Use a checklist or related tools Leonidas104; Gawande105; Trowbridge83; Ely et al106

Consider alternatives; consider the opposite; useprospective hindsight; think like an outsider

Taleb107; Wedding and Faust79; Sackett et al158; Brannick etal108; Milkman et al86; Arkes109; Croskerry78; Schwenk110;Baron111; Mitchell et al112; Bazerman and Moore86;Mussweiler113; Lord et al114; Arzy et al115; Gorman andGorman116; Hirt and Markman117; Mumma and Steven118;Singh97

Continued


Narrative review




Interventions

We identified three formal studies of training inter-ventions related to diagnostic error.12e14 One notablestudy was a highly content-specific intervention toimprove recognition of subarachnoid haemorrhage.This low-cost training programme on sudden onsetheadache for community-based physicians reducedthe baseline diagnostic error rate (12%) by 77% andimproved interactions between neurosurgeons andlocal physicians.12

Simulation

The ability to provide realistic simulations through bothscenarios and simulated patients offers the potential toimprove skills in clinical reasoning63 and the opportu-nity to expose trainees or physicians to a greater numberand variety of case presentations. Simulation is a well-established approach to improving manual, proceduralskills, but has not yet been evaluated extensively in itsability to improve cognitive skills or decision-makingrelated to diagnosis. It also remains to be demonstratedthat simulation can replace experience in actualpractice.

Interventions

We identified only two interventions in this domain,both involving trainees. Carlson et al16 demonstratedimproved diagnostic accuracy by the combined use ofsimulation with a diagnosis support tool and Bond et

al15 used simulation successfully to introduce the useof cognitive forcing strategies to emergency medicineresidents.

Feedback as a way to improve expertise, calibration and error

awareness

Deliberate practice, with immediate and focused feed-back, is viewed as an essential prerequisite to developingexpertise in any domain.144 145 Moreover, lack of feed-back is a dominant factor that sustains overconfidence,thought to be a major factor in causing diagnostic error.1

A systematic review of feedback across all medical areas(not solely diagnosis) concluded that feedback improvesperformance in selected settings, especially if the feed-back is intensive.65 Feedback is most useful if it incor-porates instruction and information on why a givenanswer was correct or not.66 67 For example, psychologytrainees improved their diagnoses if feedback provided

Table 5 Continued

Intervention idea Suggested in article authored by

Improve system 2 processingTeach principles of clinical reasoning; use evidence-based medicine and normative decision-making

Wedding and Faust79; Strauss et al119; Sox120; Dobbieet al121; Khan and Coomarasamy122; Croskerry123; Pines81;Ullman124; Pauker and Kassirer125; Kassirer126; Brannicket al108

Provide training on the typical pitfalls of specific clinicalconditions & situations

Groves et al80; Croskerry78; Pines81

GET HELP FROM OTHER PEOPLE AND/OR DECISION SUPPORT TOOLSSecond opinionsUse specialist consultants & second opinions; improveteam-based decisions, for example, by having a devil’sadvocate

Elstein49; Christensen et al34; Tatarka127

Second readings in pathology Ullman124; Raab128

Groups and librariansGet help from groups or librarians Zipperer and Sykes129; Albert130; Zipperer131

Actuarial decisionsUse of guidelines, clinical algorithms, linear models andmnemonics to reduce reliance on memory

Elstein49; Croskerry78; Berner and Graber1; Tatarka127;Milkman et al86; Wedding and Faust79; Wedding132;Fischhoff85

Focused decision supportDecision support tool on a specific condition Hunt et al133; Klassen et al134; Garg et al135; Cannon and

Allen136

Improve data display through graphics Radecki and Medow137; Reyna et al138; Bhandari et al139;Cook and Smallman140

Embedded decision support tools; Infobuttons Kawamoto and Loback141

General decision supportImprove medical records Hamm and Zubialde102; Schiff and Bates142


Narrative review




details on why they were right or wrong.18 149 Usingfeedback to improve diagnostic performance has beenmost convincingly demonstrated in radiology throughprogrammes such as the ‘PERFORMS’ system in theUK54 and the RADPEER programme in the USA.Although clinicians received immediate and dramatic

feedback on their diagnostic performance from autop-sies, the rate of autopsies is declining.150 Local‘Morbidity and Mortality’ conferences73 and creative newvenues such as the ‘Web M&M’ series sponsored by theAgency for Healthcare Research and Quality (http://www.webmm.ahrq.gov/) are alternative venues wherefeedback is provided.151 In this spirit, Eva74 has advo-cated for incorporating diagnostic error review intomedical school and postgraduate training. Alpert andHillman discuss other types of data that should be partof such feedback, such as the results of professionalaudits, peer reviews and risk managementprogrammes.68

Interventions

Our search yielded two studies on feedback to improvediagnostic performance.17 18 Both studies showedbenefits of feedback on later diagnostic accuracy. Thepositive impact noted by Wood and Tracey18 waspossibly explained by the provision of detailed feed-back to trainees on the reasons their initial diagnoseswere correct or not.

Category 1 summary

The empiric studies identified were positive, but gener-ally used trainees and specific content limiting the abilityto generalise the impact of results to actual practice.

2) Improve clinical reasoningAccording to the currently popular paradigm, diagnosesare made by some interacting combination of intuitive,automatic processing (system 1) and deliberate, rationalconsideration (system 2).152 Interventions to reducediagnostic error have been suggested in each of theseareas, and many authors have advocated for the benefitsof general training in clinical reasoning.1 23 47 49 53 75e78

The interventions in this domain are presented intable 2.

Improve intuitive processing: debiasing

Many, and perhaps most, medical diagnoses are derivedintuitively, acknowledging that most conditions arecommon and present in typical, easily recognised,fashion. Coderre et al153 found that intuitive diagnosesare more likely to be correct compared with diagnosesderived by hypothetico-deductive reasoning, and thisconcept is also consistent with the substantial literatureregarding expertise.

Experts in the field of naturalistic decision-makingemphasise that intuitive judgements cannot be taughtbecause they emerge subconsciously from the amassedexperience of the decision-maker and his or her abilityto access this knowledge instantaneously and effec-tively.48 50 However, others have argued that intuition can

be encouraged, strengthened and improved.51 101 151

Brawn highlighted several strategies to encourage use ofintuition such as showcasing examples of how intuitionwas used in discovery and insight situations.98 Noddingsand Shore100 suggest that intuition can be developed byfirst acknowledging intuition and its role in decision-making, demonstrating its capacity and successes, and bysharing how intuition is used, especially by experiencedrole models. Hogarth151 recommends a series of noveleducational interventions to teach and improve intui-tion, including creating increased motivation to learn byexposure to one’s own errors and constantly seeking toimprove one’s learning skills by reviewing and revisingskills in observation, sense-making and hypothesistesting.Croskerry and others have argued that clinicians

would make fewer errors if they learned the potentialshortcomings (biases) of intuitive decision-making so asto understand and avoid them.76 77 91 Interventions toavoid both affective bias (engendered by our inherentdiscomfort with certain types of patients or interactions)and cognitive bias (due to the known shortcomingsand pitfalls of subconscious thought) have beensuggested.Similar debiasing interventions were suggested by

Fischhoff85 and included: (1) warning about the possi-bility of bias; (2) describing how the bias distorts gooddecisions; (3) letting the individual make a bias-relatedjudgement error and giving them feedback; and (4)repeating these cycles with extended coaching. Larrick87

reported an example of successful debiasing by keepingit focused on a particular context and a particular bias.Experimental evidence suggests that hindsight bias

can be reduced by considering alternatives.154 In onesuch study, subjects were asked to choose between twoanswers to a difficult question,93 where some were askedto give the reasons they made their choice and otherswere asked to give reasons both for and against theirchoice. Considering both alternatives improved accuracyand reduced the tendency for subjects to be over-confident in their answers.92 Similarly, physicians evalu-ating a difficult test case were more likely to trusta diagnosis when asked to consider alternatives.154

Although debiasing is potentially attractive, severalauthors have expressed scepticism if this approach willwork based on the intrinsic difficulty of changing thesubconscious processing individuals use in decision-making.86 155 156


Narrative review




Interventions

Our search yielded two studies. Sherbino andcolleagues19 tested an effort to improve clinicalreasoning of trainees by teaching them cognitiveforcing strategies to counteract biases. The studylacked baseline data (no measure prior to interven-tion) or a control group, and the results were generallynegative. In addition, the reported retention of thecognitive forcing strategies that were the subject of theintervention was short-lived. Eva et al encouraged theuse of combined strategies (pattern recognition plusdeliberate consideration) in teaching students to readelectrocardiograms (ECR), and found this improvedtheir diagnostic performance in part by avoidingbiases.20

Improving metacognition and reflection

Improving metacognition, the ability to reflect on one’sown thought processes, is an appealing approach toreduce cognitive error.77 78 103 Metacognition couldpotentially alert clinicians to possible flaws in theirreasoning and help detect errors. A related andwidely endorsed recommendation is to practicereflectively,82 94 96 97 102 recently referred to as thediagnostic ‘time out’.83 Reflective practice promotesmetacognition and incorporates four distinct elements:Seeking out alternative explanations, exploring theconsequences of alternative diagnoses, being open totests that would differentiate the various possibilities andaccepting uncertainty. This process, essentially gettinga second opinion from your own conscious mind, hasthe potential to avoid many of the inherent pitfalls ofheuristic thought.82

Several tools have been suggested that might behelpful to promote metacognition and reflective prac-tice, including Trowbridge’s ‘12 Tips’ and Leonidas’‘Ten Commandments’.83 157 Using a diagnosis checklist,by promoting conscious review and reflection, has alsobeen advocated as a way to avoid pitfalls in clinicalreasoning.106 157

Interventions

Two studies were identified. Mamede and colleaguesfound that conscious reflection decreased thetendency towards availability bias,21 and Coderre et al

demonstrated that reflection on an initial diagnosiswas helpful if the initial diagnosis was wrong, and didnot lead to new errors if the initial diagnosis wascorrect.22 A limitation of both studies is that theadditional time spent on problem solving may be whatis driving the result, not conscious reflection per se.Also, both studies involved trainees in a laboratoryenvironment, so that the positive results would have tobe reconfirmed in practice settings. It is therefore

inconclusive whether these techniques successfullyreduce diagnostic errors.

Consider alternatives

A central element of reflective practice is reviewingalternative diagnoses, an approach widely endorsed asa valid approach to improved decision-making107 whichwe consider separately in this section. In this approach,clinicians should invoke what has been called theuniversal antidote, ‘Could this be something else?’ anduse appropriate tests to exclude the alternatives, ratherthan ordering tests that simply confirm original suspi-cions.107 Others79 have also suggested that clinicians ‘jotdown, in advance, outcomes that would support one’sinitial conclusions and also those that would disconfirmthem’ or consider alternatives.111 A related strategy is toassume the perspective of an outside observer,86

prompting evaluation of the decision-making strategythat was used and whether or not it was flawed. Militaryplanners have used ‘prospective hindsight’ to teach thisprinciple: one looks into the future to see that theworking diagnosis is not correct: What was missed andwhat else should have been considered?97 112

Interventions

Our search yielded one study that tested an interven-tion in this category. Wolpaw et al23 attempted toimprove clinical reasoning and decision-making skillsthrough a six-step training programme for medicalstudents to express their diagnostic reasoning process.The impact of this technique on diagnostic errors isinconclusive since the study did not assess the reduc-tion of diagnostic errors, but only assessed frequencyand thoroughness of their skills in presenting a patientcase. The study only measured the presence/amountof reasoning and not accuracy thereof, and so it isunclear how this intervention would improve diag-nostic accuracy.

Improve rational processing

Rational, deliberate review and consideration combinethe use of evidence-based knowledge158 159 with twonormative approaches, the use of expected value deci-sion-making160 to choose among a group of possiblediagnoses and Bayesian analysis to incorporate testresults in considering a single diagnosis. Kassirer et al47

describes the process of clinical reasoning as generatinginitial hypothesis which are then investigated by diag-nostic tests and Bayesian analysis until an appropriatethreshold (Treat, Don’t Treat) is reached. Kassirersuggests that the essential skills of clinical reasoning canand should be taught to medical students from their firstdays,126 and reviewers have concluded that consciousreview can be taught effectively.121 122 Trainees taught


Narrative review




principles of evidence-based medicine are more likely touse Bayesian techniques to interpret clinical findings.158

In efforts to reduce surgical cognitive errors, Brannick et

al108 oriented surgical trainees to Reason’s major errortypes using an educational video and role-playingemphasising errors. Although actual surgical error rateswere the same as in untrained controls after a month,attention to detail improved.108

Category 2 summary

We noted a major discrepancy between the breadth andenthusiasm for these interventions in the backgroundstudies, but a paucity of actual interventions. For all threecategories, there is very limited evidence addressingdiagnostic accuracy or errors. The studies identifiedinvolved trainees in laboratory-like settings, limiting theability to generalise the findings to real practice.

3) Get help: use other people and decision support toolsGiven the constraints of human cognition,78 physiciansmay be able to augment their innate cognitive abilities byobtaining advice and help from others. All of the testedinterventions in this category are detailed together inonline appendix C and were organised in the followingcategories.

Second opinions

Interventions

Several studies have demonstrated that second reviewsof surgical pathology or cytology specimens finda small but important group of errors,24e28 anda growing number of healthcare systems now requiresecond readings in case types known to have substan-tial rates of inter-observer variability. Most of thesestudies do not, however, include data on patientoutcomes (table 3).Second readings in radiology also improve test sensi-

tivity. Duijm et al31 found that multiple independentreaders (radiologists or technicians) increased cancerdetection rates with only a slight decrease in specificity,and Kwek et al32 found that second reading increasedcancer detection by 5%.The impact of second readings has been mixed in

other settings. Second reading of Emergency Room(ER) imaging studies was helpful in one study,30 but inanother, besides identifying previously missed abnor-malities, the second reading introduced new misinter-pretations leading to inappropriate changes inmanagement.29 Canon et al33 measured the impact ofindependent double reading of barium enemas andfound no effect on the sensitivity of polyp detection andan increased rate of false positives.Thus, the overall impact of ‘second opinions’ on

diagnostic errors appears to be mixed. Sensitivity

appears to improve in most but not all studies, but thesecond readings tend to introduce new errors thatdetract from the specificity of the diagnostic test. Resultscould potentially be both reliable and generalisablebecause of the relatively large number of cases reviewedin these studies, and the use of expert reviewers.Costebenefit analyses will be needed to determinewhether the costs of second readings and the seeminglyinevitable increment in false positives are offset by theincreased rate of case finding.

Groups and librarians

Groups can make better decisions than its individualmembers if the members are allowed to function inde-pendently.86 161 Diagnosing challenging cases withinteams or with peers would take advantage of this strategy.A recent novel approach leverages the use of librarianswho are experienced and skilled in identifying infor-mation, evidence, and knowledge relevant to diagnosticalternatives or testing strategies.129e131

Interventions

One study by Christensen et al34 studied team-baseddecisions. This was a well designed, controlled study,but the results were negative: performance did notimprove by using the team.A randomised trial of embedded clinical informa-

ticians at one university demonstrated a positive impacton the clinical care provided,35 although self-reportedperceptions were used in place of actual outcomes.

Decision support

Most studies of decision support tools have evaluatedimpact on process measures, user satisfaction and utilityin a limited sense,141 and are not consistently positive.A systematic review of decision support systems in 1998identified only a single study focusing on diagnosis,133

and in this study, using a decision support tool in anemergency room on patients with joint or bone injuriesactually led to more missed fractures.134

Using linear prediction models (actuarial decision-making, algorithms) has been shown to yield better‘decisions’ than most decision-makers, including experts,in a wide range of settings.111 Wedding and colleagues79132 162 report that actuarial diagnosis was more accuratethan clinical judgement in patients with neuropsychiatricconditions. However, clinicians tend to disregard advicefrom these tools or not use them even when they arereadily available.163 164 The importance of embeddingdecision support in the physician’s workflow has beenrepeatedly emphasised, for example, by incorporatingdecision support logic in computer-based order entrysystems. A systematic review of this approach identified11 controlled trials, seven of which reported improvedprofessional practice141 on ordering diagnostic tests.


Narrative review




Hamm and Zubialde and more recently Schiff andBates have called attention to many other ways in whichthe electronic medical record can enhance clinicalreasoning.102 142 Besides providing clear access to thenecessary data, good records help clinicians organisetheir thoughts, enhance collaborative thinking, enhanceefficiency and promote feedback. Another promisingtype of clinical decision support enabled by electronicrecords is the graphic display of timeline data to assist inthe interpretation of diagnostic test results and to helpdetect subtle trends.137

Interventions

A recent review identified 10 newer studies, eachfocused on a specific clinical condition135 and of theseonly four studies had positive results: one reportedimproved ability to detect and diagnose mood disor-ders in outpatients,136 two improved diagnosis in acutecoronary syndromes36 37 and one evaluated diagnosisof acute abdominal conditions on a surgical service,which improved provider performance but not patientoutcomes38 (table 4). Finally, De Simone et al39

describe a system that receives clinical informationfrom the patient directly and synthesises that infor-mation to aid the clinician in diagnosing the cause ofheadaches. Overall, the studies were all sound andresults seem to be generalisable by virtue of testinga range of subjects and case types.Another approach to supporting the diagnosis of

specific conditions in general is the Infobutton func-tionality, described by Cimino.40 The only available studyof Infobuttons included subjects from varying levels(attending physicians, residents, medical students,nurses) who were mostly satisfied with the tool. Theimpact of the tool on diagnostic accuracy and patientoutcomes was not assessed.

Computer-aided detection systems

Interventions

Five studies have examined the use of computer-aideddetection systems to aid radiologic diagnosis. Peld-schus et al165 studied the effectiveness of an automatedcomputer-aided detection system for chest CT studiesand found both new positives and false positives.Berbaum et al166 found that use of a computer-aideddetection system in chest radiography could notcounteract the satisfaction-of-search effect (being ableto find additional defects beyond the first one) in 16subjects. In another study, Kakeda et al demonstrateda significant beneficial effect of using computer-aideddiagnosis support to help analyse chest radiographs.167

In mammography, Jiang et al168 found that computer-aided diagnosis reduced inter-observer variability, but inanother study computer-assisted mammography inter-pretation had no beneficial effects on cancer detection

and significantly increased the false positive rate of thestudies and the biopsy rate.169 A recent commentary oncomputer-assisted detection noted that while use of thistechnology is increasingly the norm, the jury is still outon its utility.170 All of the intervention studies reviewedwere solid in design and in the interpretation of theresults and conclusions, but the ability to generalise islimited due to studies in just two domains (chest x-raysand mammograms).

Computer-aided interpretation systems

Interventions

Two studies focused on technology to improve ECGinterpretation. Daudelin and Selker171 reported usingan ECG-based acute cardiac ischaemia predictiveinstrument to improve triage decision-making in theER. Olsson et al172 studied the use of an artificial neuralnetwork trained to automatically detect ECGs indi-cating possible transmural ischaemia and found thatthis decision support tool was effective in improvinginexperienced interns’ interpretation of ECGs.

General decision support tools for medical diagnosis

Computer aided decision support tools have also beendeveloped to assist specifically with differential diagnosis.Anecdotally these tools succeed, in a small fraction ofsearches, in suggesting a difficult or obscure diagnosisthat was previously missed. The clinician inputs thepatient’s key findings, and these programmes suggestpossible diagnoses. Some programmes help refine thesechoices by further suggestions of questions to ask, find-ings to look for or tests to perform. Berner et al evaluatedthe first generation products (QMR, DXplain, Iliad andMeditel) using test scenarios and all the products wereeffective in providing useful suggestions.173 However, thecorrect diagnosis appeared on the suggestion list only halfto three-fourths of the time, and all of the programmesgenerated a large number of extraneous conditions.174

Some of these initial products are no longer available,although DXplain has been maintained and updated.

Interventions

Of the many newer web-based decision support tools,‘ISABEL’ has been the most extensively evaluated.Compared with first generation tools, ISABEL displaysmuch improved sensitivity in both paediatric settings4142 and in analysing adult case scenarios, in which thesensitivity approached 100%.43e45 ‘Google’ searchinghas also been evaluated inmedical settings, but suggeststhe correct diagnosis in only 58% of difficult cases.46

Category 3 summary

Overall, the technique of ‘getting help’ during thediagnostic process may be beneficial. The use of decisionsupport resources has been studied more extensively


Narrative review




than any other intervention, and these approaches, ifused, show promise in their potential to reduce diag-nostic errors. More research is needed regarding the useof second reviews, teams and librarians.

DISCUSSION

Reducing harm from diagnostic errors requires inter-ventions to improve the cognitive processes thatunderlie clinical reasoning. We identified a reasonablylarge literature on potential interventions and organisedthese interventions into three categories: (1) Increasingknowledge and expertise, (2) Improving intuitive anddeliberate consideration and (3) Getting help fromcolleagues, consultants and tools.We found that most interventions in the literature

were simply ideas or suggestions. Many of these are wellconceptualised and widely endorsed, and seem ripe tobe tested in experimental or real-world clinical settings.A major finding in each of the three categories wasa large discrepancy between the broad and enthusiasticrecommendations for the various interventions, buta relative paucity of actual trials. Of the few studies thatreported true interventions, few included robust designsor metrics. Typically, the interventions involved anobservational study design and measured outcomesbefore and after an intervention with a small number oftrainees or clinicians and/or healthcare sites, withouta control group.Our findings also affirm that the science of out-

come measurement in this area is underdeveloped.Educational interventions in particular are difficult toevaluate in terms of changing attitudes and behavioursin practice. One major issue is the difficulty of demon-strating that diagnosis can be improved by any approachin real-world settings. Definitions of diagnostic error arenot standardised and error designations are typicallysubjective judgements, often confounded by hindsightbias. Measurement instruments and methods to evaluatecognitive intervention effects are not well developed.Additionally, because diagnostic error reflects theinterplay of system-related and patient-dependentfactors, the true effect of a purely cognitive interventionmight be difficult to ascertain. All of these factors posechallenges in the design of future interventions in thisarea.The major limitation of this review is the likelihood

that we overlooked conceptual ideas to improve deci-sion-making from both medical and non-medical fields.Medical diagnosis is essentially a special case of decision-making under conditions of uncertainty, and ideas forimproving these decisions can arise from almost anydiscipline, including the social sciences, business fieldsand military scholars.

A clear challenge going forward is to identify theadvances in these areas that might be applicable toimproving the reliability of medical diagnosis. Despitethe many shortcomings of these studies, our reviewidentified promising ideas for reducing diagnostic errorin each of the three major categories.

Increase knowledgeAt the present time, disease-specific training is the onlyintervention that is both supported by evidence andseems implementable. In the future, simulation offerspotential both in terms of teaching clinicians aboutdiagnostic error and error-prevention strategies, as wellas serving as a method to rapidly build expertisethrough exposure to many types of disease variants.Feedback also offers the potential to reduce errors byhelping develop expertise. Feedback is also the key toreducing overconfidence, which in turn could open thedoor for clinicians to appreciate the possibility of theirown errors and take actions to avoid them. Deliberatefeedback is embedded in many approaches that seek toimprove individual and team performance outside ofmedicine.

Improve clinical reasoningAlthough some of the interventions to improvereasoning have been successful with trainees, most haveyet to be implemented or evaluated in practice. Reflec-tive practice and active metacognitive review may havegreat potential to reduce diagnostic error, and the toolsto promote these practices need to be further developedand evaluated in practice. These approaches expand thenumber of conditions to be considered and effectivelyaddress many of the major causes of cognitive error,including context errors, framing bias and prematureclosure. However, the cost of trade-offs is not clear. Forexample, will the broadened consideration of alternativediagnoses lead to inappropriate or costly testing,divert attention away from the correct diagnosis or bedeleterious in another way?

Get helpDecision support for diagnosis has the unique advantagethat it can be implemented at the system level, withoutrequiring some new skill or behaviour to be learnt byclinicians. Still, clinicians need to be willing to takeadvantage of these resources, and error reduction willcritically hinge on how well the support functionality isincorporated into everyday workflow and how clinicianswill deal with the specificity problem. Using informa-ticians, working more effectively in groups, taking fulladvantage of the comprehensive electronic healthrecord and relying more on actuarial tools (algorithms)may be effective strategies. Second opinions and


Narrative review




consultations bring fresh eyes to examine a case,a powerful and effective way to find and correctdiagnostic errors.

CONCLUSIONS

In conclusion, there is a surprisingly wide range ofpossible approaches to reducing the cognitive contri-butions to diagnostic error. Not all the suggestions havebeen tested, and of those that have, the evaluationstypically involved trainees in artificial settings, making itdifficult to extrapolate the results to actual practice.The field is immature and progress in reducing diag-

nostic error will require considerable research to eval-uate the relative merits of these different ideas,refinements in the methodology of defining andmeasuring outcomes in preventing diagnostic error andharm, and leveraging advances in other aspects ofmedical decision-making and cognitive sciences that maymake medical diagnosis more reliable.

Author affiliations1VA Medical Center, Northport, New York, USA2Department of Medicine, SUNY Stony Brook, New York, USA3RTI International, Research Triangle Park, North Carolina, USA4School of Biomedical Informatics, University of Texas Health Science Centerin Houston, Houson, Texas, USA5National Center for Cognitive Informatics and Decision Making in Healthcare,University of Texas Health Science Center in Houston, Houston, Texas, USA6Houston VA HSR&D Center of Excellence, and the Center of Inquiry toImprove Outpatient Safety Through Effective Electronic Communication,Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas, USA7Section of Health Services Research, Department of Medicine, Baylor Collegeof Medicine, Houston, Texas, USA8Center for Quality Improvement and Patient Safety, Agency for HealthcareResearch and Quality, Rockville, Maryland, USA

Acknowledgements We gratefully acknowledge administrative and literatureresearch assistance from Ms Grace Garey, Mary Lou Glazer, Diane Martin1and Wendy Isser.

Contributors All listed authors contributed to the conception and design, oranalysis and interpretation of data; drafting the article or revising it critically forimportant intellectual content; and final approval of the version to bepublished. MLG is the guarantor, contributed to the study design, contributedto data synthesis, wrote the paper and coordinated revisions. SK performedthe literature search, contributed to data analysis and synthesis, and writingand revising the manuscript. VLP performed the analytical review andcontributed to the manuscript revisions. ANDM performed the analyticalreview and contributed to the manuscript revisions. AVS contributed to thestudy design, the literature search, data analysis and synthesis, and reviewingand editing the manuscript. NL performed the literature search and contributedto data synthesis and analysis. ET performed the literature search andcontributed to the synthesis and manuscript revisions. KH contributed to thestudy design, reviewed the literature search, and reviewed and edited themanuscript; KAL contributed to the study design, reviewed the literaturesearch, and reviewed and edited the manuscript. HS contributed to the studydesign, data analysis and synthesis, writing and revising the manuscript.

Funding This study was funded by the Agency for Healthcare Research andQuality (AHRQ) ACTION II Task Order #8, Contract No. HHSA290200600001and in part by the Houston VA HSR&D Center of Excellence (HFP90-020).

Competing interests None.

Provenance and peer review Not commissioned; externally peer reviewed.

REFERENCES1. Berner E, Graber M. Overconfidence as a cause of diagnostic error

in medicine. Am J Med 2008;121:S2e23.2. Kostopoulou O, Delaney BC, Munro CW. Diagnostic difficulty and

error in primary care - a systematic review. Fam Pract2008;25:400e13.

3. Kostopoulou O. Do GPs report diagnostic errors? Fam Pract2008;25:1e2.

4. Graber ML, Franklin N, Gordon R. Diagnostic error in internalmedicine. Arch Intern Med 2005;165:1493e9.

5. Schiff GD, Hasan O, Kim S, et al. Diagnostic error in medicine:analysis of 583 physician-reported errors. Arch Intern Med2009;169:1881e7.

6. Zwaan L, de Bruijne M, Wagner C, et al. Patient record review of theincidence, consequences, and causes of diagnostic adverse events.Arch Intern Med 2010;170:1015e21.

7. Kachalia A, Gandhi TK, Puopolo AL, et al. Missed and delayeddiagnoses in the emergency department: a study of closedmalpractice claims from four liability insurers. Ann Emerg Med2007;49:196e205.

8. Singh H, Graber ML, Kissam SM, et al. System-related interventionsto reduce diagnostic errors: a narrative review. BMJ Qual Saf2012;21:160e70.

9. Gordon M, Findley R. Educational interventions to improve handoverin health care: a systematic review. Med Educ 2011;45:1081e90.

10. Barr H, Freeth D, Hammick M, et al. Evaluations of InterprofessionalEducation: A United Kingdom Review of Health and Social Care.http://www.caipe.org.uk/silo/files/evaluations-of-interprofessional-education.pdf

11. BEME. Best Evidence in Medical Education. BEME Collaboration,2003. http://www.bemecollaboration.org

12. Fridriksson S, Hillman J, Landtblom AM, et al. Education of referringdoctors about sudden onset headache in subarachnoid hemorrhage:a prospective study. Acta Neurol Scand 2001;103:238e42.

13. Rezvyy G, Parniakov A, Fedulova E, et al. Correcting biases inpsychiatric diagnostic practice in Northwest Russia: comparing theimpact of a general educational program and a specific diagnostictraining program. BMC Med Educ 2008;8:15.

14. Gutmark R, Halsted MJ, Perry L, et al. Use of computer databases toreduce radiograph reading errors. J Am Coll Radiol 2007;4:65e8.

15. Bond WF, Deitrick LM, Arnold DC, et al. Using simulation to instructemergency medicine residents in cognitive forcing strategies. AcadMed 2004;79:438e46.

16. Carlson J, Abe M, Bridges D, et al. The impact of a diagnosticreminder system on student clinical reasoning during simulated casestudies. Simul Healthc 2011;6:11e18.

17. Tudor GR, Finlay DB. Error review: can this improve reportingperformance? Clin Radiol 2001;56:751e4.

18. Wood DS, Tracey TJG. A brief feedback intervention for diagnosticovershadowing. Train Educ Prof Psychol 2009;3:218e25.

19. Sherbino J, Dore K, Siu E, et al. The effectiveness of cognitiveforcing strategies to decrease diagnostic errordan exploratorystudy. Teach Learn Med 2011;23:78e85.

20. Eva KW, Hatala RM, LeBlanc VR, et al. Teaching from the clinicalreasoning literature: combined reasoning strategies help novicediagnosticians overcome misleading information. Med Educ2007;41:1152e8.

21. Mamede S, van Gog T, van den Berge K, et al. Effect of availabilitybias and reflective reasoning on diagnostic accuracy among internalmedicine residents. JAMA 2010;304:1198e203.

22. Coderre S, Wright B, McLaughlin K. To think is good: querying aninitial hypothesis reduces diagnostic error in medical students. AcadMed 2010;85:1125e9.

23. Wolpaw T, Papp KK, Bordage G. Using SNAPPS to facilitate theexpression of clinical reasoning and uncertainties: a randomizedcomparison group trial. Acad Med 2009;84:517e24.

24. Raab SS, Grzybicki DM, Mahood LK, et al. Effectiveness of randomand focused review in detecting surgical pathology error. Am J ClinPathol 2008;130:905e13.

25. Raab SS, Stone CH, Jensen CS, et al. Double slide viewing asa cytology quality improvement initiative. Am J Clin Pathol2006;125:526e3.

26. Manion E, Cohen MB, Weydert J. Mandatory second opinion insurgical pathology referral material: clinical consequences of majordisagreements. Am J Surg Pathol 2008;32:732e7.

27. Nordrum I, Johansen M, Amin A, et al. Diagnostic accuracy ofsecond-opinion diagnoses based on still images. Hum Pathol2004;35:129e35.

28. Hamady ZZ, Mather N, Lansdown MR, et al. Surgical pathologicalsecond opinion in thyroid malignancy: impact on patients’management and prognosis. Eur J Surg Oncol 2005;31:74e7.


Narrative review




29. Benger JR, Lyburn ID. What is the effect of reporting all emergencydepartment radiographs? Emerg Med J 2003;20:40e3.

30. Espinosa JA, Nolan TW. Reducing errors made by emergencyphysicians in interpreting radiographs: longitudinal study. BMJ2000;320:737e40.

31. Duijm LE, Groenewoud JH, Fracheboud J, et al. Additional doublereading of screening mammograms by radiologic technologists:impact on screening performance parameters. J Natl Cancer Inst2007;99:1162e70.

32. Kwek BH, Lau TN, Ng FC, et al. Non-consensual double reading inthe Singapore breast screening project: benefits and limitations. AnnAcad Med Singapore 2003;32:438e41.

33. Canon CL, Smith JK, Morgan DE, et al. Double reading ofbarium enemas: is it necessary? AJR Am J Roentgenol2003;181:1607e10.

34. Christensen C, Larson JR Jr, Abbott A, et al. Decision making ofclinical teams: communication patterns and diagnostic error. MedDecis Making 2000;20:45e50.

35. Mulvaney SA, Bickman L, Giuse NB, et al. A randomizedeffectiveness trial of a clinical informatics consult service: impact onevidence-based decision-making and knowledge implementation.J Am Med Inform Assoc 2008;15:203e11.

36. Pozen MW, D’Agostino RB, Selker HP, et al. A predictive instrumentto improve coronary-care-unit admission practices in acute ischemicheart disease. N Engl J Med 1984;310:1273e8.

37. Selker HP, Beshansky JR, Griffith JL, et al. Use of the acute cardiacischemia time-insensitive predictive instrument (ACI-TIPI) to assistwith triage of patients with chest pain or other symptoms suggestiveof acute cardiac ischemia. A multicenter, controlled clinical trial. AnnIntern Med 1998;129:845e55.

38. Bogusevicius A, Maleckas A, Pundzius J, et al. Prospectiverandomised trial of computer-aided diagnosis and contrast radiographyin acute small bowel obstruction. Eur J Surg 2002;168:78e83.

39. De Simone R, Coppola G, Ranieri A, et al. Validation of AIDACefalee, a computer-assisted diagnosis database for themanagement of headache patients. Neurol Sci 2007;28(Suppl 2):S213e16.

40. Cimino JJ. Use, usability, usefulness, and impact of an infobuttonmanager. AMIA Annu Symp Proc 2006:151e5.

41. Ramnarayan P, Winrow A, Coren M, et al. Diagnostic omissionerrors in acute paediatric practice: impact of a reminder system ondecision-making. BMC Med Inform Decis Mak 2006;6:37.

42. Ramnarayan P, Cronje N, Brown R, et al. Validation of a diagnosticreminder system in emergency medicine: a multi-centre study.Emerg Med J 2007;24:619e24.

43. Ramnarayan P, Kapoor RR, Coren J, et al. Measuring the impact ofdiagnostic decision support on the quality of clinical decision making:development of a reliable and valid composite score. J Am MedInform Assoc 2003;10:563e72.

44. Ramnarayan P, Roberts GC, Coren M, et al. Assessment of thepotential impact of a reminder system on the reduction of diagnosticerrors: a quasi-experimental study. BMC Med Inform Decis Mak2006;6:22.

45. Graber ML, Mathew A. Performance of a web-based clinicaldiagnosis support system for internists. J Gen Intern Med 2008;23(Suppl 1):37e40.

46. Tang H, Ng JH. Googling for a diagnosiseuse ofGoogle as a diagnostic aid: internet based study. BMJ2006;333:1143e5.

47. Kassirer JP, Wong J, Kopelman R. Learning Clinical Reasoning. 2ndedn. Baltimore, MD: Lippincott Williams and Wilkins, 2010.

48. Gigerenzer G. Adaptive Thinking: Rationality in the Real World.New York: Oxford University Press, 2000.

49. Elstein AS. Thinking about diagnostic thinking: a 30-yearperspective. Adv Health Sci Educ Theory Pract 2009;14(Suppl 1):7e18.

50. Klein G. Sources of Power: How People Make Decisions.Cambridge, MA: The MIT Press, 1998.

51. Klein G. The Power of Intuition. New York, NY: Doubleday, 2003.52. Bazerman MH, Moore DA. Improving Decision Making. Judgment

in Managerial Decision Making. John Wiley and Sons,2009:179e99.

53. Norman G. Dual processing and diagnostic errors. Adv Health SciEduc Theory Pract 2009;14( Suppl 1):37e49.

54. Smith-Bindman R, Chu PW, Miglioretti DL, et al. Comparison ofscreening mammography in the United States and the Unitedkingdom. JAMA 2003;290:2129e37.

55. Singh H, Sethi S, Raber M, et al. Errors in cancer diagnosis:current understanding and future directions. J Clin Oncol2007;25:5009e18.

56. Bowen JL. Educational strategies to promote clinical diagnosticreasoning. N Engl J Med 2006;355:2217e25.

57. Davis D, O’Brien MA, Freemantle N, et al. Impact of formalcontinuing medical education: do conferences, workshops, rounds,and other traditional continuing education activities changephysician behavior or health care outcomes? JAMA1999;282:867e74.

58. Freidlander ML, Phillips SD. Preventing anchoring errors in clinicaljudgment. J Consult Clin Psychol 1984;52:366e71.

59. Gentner D, Lowenstein G, Thompson L. Learning and transfer:a general role for analogic encoding. J Educ Psychol2003;95:393e408.

60. Hershberger PJ, Markert RJ, Part HM, et al. Understanding andaddressing cognitive bias in medical education. Adv Health Sci Educ1996;1:221e6.

61. Parmley MC. Confirmation Bias on Diagnostic Decision Making.Drexel University, 2006.

62. Hogarth R. Educating Intuition. University of Chicago Press, 2001.63. Bond W, Kuhn G, Binstadt E, et al. The use of simulation in

the development of individual cognitive expertise in emergencymedicine. Acad Emerg Med 2008;15:1037e45.

64. Schiff GD. Minimizing diagnostic error: the importance of follow-upand feedback. Am J Med 2008;121:S38e42.

65. Jamtvedt G, Young JM, Kristoffersen DT, et al. Does telling peoplewhat they have been doing change what they do? A systematicreview of the effects of audit and feedback. Qual Saf Health Care2006;15:433e6.

66. Papa FJ, Aldrich D, Schumacker RE. The effects of immediateonline feedback upon diagnostic performance. Acad Med 1999;74:S16e18.

67. Stone ER, Opel RB. Training to improve calibration anddiscrimination: the effects of performance andenvironment feedback. Organ Behav Hum Decis Process2000;83:282e309.

68. Alpert HR, Hillman BJ. Quality and variability in diagnostic radiology.J Am Coll Radiol 2004;1:127e32.

69. Arkes H. Impediments to accurate clinical judgment and possibleways to minimize their impact. J Consult Clin Psychol1981;49:323e30.

70. Humble JE, Keim RT, Hershauer JC. Information systems design:an empirical study of feedback effects. Behav Inf Technol1992;11:237e44.

71. Pulford BD, Colman AM. Overconfidence: feedback and itemdifficulty effects. Pers Individ Differ 1997;23:125e33.

72. Subbotin V. Outcome feedback effects on under- andoverconfidence judgments (general knowledge tasks). Organ BehavHum Decis Process 1996;66:268e76.

73. Fischer MA, Mazor KM, Baril J, et al. Learning from mistakes.Factors that influence how students and residents learn frommedical errors. J Gen Intern Med 2006;21:419e23.

74. Eva KW. Diagnostic error in medical education: where wrongs canmake rights. Adv Health Sci Educ Theory Pract 2009;14(Suppl 1):71e81.

75. Eva KW. What every teacher needs to know about clinicalreasoning. Med Educ 2005;39:98e106.

76. Croskerry P. Cognitive forcing strategies in clinical decision making.Ann Emerg Med 2003;41:110e20.

77. Croskerry P. The importance of cognitive errors in diagnosis andstrategies to minimize them. Acad Med 2003;78:775e80.

78. Croskerry P. Diagnostic Failure: A Cognitive and Affective Approachin Advances in Patient Safety: From Research to Implementation.Rockville, MD: Agency for Healthcare Research and Quality(Publication No. 050021), 2005;2:241e54.

79. Wedding D, Faust D. Clinical judgment and decision making inneuropsychology. Arch Clin Neuropsychol 1989;4:233e65.

80. Groves M, O’Rourke P, Alexander H. Clinical reasoning: the relativecontribution of identification, interpretation and hypothesis errors tomisdiagnosis. Med Teach 2003;25:621e5.

81. Pines JM. Profiles in patient safety: confirmation bias in emergencymedicine. Acad Emerg Med 2006;13:90e4.

82. Moulton CE, Regehr G, Mylopoulos M, et al. Slowing down whenyou should: a new model of expert judgement. Acad Med 2007;82:S109e16.

83. Trowbridge RL. Twelve tips for teaching avoidance of diagnosticerrors. Med Teach 2008;30:496e500.

84. Kuhn GJ. Diagnostic errors. Acad Emerg Med 2002;9:740e50.85. Fischhoff B. Debiasing. In: Kahneman D, Slovic P, Tversky A, eds.

Judgment Under Uncertainty. New York: Cambridge UniversityPress, 1982:422e44.

86. Milkman KL, Chugh D, Bazerman MH. How can decision making beimproved? Perspect Psychol Sci 2009;4:379e83.

87. Larrick RP. Debiasing. In: Koehler DJ, Harvery N, eds. BlackwellHandbook of Judgment and Decision Making. Oxford, UK:Blackwell, 2004.


Narrative review




88. Koriat A, Bjork RA, Sheffer L, et al. Predicting one’s own forgetting:the role of experience-based and theory-based processes. J ExpPsychol Gen 2004;133:643e56.

89. Renner C, Renner M. But I thought I knew that: using confidenceestimation as a debiasing technique to improve Classroomperformance. Appl Cogn Psychol 2001;15:23e32.

90. Estrada CA, Isen AM, Young MJ. Positive affect facilitatesintegration of information and decreases anchoring in reasoningamong physicians. Organ Behav Hum Decis Process1997;72:117e35.

91. Scott IA. Errors in clinical reasoning: causes and remedialstrategies. BMJ 2009;338:b1860.

92. Slovic P, Fischoff B. On the psychology of experimental surprises.J Exp Psychol Hum Percept Perform 1977;3:544e51.

93. Lichtenstein S, Fischoff B, Bishop B. Calibration of probabilities: Thestate of the art to 1980. In: Kahneman D, Slovic P, Tversky A, eds.Judgment Under Uncertainty: Heuristics and Biases. New York:Cambridge University Press, 1982.

94. Schon DA. Educating the Reflective Practioner. San Francisco:Jossey-Bass, 1987.

95. Greenhalgh T. Intuition and evidenceeuneasy bedfellows? Br J GenPract 2002;52:395e400.

96. Mamede S, Schmidt HG, Rikers R. Diagnostic errors and reflectivepractice in medicine. J Eval Clin Pract 2007;13:138e45.

97. Singh H, Petersen LA, Thomas EJ. Understanding diagnostic errorsin medicine: a lesson from aviation. Qual Saf Health Care2006;15:159e64.

98. Brawn R. The formal and the intuitive in science and medicine. In:Atkinson T, Claxton G, eds. The Intuitive Practitioner. BuckinghamUK: Open University Press, 2000:149e64.

99. Gregory G. Developing intuition through management education. In:Atkinson T, Claxton G, eds. The Intuitive Practitioner. Buckingham,UK: Open University Press, 2000.

100. Noddings S, Shore PJ. Awakening the Inner Eye: Intuition inEducation. New York: Teachers College Press, 1984.

101. Quirk M. Intuition and Metacognition in Medical Education. NewYork, NY: Springer Publishing Co., 2006.

102. Hamm RM, Zubialde J. Physicians’ expert cognition and the problemof cognitive bias. Med Decis Making 1995;22:181e213.

103. Croskerry P. The cognitive imperative: thinking about how we think.Acad Emerg Med 2000;7:1223e31.

104. Leonidas L. Ten commandments to reduce cognitive errors. In:Graber, ML. Educational interventions to reduce diagnostic error.Adv Health Sci Educ 2009;14:63e9.

105. Gawande A. The Checklist Manifesto: How to Get Things Right. 1stedn. New York: Metropolitan Books, 2010.

106. Ely JW, Graber M, Croskerry PC. Checklists to reduce diagnosticerrors. Acad Med 2011;86:7.

107. Taleb N. The Black Swan. New York: Random House, 2007.108. Brannick MT, Fabri PJ, Zayas-Castro J, et al. Evaluation of an error-

reduction training program for surgical residents. Acad Med2009;84:1809e14.

109. Arkes H. Why medical errors can’t be eliminated: uncertainties andthe hindsight bias. Chron High Educ 2000;46:591e600.

110. Schwenk CR. Cognitive simplification processes in strategicdecision-making. Strategic Manag J 1984;5:111e28.

111. Baron J. Thinking and Deciding. 3rd edn. New York, NY: CambridgeUniversity Press.

112. Mitchell DJ, Russo JE, Pennington N. Back to the future: temporalperspective in the explanation of events. J Behav Decis Making1989;2:25e38.

113. Mussweiler T, Strack F, Pfeiffer T. Overcoming the inevitableanchoring effect: considering the opposite compensates forselective accessibility. Pers Soc Psychol Bull 2000;26:1142e50.

114. Lord CG, Lepper MR, Preston E. Consider the opposite:a corrective strategy for social judgment. J Pers Soc Psychol1984;37:1231e43.

115. Arzy S, Brezis M, Khoury S, et al. Misleading one detail:a preventable mode of diagnostic error? J Eval Clin Pract2009;15:804e6.

116. Gorman M, Gorman M. A comparison of disconfirmatory,confirmatory and control strategies on Watson’s 2-4-6 task. Q J ExpPsychol 1984;36A:629e48.

117. Hirt E, Markman K. Multiple explanation: a consider-an-alternativestrategy for debiasing judgments. J Pers Soc Psychol1995;69:1069e86.

118. Mumma G, Steven W. Procedural debiasing of primary/anchoring effects in clinical-like judgments. J Clin Psychol1995;51:841e53.

119. Strauss SE, Glasziou P, Richardson WS, et al. Evidence-BasedMedicine, 4th edn. How to Practice and Teach. New York, NY:Churchill Livingstone, 2011.

120. Sox HC. Medical Decision Making. Butterworth-Heinman, 1988.121. Dobbie AE, Schneider FD, Anderson AD, et al. What evidence

supports teaching evidence-base medicine? Acad Med2000;75:1184e5.

122. Khan KS, Coomarasamy A. A hierarchy of effective teaching andlearning to acquire competence in evidenced-based medicine. BMCMed Educ 2006;15:59.

123. Croskerry P. Clinical cognition and diagnostic error: applications ofa dual process model of reasoning. Adv Health Sci Educ TheoryPract 2009;14(Suppl 1):27e35.

124. Ullman DG. Making Robust Decisions: Decision Management forTechnical, Business, and Service Teams. Victoria, BC: TraffordPublishing, 2006.

125. Pauker SC, Kassirer JP. The threshold approach to clinical decisionmaking. N Engl J Med 1980;302:1109e17.

126. Kassirer JP. Teaching clinical reasoning: case-based and coached.Acad Med 2010;85:1118e24.

127. Tatarka CJ. Overcoming biases in military problem analysis anddecision-making. Mil Intelligence 2002;28:8e10.

128. Raab SS. Improving patient safety by examining pathology errors.Clin Lab Med 2004;24:863.

129. Zipperer L, Sykes J. The role of librarians in patient safety: gapsand strengths in the current culture. J Med Libr Assoc2004;92:498e500.

130. Albert KM. Integrating knowledge-based resources into theelectronic health record: history, current status, and role oflibrarians. Med Ref Serv Q 2007;26:1e19.

131. Zipperer L, Sykes J. Engaging as partners in patient safety: theexperience of librarians. Patient Saf Qual Healthc. March/April2009;6:28e33,32e3.

132. Wedding D. Comparison of statistical and actuarial models forpredicting lateralization of brain-damage. Clin Neuropsychol1983;5:15e20.

133. Hunt DL, Haynes RB, Hanna SE, et al. Effects of computer-based clinical decision support systems on physician performanceand patient outcomes: a systematic review. JAMA1998;280:1339e46.

134. Klassen TP, Ropp LJ, Sutcliffe T, et al. A randomized controlled trialof radiograph ordering for extremity trauma in a pediatric emergencydepartment. Ann Emerg Med 1993;22:1524e9.

135. Garg AX, Adhikari NK, McDonald H, et al. Effects of computerizedclinical decision support systems on practitioner performance andpatient outcomes: a systematic review. J Am Med Assoc2005;293:1223e38.

136. Cannon DS, Allen SN. A comparison of the effects of computerand manual reminders on compliance with a mental healthclinical practice guideline. J Am Med Inform Assoc2000;7:196e203.

137. Radecki RP, Medow MA. Cognitive debiasing through sparklines inclinical data displays. AMIA Annu Symp Proc 2007:1085.

138. Reyna VF, Nelson WL, Han PK, et al. How numeracy influences riskcomprehension and medical decision making. Psychol Bull2009;135:943e73.

139. Bhandari G, Hassanein K, Deaves R. Debiasing investors withdecision support systems: an experimental investigation. DecisSupport Syst 2008;46:399e410.

140. Cook MB, Smallman HS. Human factors of the confirmation bias inintelligence analysis: decision support from graphical evidencelandscapes. Hum Factors 2008;50:745e54.

141. Kawamoto K, Lobach DF. Clinical decision support provided withinphysician order entry systems: a systematic review of featureseffective for changing clinician behavior. AMIA Annu Symp Proc2003:361e5.

142. Schiff G, Bates DW. Can electronic clinical documentation helpprevent diagnostic errors? N Engl J Med 2010;362:1066e9.

143. Norman G. Building on experienceethe development of clinicalreasoning. N Engl J Med 2006;355:2251e2.

144. Chi MTH, Glaser R, Farr MJ. The Nature of Expertise. Hillsdale, NJ:L. Erlbaum Associates, 1988.

145. Ericsson KA. Deliberate practice and acquisition of expertperformance: a general overview. Acad Emerg Med2008;15:988e94.

146. Norman G. Research in clinical reasoning: past history and currenttrends. Med Educ 2005;39:418e27.

147. Bordage G. Why did I miss the diagnosis? Some cognitiveexplanations and educational implications. Acad Med 1999;74:S138e43.

148. Lee JK. Qualityda radiology imperative: interpretation accuracy andpertinence. J Am Coll Radiol 2007;4:162e5.

149. Wood D. An Intervention for Diagnostic Overshadowing [e-book].US: ProQuest Information & Learning. Ipswich, MA: PsycINFO,2005.


Narrative review




150. Shojania KG, Burton EC, McDonald KM, et al. The autopsy asan Outcome and Performance Measure (Evidence Report/Technology Assessment No. 58; AHRQ Publication No.03eE002). Rockville, MD: Agency for Healthcare Research andQuality, 2002.

151. Hogarth RM. Educating Intuition. Chicago: University of ChicagoPress, 2001.

152. Croskerry P. A universal model of diagnostic reasoning. Acad Med2009;84:1022e8.

153. Coderre S, Mandin H, Harasyn PH, et al. Diagnostic reasoningstrategies and diagnostic success. Med Educ 2003;37:695e703.

154. Arkes HR, Faust D, Guilmette TJ, et al. Eliminating the hindsightbias. J Appl Psychol 1988;73:305e7.

155. Graber M. Metacognitive training to reduce diagnostic errors: readyfor prime time? Acad Med 2003;78:781.

156. Fioratou E, Flin R, Glavin R. No simple fix for fixation errors:cognitive processes and their clinical applications. Anaesthesia2010;65:61e9.

157. Graber ML. Educational strategies to reduce diagnostic error: canyou teach this stuff? Adv Health Sci Educ Theory Pract 2009;14(Suppl 1):63e9.

158. Sackett DL, Strauss SE, Richardson WS, et al. Evidence-BasedMedicine: How to Practice and Teach. New York: ChurchillLivingstone, 1997.

159. Sox HC Jr, Blatt MA, Higgins MC, et al. Medical Decision Making.Stoneham, MA: Butterworth-Heinemann, 1988.

160. Janis IL, Mann L. Decision Making: A Psychological Analysis ofConflict, Choice, and Commitment. New York: Free Press, 1977.

161. Surowiecki J. The Wisdom of Crowds. New York, NY: AnchorBooks, 2005.

162. Wedding D. Clinical and statistical prediction in neuropsychology.Clin Neuropsychol 1983;5:49e55.

163. Smith WR. Evidence for the effectiveness of techniques to changephysician behavior. Chest 2000;118:8Se17S.

164. Militello L, Patterson ES, Tripp-Reimer T, et al. Clinical reminders:Why don’t people use them? Proceedings of the human factors andErgonomics Society - 48th Annual Meeting, New Orleans,Louisiana, 2004;48:1651e5.

165. Peldschus K, Herzog P, Wood SA, et al. Computer-aided diagnosisas a second reader: spectrum of findings in CT studies of the chestinterpreted as normal. Chest 2005;128:1517e23.

166. Berbaum KS, Caldwell RT, Schartz KM, et al. Does computer-aideddiagnosis for lung tumors change satisfaction of search in chestradiography? Acad Radiol 2007;14:1069e76.

167. Kakeda S, Moriya J, Sato H, et al. Improved detection of lungnodules on chest radiographs using a commercialcomputer-aided diagnosis system. AJR Am J Roentgenol2004;182:505e10.

168. Jiang Y, Nishikawa RM, Schmidt RA, et al. Potential of computer-aided diagnosis to reduce variability in radiologists’ interpretations ofmammograms depicting microcalcifications. Radiology2001;220:787e94.

169. Fenton JJ, Taplin SH, Carney PA, et al. Influence of computer-aideddetection on performance of screening mammography. N Engl JMed 2007;356:1399e409.

170. Hall FM. Breast imaging and computer-aided detection. N Engl JMed 2007;356:1464e6.

171. Daudelin DH, Selker HP. Medical error prevention in ED triage forACS: use of cardiac care decision support and quality improvementfeedback. Cardiol Clin 2005;23:601e14, ix.

172. Olsson SE, Ohlsson M, Ohlin H, et al. Decision support for the initialtriage of patients with acute coronary syndromes. Clin Physiol FunctImaging 2006;26:151e6.

173. Berner ES, Webster GD, Shugerman AA, et al. Performance of fourcomputer-based diagnostic systems. N Engl J Med1994;330:1792e6.

174. Kassirer JP. A report card on computer-assisted diagnosisdthegrade: C. N Engl J Med 1994;330:1824e5.

PAGE fraction trail=22.5


Narrative review




doi: 10.1136/bmjqs-2011-0001492012

2012 21: 535-557 originally published online April 27,BMJ Qual Saf Mark L Graber, Stephanie Kissam, Velma L Payne, et al. error: a narrative reviewCognitive interventions to reduce diagnostic

http://qualitysafety.bmj.com/content/21/7/535.full.htmlUpdated information and services can be found at:

These include:

Data Supplement http://qualitysafety.bmj.com/content/suppl/2012/06/18/bmjqs-2011-000149.DC1.html

"Supplementary Data"

References

http://qualitysafety.bmj.com/content/21/7/535.full.html#related-urlsArticle cited in:

http://qualitysafety.bmj.com/content/21/7/535.full.html#ref-list-1This article cites 140 articles, 25 of which can be accessed free at:

serviceEmail alerting

the box at the top right corner of the online article.Receive free email alerts when new articles cite this article. Sign up in

Notes

http://group.bmj.com/group/rights-licensing/permissionsTo request permissions go to:

http://journals.bmj.com/cgi/reprintformTo order reprints go to:

http://group.bmj.com/subscribe/To subscribe to BMJ go to:


http://qualitysafety.bmj.com/content/21/7/535.full.html

http://qualitysafety.bmj.com/content/suppl/2012/06/18/bmjqs-2011-000149.DC1.html

http://qualitysafety.bmj.com/content/21/7/535.full.html#ref-list-1

http://qualitysafety.bmj.com/content/21/7/535.full.html#related-urls

http://group.bmj.com/group/rights-licensing/permissions

http://journals.bmj.com/cgi/reprintform

http://group.bmj.com/subscribe/



narrative review cognitive interventions to reduce ... · pdf filecognitive interventions to...

Documents