Speech Recognition: Pitfalls and Potential Solutions

Leslie E. Quint, M.D.

Speech Recognition• Large proportion of radiology practices

now use automatic speech recognition technology for report generation

• Many advantages and problems

Speech RecognitionAdvantages• Rapid report turnaround time

• During normal work hours• After hours (no transcriptionists)

• More timely presumably enhanced pt care• Fewer workday interruptions for unread cases• Ability to edit report while interpretation is still

fresh in mind and images are displayed• Change nuances• Catch errors

• Transcription cost savings

Speech RecognitionProblems• Increased “look away” time• High transcription error rates • High error rates in finalized reports• Decreased radiologist productivity

Problem• Looking at desktop computer while

dictating instead of looking at images

• Esp. if using templates and filling in findings/data

• May compromise image interpretation

Increased “Look Away” Time

Increased “Look Away” TimePotential solutions• Dictate entire report while looking at

images (similar to days of yore)• Then turn to computer screen and

edit• (Cannot do this with a template) • Do not use continuous typing mode

High Transcription Error RatesProblem• Manufacturers claim “up to 99%

accurate”• Word by word basis• Typical body imaging reports may

have hundreds of words• Reported error rates 2-13% of words

Quint LE. JACR 2008;5:1196Ramaswamy MR. AJR 2000;174;617

High Transcription Error Rates• 74% of plain film reports• 92% of CT reports

Bhan SN. CARJ 2008;59:203

• 90% of MR reports (vs 10% for conventional dictation)

• 5.1 errors per report (vs 0.12 for conventional dictation)

Pezzullo JA. J Digital Imaging 2008;21:384

High Transcription Error Rates

Factors leading to increased error rates:

• Busy (noisy) inpatient environment• Non-native English speakers

McGurk S. British J Radiology 2008;81:767

High Transcription Error Rates• High ambient noise affects error rates• Sound masking may be helpful (mask

extraneous sounds) or detrimental (interfere with word recognition)

• Low level sound masking provided slightly improved SR accuracy (not statistically significant)

• Higher level masking worse SR accuracy

Zwemer J. Radiology 2009;252:691

High Transcription Error Rates

Voll K. J Digital Imaging 2008;21:371

• “CAD” first pass error detection for reports• Statistical method for error detection• Automatically flags potential errors• Captures words whose occurrence in their

context window is highly improbable• Error detection rates as high as 96%• Mean sens 83% (few false negatives)• Mean spec 26% (many false positives)

High Transcription Error RatesPotential Solutions

• Use latest software updates• Make sure microphones functioning well• Reduce ambient noise• Use all available training (voice,

punctuation, personal vocabulary)• Use available microphone adjustment

techniques• Consider use of macros and templates

High Transcription Error RatesPotential Solutions

• Use language model tailored to specific foreign accents• Indian•Australian•?New York ?Midwestern

• Future: CAD, language model tailored to modality

Paulett JM. J Biomed Informatics 2009;42:53

High Final Report Error RatesProblem

• Frequent errors in finalized reports• Potential impact on patient care• Erodes credibility of radiology• The report is our ultimate product

High Final Report Error Rates

• Front end mode: • self editing by radiologist

• Back end mode: • initial editing by transcriptionist• final editing by radiologist

High Final Report Error Rates

• Front-end editing by radiologist• 35% for SR vs 3% for conventional

dictation• None significant to alter patient

management

Pezzullo JA. J Digital Imaging 2008;21:384

High Final Report Error Rates• Front-end editing by radiologist • 4.8% for SR vs 2.1% for

conventional dictation (p=0.002)• 52% of all errors affected

understanding• Probably none adversely affected

patient management

McGurk S. British J Radiology 2008;81:767

High Final Report Error Rates• Front-end editing by radiologist • 22% of CT reports contained

significant errors that might render the report confusing

• Report error rates by individual radiologists ranged from 0-100%

• No significant diff between native and non-native English speakers

Quint LE. JACR 2008;5:1196

High Final Report Error RatesMost frequent:

• Wrong word substitution• Often (not always) sound-alike words

•Ill-defined / well-defined•Renal / adrenal

• Missing words• “left” “no”• “[no] definitely significant lung nodules”

Quint LE. JACR 2008;5:1196

High Final Report Error Rates

Incorrect numbers• Dates, measurements

•14 instead of 4•5 instead of 2.5

Quint LE. JACR 2008;5:1196

High Final Report Error Rates

Awareness of problem:• 63% of radiologists believed overall

dept report error rate < 10%• 76% of radiologists believed that

their own report error rate < 10%• Overall rate 22%

Quint LE. JACR 2008;5:1196

High Final Report Error RatesPotential Solutions

• First pass back end editing by transcriptionist

• Reward good reports rather than fast reports

• Self editing:• Don’t batch read; edit after each dictation• Read (mumble) out loud • Be meticulous!

High Final Report Error RatesPotential Solutions

• Let it sit before one last look and signing

High Final Report Error RatesPotential Solutions• Beware of inappropriate macros

and pre-filled (default) templates• CT of the chest using [80] ml

[Isovue-300] [without complication]. • CT of the chest using [ ] ml [ ] [ ].

Decreased Radiologist Productivity• Signif extra time to dictate and edit• Especially in self edit mode (front

end editing)• Contributes to radiologist fatigue and

frustration• Work shift from transcriptionists (low

paid) to radiologists (high paid) not cost effective

Bhan SN. CARJ 2008;59:203

Pezzullo JA. J Digital Imaging 2008;21:384

Decreased Radiologist ProductivityPotential solutions• Back end editing by transcriptionists• Caveats:

• Meticulous back end editing needed to catch numerous errors

• Careful final editing still necessary

Speech RecognitionConclusions• Many advantages and problems• Awareness of scope of problems

may help to devise and enact potential solutions

• SR is here to stay