development of an automated language interpreting tool for ... · language interpreting tool for...

Development of an Automated Language Interpreting Tool

for Hispanic Mental Health Patients

The Way Forward:Advancing Mental Health Equity in California

March 30, 2017

Behavioral Health Center for Excellence

Colaborators

• Steven Chan• Alberto Odor• Alvaro Gonzalez• Andrés Sciolla• Glen Xiong• Michelle Burke Parish• Shang Wei• Ana-Maria Iosif• Peter Yellowlees (PI)

Support / Grants

• Blue Shield of California• Virtual Collaborative

Health Network• Behavioral Health Center

for Excellence• AHRQ – 2 R01 Grants

Behavioral Health Center for Excellence

Research• 2006 – 2020

• Transcultural Telepsychiatry Across Language Barriers (AHRQ – R01)

• 2013 – 2020• Automated Translation System for Transcultural Telepsychiatry (AHRQ - RO1)

• 2017 –• Automated Facial Expression Recognition in Telepsychiatry (Pilot)

• 2017 –• Augmented Reality to Show Visual and Auditory Hallucinations to Non-Psychotics

• Need to buy some FTE

Behavioral Health Center for Excellence

The Problem

4.7%of the population(over 5 years of age)had limited English-speaking abilities.

This is nearly the size of Illinois.

Behavioral Health Center for Excellence

The Problem

79%16%

3%2%

21%

English only English OK + Other

English "not well" + Other English "not at all" + Other

2010 U.S. CensusLanguage Use

Behavioral Health Center for Excellence

The Problem

Sacramento County languages

90+non-English languages, like

SpanishMiaoChineseVietnameseTagalog

RussianUkrainianHindiGermanHmong

Behavioral Health Center for Excellence

Old Practice

Patient

Provider

Ad hoc

Behavioral Health Center for Excellence

Current PracticePatient

Provider

Behavioral Health Center for Excellence

Current Practice

Patient

Provider

Live interpreter

Behavioral Health Center for Excellence

Current Practice Problem:

•Unfunded mandate.

•Additional $30 - $50/hour

•Most private insurers do not reimburse interpretation. Except in

California. (Wilson 2013, Guntzel 2012)

•35% difference in interpreter salary

Money, Time Quality

Behavioral Health Center for Excellence

Current Practice Problem

Money, Time, Quality

• Increased encounter time by 57% (live vs phone interpreters) (Locatis 2010).

•Travel time delays (Masland 2010)

•Perceived time constraints (Diamond 2009)

Behavioral Health Center for Excellence

Current Practice Problem

Money, Time, Quality

• Health systems determine what’s “reasonable” for quality of language interpretation (The Joint Commission, 2010)

• Hospitals often don’t test staff competency, utilization, or effectiveness of interpreting services(Tschurtz et al., 2011)

• Only 100 of 3,500 medical interpreters in Oregon have proper qualifications (Foden-Vencil, 2014)

Behavioral Health Center for Excellence

Current Solutions

Live interpretersPhone interpretersVideoconferencing interpreters

• Little information about accuracy of live interpretation

• Individual variabilities

• Hypothesis: Machine translation has similar levels of accuracy in a clinical setting.

Behavioral Health Center for Excellence

Ultimate Goal

• Analyze language as data.

• Create new clinical processes for in-person

care and telemedicine.

• Create a better standard of care with

improved interpreting, more efficiently, and

less expensively.

Behavioral Health Center for Excellence

Asynchronous S&F Telepsychiatry Workflow

History (10 years ago)

Visalia

Tulare

History (10 years ago)

Behavioral Health Center for Excellence

Current State of Automated Translation

Behavioral Health Center for Excellence

SDKs Assessment

Se

rvic

e

Ac

ce

ss

Wo

rks

wit

h a

ny

us

er

wit

ho

ut

pre

tra

inin

g

HIP

AA

Tra

ns

cri

pti

on

Tra

ns

lati

on

Ha

nd

les

str

ea

min

ga

ud

io

Un

de

rsta

nd

s

me

dic

al

term

ino

log

y

Co

ns

um

er

ap

p

ava

ila

ble

to

de

mo

?

Google Translate SaaS API ✓ – – ✓ ✓ ? ✓ ? Android, iOS

Nuance online SaaS API ✓

Nuance Dragon NaturallySpeaking SDK

Client DSClocal software –

✓—

local✓ – – ✓

Nuance Dragon NaturallySpeaking SDK

Server DSSWindows local ✓ ✓— ✓ – ✓ – ?

Nuance Dragon Dictate local software –✓—

local✓ ✓

Nuance 360 SaaS API ✓ ✓ – ✓ ✓ third-party apps

Nuance NDEV SaaS API ✓ – ✓ – ✓ Android, iOS

Mac OS X

Client & Apple-

operated

servers✓ – ✓ – – ✓

Microsoft Bing Voice Recognition &

Microsoft Translator on AzureSaaS API ✓ ✓ ✓ ✓ ✓

Windows Phone, web

translation

IBM BlueMix SaaS API –✓ English

only✓ ✓ – web translation

SRIlocal & client-

server✓ ✓ ✓ ✓ ✓ –

SysTran 7 Business Translatorlocal, server, &

SaaS✓ ✓ – ✓ N/A ✓ iPhone, web

Behavioral Health Center for Excellence

Transcripts

Target

language (HUMAN)

Original

language

Target

language (COMPUTER)

Behavioral Health Center for Excellence

Error Types

omission, in which the interpreter (human or machine) did not interpret a word/phrase uttered;

addition, in which the interpreter added a word/phrase not uttered by the patient;

substitution, in which the interpreter substituted a word/phrase for a different word/phrase; and

false fluency, in which the interpreter used an incorrect word/phrase that does not exist in that particular language

Behavioral Health Center for Excellence

Errors in a PTSD Transcript

Inte

rpre

ter

1

Aggregate counts Transl #Data types J L K M NGrand Totaltotal_added_word_count 2 11 1 0 1 15total_drastic_word_count 214 225 277 189 265 1170total_unusual_word_count 25 12 21 33 31 122total_word_count 1202 1303 1181 1189 1253 6128Grand Total 1443 1551 1480 1411 1550 7435

Aggregate counts Transl #Data types J L K M NGrand Totaltotal_added_word_count 4 3 0 6 2 15total_drastic_word_count 244 343 398 278 477 1740total_unusual_word_count 19 68 74 96 71 328total_word_count 1214 1305 1186 1192 1261 6158Grand Total 1481 1719 1658 1572 1811 8241

Inte

rpre

ter

2

Behavioral Health Center for Excellence

Automated Evaluation of Languages

BLEU (Bilingual Evaluation Understudy), an error-scoring algorithm that has found to have high correlation with human error quality with a score between 0 to 1.

Higher scores indicate a higher degree of similarity between human-translated and computer-translated texts.

The BLEU formula is the most widely used automated metric for testing MT apps.

Behavioral Health Center for Excellence

Implications

• Language apps can be incorporated in both asynchronous and synchronous telemedicine video encounters and workflows.

• Not all translation apps are alike: significant differences in the quality between apps.

• These apps are already being used in commercial industries to replace or augment language interpreters. Expect increased reliability and accuracy in the future.

Thank you

aodor@ucdavis.edu