Multimodal User Interface with Natural Language Classification for

Clinicians At Point of Care

Health Informatics Showcase

Peter Budd

Sponsors: NCCH - Donna Truran Microsoft - Steven Edwards

A Language Model of Health Information

>80% of information of interest is language. Patients and clinicians use language for

> 90% of information exchange. An EMR should be more like a document

than a database record. Data Capture is a language processing

problem more than a form filling problem

Purpose of Hospital Information Systems

Retrieve patient records for clinicians Provide data to answer research questions Provide data to answer Management

questions Provide clinical alerts for critical incidents Provide decision support for patient care

management plan Provide auditing of patient care

Data Analytics

The primary purpose of HIS is to provide extensive support for patient care,

It is not for the medico legal protection of clinicians interests

Data Analytics should be the fundamental objective of a HIS

The storage repository has more in common with a Content Management System than a relational database IS.

Language should be reduced to a canonical form – SNOMED CT

Data Entry - Objectives

Mimic the workplace processing as closely as possible

Identify text as the primary content Make canonical encoding as automatic as

possible Make canonical encoding as hidden from

view as necessary Maximise flexibility in data entry modes

Technology Strategy

Multimodal Interface– Developed on the Tablet PC

Handwriting & Drawing Capabilities Sub-vocal microphones for speech input

– Designed to closely mimic “real” paper forms Generic Form Generation Able to be localised for individual hospitals

– Automatically classify Natural Language Classify free text into SNOMED-CT ontology

Top Level Overview

InterfaceInterface

Form Generator

Form Generator

Augmented Lexicon &

Standard Lexicon

Augmented Lexicon &

Standard Lexicon

Token MatcherToken

MatcherClinicianClinician

DatabaseDatabase

Token Matching

Phase 1– Currently implemented– Matching based off sequence runs of medical

terms– Adjacent words compared against each other– Match with most words used chosen as optimal

match– SNOMED-CT Description table used; Multiple

descriptions map to the same concept

Token Matching

Phase 2– To be implemented as future work– If bad matches are found, words close in spelling

may be used to accommodate mistakes in the handwriting or speech recognition

– Matching algorithm allows inconsistencies/ missing elements in the input

– Uses language knowledge to fill in the gaps

Token Matching

Phase 3– Also not yet implemented– Uses sophisticated Natural Language Processing

techniques to break sentences into “clumps”– Token Matching is then run on the clumps– Allows the negation of SNOMED terms based off

sentence clumps

Form Generation

Necessary attributes of the form are extracted out into an XML format

Form generated “on-the-fly” at program runtime

Allows hospitals to have non-technical staff use interface generator software to localize standard forms or create their own

Output into standard XML for saving into Database

Form Generation

Next Phase of Implementation– Form can be loaded pre-filled or seeded with data

based off statistically average usage– Allow multiple clinicians (Doctors and Nurses)

access to the same form at the same time (from multiple Tablet PCs) to speed up data entry and reduce duplication

– Add speech recognition and video capture to the interface

Conclusion

Project outcomes– User Interface was created which closely mimics actual

forms currently used in the workplace– Automatically classifies natural language into a medical

ontology

Performance issues– Classification runs in acceptable time as a background

process– Form Generation runs in pseudo-real time– Time for form generation well inside time required to pick up

a real paper form

Current Progress

Building an ED information system based on this model

Using Process diagram collated from 3 month study at Westmead ED

Subject of ARC Linkage grant application with Sydney West Area Health Service

Questions