RadQuery: An Interface for Semantic Query of Medical Images to Facilitate Multi-site Research and Discovery

Francisco Gimenez1, Katie Planey1, Sushant Shankir, Vanessa Sochat1, Linda Szabo1

Stanford University Program in Biomedical Informatics

Abstract Methods & Materials Interface Results

Medical imaging is an expansive field that constitutes a wide array of subspecialties including anatomical imaging, functional imaging, nuclear medicine, and image-guided surgery. Rather than segmenting the field, current research is revealing how these subspecialties complement each other, and what is needed is a unifying framework for the storage and transmission of information in such data.

While the Digital Imaging and Communications in Medicine (DICOM) standard was created as a format to meet this need, it falls short of encoding semantic meaning in such imaging data. This current workflow (second column) has several shortcomings that prevent clinicians and researchers from harnessing the true potential of medical imaging data:

Our project uses ontologies in two ways: the existing XML radiologist annotation files in our database were created using ontological terms from RadLex, and our UI query that pulls from this database is limited to RadLex terms. The RadLex ontology allows for a comprehensive description of the entire medical imaging workflow, from the organ (anatomical entity) being imaged to the type of scanner (imaging modality) to the actual radiologist report/annotation. We use three classes from AIM that use RadLex ontological terms from these three categories to query liver lesions. The flow chart below shows how the RadLex, AIM, and RadQuery models work together to pass down RadLex terms in a software implementation.

The AIM model directly uses RadLex terms but does not provide any translation from these RadLex axes. The AIM model consists of 6 sub-concepts or classes: anatomic entity, imaging observation, inference, image reference (pointer to image file), calculation (numerical analysis), and geometric shape (location coordinates).

Shortcomings of Annotation Workflow•Annotations are stored in free text, not easy for computational interpretation.•Reports do not contain standardized semantic data, which inhibits data sharing across sites.•PACS are not set up for cohort-wide queries; a clinician can only conduct a search via a single patient ID or name.•PACS user interfaces are not conducive to viewing multiple patients at once.

The Annotated Imaging Markup (AIM) information model, developed here at Stanford, aims to standardize radiologist observations and capture reports in controlled templates rather than free text (Rubin, 2009). Our project, RadQuery, builds upon this AIM model framework to solve the additional workflow issues of ambiguous semantics in reports, limited cohort query ability, and non-intuitive user interface. RadQuery indexes AIM files in an XML database while storing imaging data on disk, allowing for semantic, ontology-driven query of images.

Automated, digital acquisition of medical images across various imaging modalities offers a non-invasive method for the diagnosis and classification of disease. Historically, the usage of a diagnostic image was limited to a single point in time. A medical expert would read an image, make observations to support a diagnosis, and archive the image for future reference. Missing from this procedure was a step to directly tie the expert’s interpretation to the features present in the image. The advent of AIM (Annotated Image Markup), combined with the increasing availability of expert-annotated images, provides a valuable resource for large-scale data mining of image findings. In this paper, we present an interface that allows for an ontology-based query of semantic data and medical image content to demonstrate how this technology facilitates meta-query, large-scale discovery, and improved diagnosis.

Our flexible search GUI allows a radiologist to quickly search for any relevant text term relating to an inference, entity, or description. It is simple and intuitive because the user does not need to understand the AIM model and how the data was originally collected. We are excited by the possibilities of RadQuery; RadQuery can already provide radiologists with a fully functional database query. With its standardized ontological foundation, RadQuery can be shared across multiple institutions for clinicians and researchers.

Introduction

Figure : How the Radlex, AIM and RadQuery models align to use the RadLex ontology. A RadQuery instance encompasses a user query instance and a database response instance, both of which have slots that are only filled with RadLex terms.

The images in our database were initially annotated by radiologists using terms from the RadLex ontology as part of a liver lesion classification study (Napel, 2010). As text is being entered in the search box, a list of RadLex terms that contain the search text are displayed so the user can select a valid term.

Our system is implemented such that a user is able to search using terms from any level of the ontology and images that have been annotated with either the exact search term or any child term will be returned in the results. Available semantic data that is relevant to the radiologist’s task is displayed along with each image on the results page.

We conducted a quantitative evaluation by manually counting the number of instances of a particular term in our database (a “gold standard” for a particular query), and then manually verifying the results returned matched the gold standard. We successfully returned all corresponding images for our queries of “circumscribed margin” and “enhancement.” This level of testing is only appropriate for the system in its early stages. We have plans for a second evaluation to quantitatively assess performance (see Discussion and Future Work) when being used by medical experts after implementing several changes, discussed below.

Evaluation of RadQuery is important to test functionality, increase awareness of the product, and gain insight to whether or not the product meets the needs of its users. Thus, for the evaluation of our system, we spoke with Dr. Jafi Lipson, a breast cancer radiologist in the Stanford Cancer System. Dr. Lipson demoed the system and provided both formative (suggestions for improvement) and summative (criticism of current implementation) feedback about the interface, use-cases, and query functionality.

Conclusions