A Biomedical Information System for Combined Content-Based Retrieval ofSpine X-ray Images and Associated Text Information

Sameer Antani L. Rodney Long George R. ThomaU.S. National Library of Medicine

8600 Rockville Pike, Bethesda, MD 20894{antani,long,thoma }@nlm.nih.gov

Abstract

We present the status of ongoing work toward the devel-opment of a biomedical information system at the ListerHill National Center for Biomedical Communications, a re-search and development division of the National Library ofMedicine (NLM). For any class of biomedical images, theproblems confronting the researcher in image indexing are(a) developing robust algorithms for localizing and identi-fying anatomy relevant for that image class and relevant tothe indexing goals, (b) developing algorithms for labelingthe segmented anatomy based on its pathology, (c) develop-ing a suitable indexing and similarity matching method forvisual data, and (d) associating the text information on theimaged person, indexed separately, for query and retrievalalong with the visual information. We are in the process ofbuilding a content-based image retrieval system which sup-ports hybrid image and text queries and includes a biomed-ical image database. The paper describes this prototypeCBIR2 system and the algorithms used in it. Image shape-based retrieval is done by image example and user sketch ofthe vertebrae on the spine x-ray images from the NationalHealth and Nutrition Examination Survey (NHANES) data.

1. IntroductionThe general problem of developing algorithms for the au-tomated or computer-assisted indexing of images by struc-tural contents is a significant research challenge [1]. Thisis particularly so in the case of biomedical images, wherethe structures of interest are commonly irregular, and maybe partially occluded. Examples are the images createdby digitizing film x-rays of the human cervical and lum-bar spines, digitized color slides of the uterine cervix, colorendoscopy, endoscopic ultra-sonography, etc. Text data inthe form of patient or survey data is commonly associatedwith medical images. Systems in use allow retrieval of im-age data through a text based query. Content-based imageretrieval aims to allow researchers and medical practitionersaccess to the images directly by their content. We envisagethat the development of a system that provides such access

would have many applications in education, research, clini-cal trials, diagnosis, etc. For example, a medical school fac-ulty member who is an expert in degenerative spine diseasecould query for examples of severe disc space narrowing forboth sexes for the cervical spine; or a clinician could use itfor searching for images similar to patient’s present imagepathology or injury.

The Lister Hill National Center for Biomedical Commu-nications, a research and development division of the U.S.National Library of Medicine (NLM), maintains a digitalarchive of 17,000 cervical and lumbar spine images col-lected in the second National Health and Nutrition Exam-ination Survey (NHANES II) conducted by the U.S. Na-tional Center for Health Statistics (NCHS). Along with the10,000 cervical spine and 7,000 lumbar spine images, theNHANES II survey also included information on demo-graphics, health questionnaire responses and physician’s ex-amination results. Over 2000 fields of information are avail-able on each surveyed person, providing a large body of textinformation. Sub-sampled examples of the cervical spinex-ray is shown in Figure 1. Classification of the imagesfor biomedical researchers, in particular the osteoarthritisresearch community, has been a long-standing goal of re-searchers at the NLM, collaborators at NCHS, and the U.S.National Institute of Arthritis and Musculoskeletal and SkinDiseases (NIAMS), and capability to retrieve images basedon geometric characteristics of the vertebral bodies is of in-terest to the vertebral morphometry community.

In this paper we describe the design of the prototypeCBIR2 system that supports hybrid image and text queries.Image queries are posed by image-example or user-sketch.The CBIR2 system was developed after an initial feasibil-ity study done through the development of the CBIR1 sys-tem [5]. This system was used to develop initial ideas onthe algorithms, functionality, and interface characteristicsof a CBIR system for digitized x-rays. The lessons learnedfrom CBIR1 helped us establish clear objectives and definea superior architecture for the CBIR2 system.

The remainder of the paper is organized as follows. Sec-tion 2 outlines the research objectives. The design descrip-

Figure 1: A sub-sampled cervical spine x-ray image

tion, algorithms used and current status of our research ispresented in Section 3. We summarize our work and projectfuture work in Section 4.

2. GoalsThe open research goals within the scope of the content-based biomedical image retrieval system include (i) contentextraction, representation, feature classification, and simi-larity computation; and (ii) development of suitable queryparadigms, indexing schema, and algorithms for efficientretrieval. For biomedical images the notion of image con-tent, its representation, and similarity has an added dimen-sion of relevance. It is very likely that the queries presentedto the CBIR system are going to be specific to the pathologywhich may be subtle differences from what is considered tobe normal. It is important that the extracted image featuresdescribing it are retained through the data-reducing featurerepresentation process.

In general, our overall goal is to make a significant con-tribution to the image- and multimedia-rich digital libraryof the future by advancing CBIR techniques applied to theNHANES x-ray images, and eventually to other biomedicalimages. Specific objectives include:

• Conduct R&D in the steps needed for biomedicalCBIR, viz., shape segmentation, feature extraction,feature vector organization and classification.

• Conduct R&D into query techniques suitable forbiomedical images.

• Develop the algorithms needed to implement both in-dexing and retrieval.

• Design and develop a next-generation CBIR system(incorporating these algorithms).

Indexing System

Calculate FeatureVector

Representation &Boundary

Landmark Selection

Build FV Tree

Segmentation

GUI

Collect Text Data

Build RDB Table

NHANES Text

and Image Data

Figure 2: Architecture of the indexing system

• Develop a ground truth, evaluate the system, validatethe results, and characterize the performance of thesystem.

• In the long term, extend CBIR techniques developedfor the NHANES images to other biomedical images.

3. CBIR2 System Design and currentWork

In this section we describe the design architecture of ourcurrent prototype system, CBIR2. The system designed ina modular fashion and is logically composed of two separatesystems, viz., the indexing system and the retrieval system.The indexing system includes methods for automated imagesegmentation, image feature extraction, feature vector com-putation, feature organization, and text data organization.The retrieval system provides the interface and the meth-ods for image and text retrieval. For this it includes meth-ods for extracting features from example images, comput-ing the feature vector, and determining similarity betweenfeatures extracted from the query visual and those stored inthe database. In addition, text retrieval via SQL and meth-ods to combine the text and image queries are also included.The architectures for the indexing and retrieval systems areshown in Figures 2 and 4. The modular design of CBIR2allows us to place simplified and possibly inefficient algo-rithms in place and yet have a completely developed testbedsystem while we wait on results from our research in eachCBIR task.

Figure 3: Active Contour Segmentation Tool: Main Win-dow

3.1 Indexing system

The indexing process is currently semi-automated and donevia a graphical interface. This interface allows indexing oftwo types of data. The text data is organized as fields in arelational database table from which data can be retrievedusing the MySQL relational database manager. The index-ing of the image data on the other hand is a more involvedprocess. We describe the system modules here.

I Segmentation: The first step in indexing the imagesis the segmentation of the objects of interest, the ver-tebrae. The image quality in the spine x-ray images isfairly poor with ambiguous vertebral boundaries, mak-ing a reliable segmentation a challenging task. Cur-rent implementation contains human assisted bound-ary segmentation using Active Contour Segmentation(ACS). For this we have developed a tool shown in Fig-ure 3. Our current research efforts into reliable shapesegmentation include applying Active Shape Modeling(ASM) techniques [2] to extract the vertebra bound-aries. The ACS tool allows the user to place an ini-tial template on the vertebra and apply the ACS al-gorithm. The user may enhance the image using his-togram equalization before applying the method. Thetool allows the user to create a template by markingpoints around the vertebra. The template can then besaved for future use. The position and size of the tem-plate can be controlled by rotation, translation and scal-ing prior to invoking the ACS algorithm. After the seg-mentation, the user can accept and save or discard thesegmentation results. If the segmentation is accepted,the tool estimates the location of the next vertebra and

places the template on it and the process is repeated.If at any point the segmentation is not acceptable, theuser can perform manual segmentation on the vertebra.We have incorporated this feature in the tool to allowdevelopment of CBIR2 to proceed while we developmore reliable automated segmentation techniques. Thetool generates segmentation output in a XML style fileand is given a.cbr file extension. Here we store foreach object, the template, and automated, and manualsegmentation results. This way, entries can be modifiedas needed following future developments. The.cbrfile records the information about an image, databasesource, view (e.g., lateral, sagittal, AP), and the hu-man segmentor in the header structure. Additionally,the coordinate system origin is also specified. This isused by the objects within the image as a referencepoint. The image identifier is the same as that in thetext database, allowing cross indexing across the im-age and text databases. The segmented objects arestored with a unique object identifier, anatomy iden-tifiers, region of the anatomy, the segmented boundarypoints, the bounding box, the oriented bounding box,etc. The unique object identifier allows many versionsof the segmented object to be retained. This has beendesigned so that a variety of database schema can bemaintained. For example, the current best segmenta-tion could be exposed for user CBIR searches, whileresearch could proceed with other segmentation tech-niques.

II Feature extraction and representationThe next stepafter shape segmentation in a CBIR system is repre-senting the shape boundary information. The denseboundary points extracted as(x, y) coordinates in theimage space stored in the.cbr file need to be rep-resented in a form suitable for archiving, indexing,and similarity matching. For this they are reduced toa small set of meaningful representative points by ashape representation algorithm. This coarse boundaryand a binary image representation of the vertebra areused to find meaningful shape features that are invari-ant to translation, rotation, scaling and starting-pointshift.

Most shape-based CBIR methods, to date, have beenapplied to art, trade-mark databases, fish images, sil-houettes of tools, etc. From the literature, we observethat most shape representation methods use the globalshape characteristics for indexing; i.e. the final shaperepresentation is controlled by the distribution of allthe boundary points in the image space. Such an ap-proach may not be suitable for biomedical shapes dueto (i) high similarity across anatomical shapes, such asvertebrae (ii) loss of subtle differences in boundary rep-resentation which could be indicative of certain pathol-

ogy, and (iii) in adequacy of shape representation meth-ods for supporting region localized queries.

The methods selected are briefly described below andare all currently implemented in CBIR2. We are in theprocess of developing other more suitable shape repre-sentation approaches.

a Global Shape Properties and Invariant Moments.Global shape properties and moments are featuresintrinsic to a shape. The properties used are ma-jor and minor axis length and angle, compactness,roughness, and elongation. In order to computethese we converted the shape contour to a binary im-age and gave the same weight for each pixel insidethe shape contour boundary. In addition, we alsoused first and second order 2D invariant momentsdefined by Hu [4].

b Scale Space Filtering. Scale space filtering re-formats the shape boundary points to represent theshape at different levels of detail. It is said to fol-low human perception of shapes [3]. It providesmore detail at scale higher level and progressivelyreduces the detail level until the shape becomes anoval shape. While capable of shape matching, aproblem with this method is that the shape shrinksas it progresses from high detail level toward lowdetail level making comparison scale sensitive.

c Polygon Approximation. Polygon approximationor curve evolution is a process that eliminates in-significant shape features and reduce the number ofdata points. The resultant representation is one thatuniquely describes the shape. The approximatedcurve was then converted to tangent space for simi-larity measurement [6].

d Fourier Descriptors. The position of a point ona closed contour is a periodic function. Thus, theFourier series may be used to approximate the con-tour. The resolution of the contour is approxima-tion is determined by the number of terms in theFourier series. Since simple operations such as scal-ing and translation are related to simple operationsof the boundary’s Fourier descriptors, they are at-tractive for use with boundary matching [12]. Ro-tation invariance computation, however, requires thebend angle function to be computed.

III Feature organizationA feature vector is then createdfrom various computed features and organized into adata structure for efficient retrieval. The developmentof a feature organization strategy is strongly correlatedwith the feature vector used, the query types supported,and the image semantics. We are at a stage where wehave some of the these requirements identified. While

Search FV Tree

GUI

RDB Tree

FV Tree

Image Archive

Calculate FeatureVector

Representation &Boundary

Landmark Selection

Segmentation

Query by Image

ExampleQuery by User

SketchProcess Text

Query

Rank Results

Combine Results

Retrieval System

Figure 4: Architecture of the retrieval system

research is proceeding toward determining an effectivefeature vector organization strategy, we are currentlyusing a flat structure and linear search for retrieval.Having an inefficient, but working system enables usto improve on various modules as the research evolves.

IV Feature classification Our work toward the index-ing of spine images for features of interest in the os-teoarthritis and vertebral morphometry research com-munities requires the segmentation of the images intovertebral structures with sufficient accuracy to distin-guish pathology on the basis of shape, labeling of thesegmented structures by proper anatomical name, andclassification of the segmented, labeled structures intogroups corresponding to high level semantic featuresof interest, using training data provided by biomedicalexperts. We have adopted a hierarchical approach tosuch indexing that consists of high-level region classi-fication, spine region localization, vertebra localizationand identification, vertebral segmentation, and classi-fication of the vertebrae by presence/absence of thebiomedical features above [7, 8, 9]. Classification ofthe vertebra for biomedical features is being collabora-tively done with Stanley [10, 11]. However for shapebased retrieval, labeling, classification, and extractionof features of interest, the vertebra boundaries need tobe segmented from the images.

Figure 5: Query Dialogs: Main Screen

Figure 6: Query Dialogs: Options Dialog

3.2 Retrieval system

As mentioned before, CBIR2 is logically organized as in-dexing and retrieval systems. However, in reality, the in-terface presented to the user is the “retrieval” system. Theindexing process can be conducted through the same inter-face. The system is implemented in MATLAB. The basictypes of queries supported are to the text data, image dataand combined queries to both. The retrieval of the text datais supported through Open-Database Connectivity (ODBC)protocol to retrieve results using the MySQL DBMS. Thequeries to the image data can be specified in using an ex-ample image to retrieve images that are visually similar orby drawing a sketch of the indexed feature, in this case thevertebra boundary. The system presents the user with a GUIfor creating queries and supports text, image example, andimage sketch queries, and queries that combine text and im-age example or image sketch. Figures 5 and 6 show theinitial screen and the options screen for generating the basicquery. A feature in CBIR2 allows users to save and recalltheir queries. The retrieval paths for image-example basedqueries and sketch-based queries are the same except for thefeature extraction phase necessary for the former. The samefeature extraction phase as in the indexing process is ap-plied to the example image. The user is presented with theACS tool for segmenting the image. The extracted image

Figure 7: Query Dialogs: Query-by-example dialog.

Figure 8: Query-by-Sketch (Template) Dialog

features in the query are then matched by a shape similar-ity algorithm to determine the similarity distance betweenthe query and the database shape. The greater the distancebetween two feature vectors the greater is the dissimilarity.The system allows users to specify an image for an image-example based query as shown in Figure 7. For an sketch-based query, the users may choose to either use one of theprovided templates, or use their own template and modify itor draw an outline from scratch. The user can save the tem-plate for a future use. The dialog boxes for these are shownin Figures 8 and 9. It is also possible to narrow the query to

Figure 9: Query-by-Sketch (User Sketch)

Figure 10: CBIR2 Results

the segmentation done by a particular algorithm. This is be-cause the.cbr file allows the user to store several segmen-tations, including manual segmentations. Additionally, withseveral shape representation methods included, the user canalso query on a specific representation. This feature is ex-tremely useful in conducting evaluations for segmentationand representation algorithms. The system also reports aprofile of the execution time taken for making a databasequery, computing the similarity, etc. This will be usefulin evaluating overall system performance and advantagesgained through better feature organization. The results fromthe query are presented as shown in Figure 10.

4. Conclusions and Future work

In this paper we have described the work in progress to-ward building a complete biomedical information system.The system has a multimedia component and a text databasecomponent. This is in line with our eventual goal of de-veloping a system that not only performs conventional textqueries, provides content-based image retrieval queries, butalso provides the user with meaningful information fromthe image classifications. The CBIR2 prototype describedabove provides a testbed for this.

The development of CBIR2 allows us to investigate vari-ous tasks at hand towards achieving the goals identified ear-lier. Automated shape segmentation and the issue of vali-dation of the segmented boundary are yet to be addressed.The shape representation and similarity methods are limitedin their abilities; i.e., while they allow matching of the en-tire vertebra shape, it is not possible to pose queries on localfeatures of importance such as anterior osteophytes. Addi-tionally, we are still in the process of identifying a suitablefeature organization structure. Finally, development of aground truth for evaluation and system performance char-acterization is planned.

References

[1] S. Antani, R. Kasturi, and R. Jain. A survey on the use ofpattern recognition methods for abstraction, indexing and re-trieval of images and video.Pattern Recognition, 35(4):945–965, 2002.

[2] T. F. Cootes and C. J. Taylor. Statistical models of ap-pearence for computer vision. Technical report, Univer-sity of Machester, Wolfson Image Analysis Unit, ImagingScience and Biomedical Engineering, University of Manch-ester, Manchester, M12 9PT, U.K., February 2001.

[3] A. Del Bimbo and P. Pala. Shape indexing by multi-scalerepresentation.Image and Vision Computing, 17(3-4):245–261, 1999.

[4] M. K. Hu. Visual pattern recognition by moment invariants.IRE Transactions on Information Theory, 8:179–187, 1962.

[5] D. M. Krainak, L. R. Long, and G. R. Thoma. A methodof content-based retrieval for a spinal x-ray image database.Proceedings of IS&T/SPIE Medical Imaging 2002: PACSand Integrated Medical Systems, 4685:108–116, February2002.

[6] L. J. Latecki and R. Lak̈amper. Shape description and searchfor similar objects in image databases. In R. C. Veltkamp,H. Burkhardt, and H. P. Kriegel, editors,State-of-the-Art inContent-Based Image and Video Retrieval, volume 22 ofComputational Imaging and Vision, pages 69–96. KluwerAcademic Publishers, 2001.

[7] L. Long, D. Krainak, and G. Thoma. Identifying imagestructures for content-based retrieval of digitized spine x-rays. InProceedings of IS&T/SPIE Medical Imaging 2002:Image Processing, Vol. SPIE 4684, pages 1204–1214, SanDiego, CA, February 2002.

[8] L. Long and G. Thoma. Computer-assisted retrieval ofbiomedical image features from spine x-rays: Progress andprospects. InProceeding of the14th IEEE Symposium onComputer-Based Medical Systems (CBMS 2001), pages 46–50, Bethesda, MD, July 2001.

[9] L. Long and G. Thoma. Landmarking and feature local-ization in spine x-rays. Journal of Electronic Imaging,10(4):939–956, 2001.

[10] R. J. Stanley, M. Cherukuri, L. R. Long, and G. R. Thoma.The application of shape features to cervical spine vertebraimage analysis. Submitted to IEEE Transactions on MedicalImaging, 2002.

[11] R. J. Stanley and L. R. Long. A radius of curvature-basedapproach to cervical spine vertebra image analysis. In38th

Annual Rocky Mountain Bioengineering Symposium, vol-ume 37, pages 385–390, 2001.

[12] C. Zahn and R. Roskie. Fourier descriptors for plane closedcurves.IEEE Computer, C-21(3):269–281, 1972.