Model Based Segmentation of RadiologicalImages of the Cranium

M. G. Cawley and K. Natarajan

School of Computer ScienceThe University of Birmingham

Edgbaston, Birmingham. B15 2TT

This work is part of the Alvey project MMI/134 "Modelbased processing of radiological images".

Different imaging modalities, for example X-Ray CTand NMR, generate markedly different images from thesame anatomy. Investigating the nature of the imagesproduced has allowed us to develop a low level domainindependent segmentation algorithm.

A scheme to perform further merging has been de-veloped, based on rules and a current world model de-rived from the anatomy, the modality and non-visualknowledge^.

This final stage produces a segmented world represen-tation of the image which is directly comparable to theoriginal current world model.

The X-Ray CT and NMR imaging techniques pro-vide a non-invasive window for the examination of invivo human morphology and physiology, and generate atwo dimensional representation of tissue behaviour underthat imaging modality. These two-dimensional views arenot directly comparable to the knowledge of morphologyand physiology which exists in medicine. The interpreta-tion of radiological images, therefore, requires new skills.This is particularly evident when non-standard scansare taken or when pathology has severely distorted thenormal anatomy. Radiological training, therefore, takesmany years and relies on experience gained through re-porting on many scans. These scans or grey level im-ages are the only visual means of expressing the relativemeasurement of the physical attributes involved in theimaging technique. Automatic segmentation of X-RayCT and NMR images is therefore reliant on the correctuse of domain constraints to produce plausible group-ings of these grey levels. By combining radiological andphysical knowledge in an appropriate scheme a robustsegmentation can be achieved.

The domain constraints derived from the applicationof such knowledge provide a requirement for the initialsegmentation, that is in the instantiation of a currentworld model which can provide the cues dependent onthe anatomical feature to be found. These cues are anexpectation of how difficult the feature will be to findand therefore how finely the image should be segmented.

At the lowest level of the segmentation this allows theprocess to be data-driven and prevents inappropriate tes-sellations from being imposed on the image'

>„* \

• * ^

Hit"

J.I

i i i JfltTro

mmmm

Caudatenucleus

i^^^^P^Jx^ \nteriorhornObrliliffi^il"' the latcral^ S ^ ^ ^ i | | ventricle

! ^

i * *

Internalcapsule

Figure 1:Section through the interventricular foramenand the hypothalamus.

?-sJL ; \

Figure 2:A 64x64 pixels X-ray CT scan showing thegross structures in figure 1.

Figure I1 shows some of the neuroanatomical struc-tures, their positions with respect to landmarks, mutual

1This diagram is taken from [5].

265 BMVC 1990 doi:10.5244/C.4.47

spatial relationships and relative sizes. The figure 22

shows some of these structures but under the CT imag-ing modality. The CT scan clearly shows how the imag-ing modality can drastically change the appearance, andeven visibility, of anatomical structures.

Overview

The components of the model based radiological in-terpretation system described in this paper are shown infigure 3.

Model-based Radiological Imaee Interpretation

User-defined task

Anatomical Knowledge BasevModality-dependent BaseNon-visual knowledge Base

CurrentWorld •*•Frames

Matching andIntelligentControl

Domain-dependent rule base

legion grouping rule base

Attribute generation rule base

Constraint propagation

• Recognition

Segmented• World

Frames

Re-merge regions/re-segment image

Knowledge basedSegmentation

High-levelSegmentation

* tLow-levelSegmentation

Radiological lunge

Figure 3:Schematic view showing the components of thesystem

The static anatomical model is transformed usingmodality dependent knowledge into a dynamic CurrentWorld modeU ' which can be changed or updated de-pending on the results of the segmentation. Initially thecurrent world describes how the anatomy in a specifiedslice of a particular patient is expected to appear un-der one of the imaging modalities; X-Ray CT or NMR.The current world comprises a number of frames3 onefor each discernible anatomical feature. References toother features in the static anatomical model that arenot expected to appear in the selected slice are removedfrom the current world. Within each frame there is a setof feature vectors describing that anatomical structureand its relationship to other structures in the currentworld model by means of a vocabulary. It is throughthis vocabulary that constraints are propagated to thesegmentation phases. The main attributes used so farin the segmentation task are: contrast, relative posi-tion, tissue type and absolute position with respect to agrid.At the lowest level of the segmentation stage, these

2This image is not a CT scan of figure 1, but rather, intends

to show how the anatomical structures, such as the lateral ventri-

cles, caudate nucleus, globus pallidus and internal capsule, appear

under the CT imaging modality.3In our implementation the dynamic current world are Minsky-

like' ' frames.

constraints manifest themselves in two ways, firstly sug-gesting a suitable clustering criterion and secondly sug-gesting image transformation pre-processing (for exam-ple blurring). At the higher level of segmentation, theconstraints are used as expectation to invoke the appro-priate domain-dependent knowledge.

The segmentation stage covers the conversion of partsof the image into a sets of segmented world (SW) frames.Each of these frames is a possible interpretation of theimage under the goal derived from the current world.The development of segmentation precepts has been ac-complished by using combinations of operators (for ex-ample standard deviation and gradient maps).

To summarise, the task of the segmentation is to pro-vide a plausible partitioning of the image given a setof constraints. These constraints are derived from thecurrent world and are therefore tailored to the particu-lar image and patient details. The current world can besubjected to alterations instigated by the matcher but interms of the segmentation stage the current world actsonly in the production of a goal.

Low level segmentation

The first stage in the segmentation of the image isthe production of a symbolic representation which en-compasses some attribute derived from the image. Therepresentation is in terms of regions of the image, eachregion conforming to an attribute value.

The experience gained in dealing with radiological im-ages builds up image processing knowledge or model oftissue behaviour within the image. This is distinct fromphysiology and from the imaging modality modelling.The tissue model is a combination of both the anatomicalstructure and the physical behaviour of materials whenexposed to the imaging process. This type of knowledgeallows us to experiment with standard segmentation al-gorithms. For the low-level segmentation we have chosento segment the image into a number of homogeneous re-gions. This was done using a single criterion split andmerge!A approach in the first instance^ and latterly anadaptive split and merged algorithm. The homogeneoussplit and merge algorithm exploits the assumption thatfor a single tissue the texture is uniform. The adaptivesplit and merge takes as input a "criterion map" whichmeans that any property, for instance statistical mea-sures, can be used to guide the low level segmentation.The importation of parameters into the split and mergealgorithm allows for the passing of domain-dependentconstraints via the use of rules. This does not controlthe segmentation but simple changes the tightness of thecriterion and thus the data driven aspect is retained.

These rules represent the more fundamental sequencesof operations, which under the control of parameters al-low reasonable execution times for processing whilst re-taining flexibility. The split and merge algorithm nowuses a split criterion map rather than a single inputvalue. Rules can still be used, however, to generate this

266

split criterion using pre-defined operators. The rulesthat are used are of a standard form and are invokedfrom a rule interpreter. The basis for this interpreter isa modified version of that developed at Edinburgh forthe Simple Blackboard System[1]'[3l.

The output of the low level segmentation stage is a listof regions each of which is a list of blocks. The blocksremain from the initial quadtree mechanism used to splitthe image into parts. The low level merging stage usesthe same criterion as the split stage and so the quadtreeshould not impose any systematic structure on the data.The blocks, which are represented in symbolic form, arethe basic description of the image and are defined ac-cording to the split and merge criterion. Variation ofthis criterion does not change the symbolic vocabulary.So for instance the blocks are always considered to besmooth and yet by increasing the split and merge cri-terion the actual composition of the blocks can becomemuch more variable.

The low-level segmentation stage, when employingcurrent world constraints such as determining an ap-propriate focus of attention, would typically yield 900regions comprising approximately 4000 blocks.

High level segmentation

The use of more abstract knowledge is required tomerge together regions presented to the high level seg-mentation stage. Unlike the low level stage, this knowl-edge has to bridge the gap between the goal, as derivedfrom the current world, and the symbolic representa-tion derived - with constraints - from the image. Thepossibility also arises of merging regions using the sameattributes as the initial segmentation.

The high level knowledge is split into three parts, eachrepresented using the same form4. Firstly, the controlknowledge transforms the rule requests into a form suit-able for the more domain specific rules. For instancethe region lists are, when applicable, converted into seg-mented world (SW) frames. The SW frames are in ofsame form as the current world frames but are not com-patible with the original region lists. The control knowl-edge produces a common format from either the SWframes or the region lists. This also applies to the currentworld frames when necessary. This could be requiredif, for instance, analysis of the shape of both the SWand the CW were required to match them. This typeof knowledge is primarily concerned with limiting thesearch space. In practice this is achieved by selecting themost appropriate attribute or conjunction of attributes(strategy knowledge), given expectation from the cur-rent world, in order to generate a candidate region list.For example, a simple rule would form a candidate re-gion list using a threshold taken from the current worldfor the particular goal feature.

Secondly there are rules which reorganise the data and

facilitate the writing of merging or grouping rules. Thesetake the formatted information, derived from the controlrules and try to trigger on the premises which may havebeen directed through a selected strategy rule. Theseintermediate rules generally refer to some attribute, orcombination of attributes, which can be derived fromthe image. At present we use attributes such as contrastand average gradients, but because the regional represen-tation of the image is an accurate representation otherattributes may be subsequently calculated. The first ofthese, the contrast between two regions, is calculatedfrom the block descriptions but now yields a symbolicterm such as low, or medium or high. This is shownin figure 4 below. Each of these terms does not havea fixed numerical value although the relationships be-tween the terms are fixed, i.e high is always greater thanmedium,etc.

HIGH

-

max <

region A

region B

d

r

mind>3r

MEDIUM

i

region A

region B

d

r

d > r

LOW

region BminB

max Aregion A

min B > max A

OVERLAPPING

region B

region A

max A > min B

Figure 4:Contrast between neighbouring regions in fourcases. Other possibilities are taken into ac-count but are not shown here.

Thirdly attribute rules are the means by which theactual values are found from the data. They form alink between the rules and the predicates and allow therules to call for such information without duplication ofactions. The rules in the end are property based.

The result of the high-level segmentation is a set ofSW frames containing regions that have been groupedtogether according to the expectations dictated by thecurrent world. When this is not possible, regions aregrouped together according to the actual regional dataattributes. These SW frames, which are in the sameformat as the current world frames, are subsequentlycompared to each other by an intelligent matcher. If asufficient level of match occurs then the recognition taskis completed. Failing this, merging rules that had notpreviously been chosen or fired (possibly due to priori-tisation) are activated, or alternatively a new low-levelsegmentation is requested.

Results

"The knowledge representation scheme adopted by the integral The following shows model based Segmentation beingtask is described in [li] applied to the area of interest depicted in figure 2.

267

The first verticle set of three images are for the non-adaptive split and merge.

The second showing the results when the adaptivemethod of split and merge is applied.

Figure 5:Split criterion set to 2.5 (smallest <r for ex-pected tissue types).

Figure 6:Results of high level grouping according to spa-tial and contrast criteria (rule based) of figure5.

iflWW £ A • rnVftVA J£ • J W L • • A J

Figure 7:Results of high-level grouping super-imposedthe on original image.

Figure 8:Adaptive split and merge - using standard de-viation map.

Figure 9:Results of high level grouping according to spa-tial and contrast criteria (rule based) of figure

Figure 10:Results of high-level grouping super-imposedon the original image.

268

In order to illustrate the segmentation we use as inputFigure 2 and attempt to label the anterior horn of thelateral ventricle, (that is the dark curved structure inthe top left of Figure 2). This scan is firstly subjectedto non-adaptive and adaptive split and merge, figures 5and 8 respectively.

These low-level segmentations are reasonably accuraterepresentations of the original image5 with the data nowrepresenting, symbolically, the underlying homogeneitywithin the image. Figure 5 comprises 3200 blocksgrouped together as 1040 regions, whereas figure 8 con-tains 3150 blocks and 985 regions. These low-level seg-mentations are now grouped together according to ex-pectations dictated by the left anterior horn's currentworld frame. At present this takes the form of select-ing a list of candidate regions whose intensity is withina range defined by the tissue type- cerebral spinal fluidunder CT. This has the effect of limiting the search spaceto approximately 100 regions. Figure 6 and 9 are theresults of applying spatial relationship and contrast con-straints. Finally figures 7 and 10 show the high-levelsegmentation results super-imposed on the original im-age. We can see from these images that both have ex-tracted the anterior horn and this knowledge can nowbe used for further recognition and refinement tasks. Itis also worth noting that had we used the current worldframe for the caudate nucleus (to the right of the anteriorhorn), this structure would also have been extracted.

Summary

The aim of the work, developing an automated imageinterpretation system, has been to create expert modulesfor each step in the interpretation task. Within the seg-mentation stage itself there are two separable tasks, low-level and high-level segmentation. Low-level segmenta-tion is predominantly data driven employing constraintsprimarily derived from the imaging modality. High levelsegmentation propagates these constraints for low levelsegmentation and also sets up goals derived from knowl-edge about neuroanatomy and patient details. Furtherto this the segmentation stage is driven by an intelli-gent controller whose task is to define a strategy usingprocedures such as neurological landmarking.

The result to date shows that the use of simple at-tribute based rules, when invoked through a knowledgebased system providing goal and data driven segmenta-tion, applied with domain-dependent knowledge, can goa long way in the segmentation of a fairly complex butclosed world scenes. In the future the addition of moreattributes to the base rules will be investigated. Theflexibility of the system gives easy access to new ruleswhilst catering for their application through the use ofthe strategy rules.

References

[1] Baldock R. and Towers S., First Steps Towards A

Blackboard Controlled System for Matching Image andModel in the Presence of Noise and Distortion, Proceed-ings of the 4fh International Conference Pattern Recog-nition ,pages 429-438, Cambridge U.K, March 1988.

[2] Pavlidis T., Structural Pattern Recognition, Springer-Verlag, 1977.

[3] Clark P., "SCS - A Simple Control System forthe Control of Software 'Experts' Performing a Col-laborative Task", Alvey MMI/134 Internal ReportMOBPRIM/TI/REP2/880505.May 1988.

[4] Minsky M., "A Framework for Rep-resenting Knowledge",in The Psychology of ComputerVision, P.Winston (Ed.), McGraw-Hill, New York 1975.

[5] Nieuwenhuys R-, Voogd J., van Huijzen C, TheHuman Central Nervous System: A synopsis and atlas,Springer-Verlag, 1988.

[6] Sokolowska, E., Newell, J.A., "Multi-layered imagerepresentation: Structure and application in recognitionof parts of brain anatomy", Pattern Recognition Letters,Vol.4, 223-230, 1986.

[7] Cawley, M.G. and Natarajan, K., "Development ofa model for use in medical image interpretation", Pro-ceedings of the fifth Alvey Vision Conference, AVC89,305-308, 1989.

[8] Natarajan, K., Cawley, M.G. and Newell, J.A.,"A Knowledge-based System Paradigm for AutomaticInterpretation of CT Scans", Accepted for the Biolog-ical Engineering Society meeting "Computer Aided In-terpretation of Medical Images", London, June 1990.

[9] Cawley, M.G., Natarajan, K. and Newell, J.A.,"Data representation for subsequent image interpreta-tion" Accepted for the Biological Engineering Societymeeting "Computer Aided Interpretation of Medical Im-ages", London, June 1990.

[10] Cawley, M.G., Natarajan, K. and Newell, J.A.,"Region based segmentation of CT scans", AlveyMMI/134 Internal Report, April 1990.

[11] Natarajan, K., Cawley, M.G. and Newell, J.A.,"An Integrated Frame and Rule based system for Radi-ological Image Segmentation", Alvey MMI/134 InternalReport, to be released.

5When compared through the use of discrepancy graphs.

269