Tracking Cataract by the Four-Line Method

K. J. Hanna* L. Tarassenko

Robotics Research Group & Medical Engineering UnitDepartment of Engineering Science

University of Oxford

An implementation of Scott's 'four-line' algorithm for es-timating optic flow is presented. The application is theanalysis of cataract motion in the human eye lens. Im-ages are recorded using retro-illumination photography.Successive images recorded at up to a 6 month intervalare used as the source for the flow recovery procedure.The four-line algorithm itself uses edge data from theseimages at several scales. A procedure for performing im-age registration and optic flow recovery using the samefour-line algorithm is presented. Several examples of thealgorithm output are given.

1. INTRODUCTIONThis paper presents the results of an implementation ofScott's four-line method for recovering optic flow1. Edgedata at several scales derived from the Canny algorithm2

are used as the image features to be tracked. The flow re-covery procedure was performed on a sequence of in vivocataract images recorded using retro-illumination pho-tography.

Section 2 gives a brief medical background to cataractin relation to this paper so that the results of the al-gorithms may be interpreted. Section 3 discusses par-ticular experimental and clinical constraints associatedwith the application, and shows that edge data at sev-eral scales are suitable features for the tracking proce-dure. Section 4 briefly discusses the basis of the four-linemethod and some other optic flow recovery algorithms.It is shown that an algorithm based on the four-line tech-nique is suited to this application. Section 5 summarisesthe four-line technique, and section 6 describes the algo-rithms used for tracking cataract. The complete trackingprocedure is then summarised in section 7. Several ex-amples of the tracking algorithms are then presented insection 8.

2. MEDICAL BACKGROUND

It is known that certain types of motion are associatedwith certain types of cataract3. Much information aboutthe progression of the disease could be gained if cataract

regions could be tracked over a time period. This is par-ticularly relevant in relation to drug therapy, in whichthe efficacy of a therapeutic agent must be quantified.

Images are initially recorded on photographic film us-ing retro-illumination photography4 5. The techniqueproduces an en face view of the human lens by the re-flection of light from the optic disc through the humanlens and onto a film plane. Figure 1 gives a typical exam-ple of the resultant image. The large, circular region inthe centre of the image is the human lens. The dark re-gion surrounding the lens is the iris. Any cataract regionsin the lens results in a reduced light return at that point,leading to the dark regions within the lens shown in fig-ure 1. It is these individual cataract regions which areto be tracked using successive retro-illumination imagestaken at up to 6 month intervals.

*The author acknowledges the support of the Science and Engi-neering Research Council.

Figure 1: Retro-illumination image of a human lens

3. EXPERIMENTAL AND CLINICALCONSTRAINTS

Due to experimental variation, the angular and trans-lational position of the eye lens varies between imageswithin a sequence. Therefore, an exact lens position isunknown from a priori information, leading to a registra-tion problem when analysing a sequence of images. Also,a secondary effect is the slight motion effect due to im-age foreshortening. The difficulty here is the separationof motion due to experimental variation and that due to

253

AVC 1988 doi:10.5244/C.2.39

true cataract motion. The same experimental variationcan cause changes in the global image intensity due toincreased or decreased reflection from the optic disc. Afinal variation is the introduction of low frequency spatialgradients within the image due to the addition of light re-flected from the cornea of the eye.

A clinical constraint which may be imposed is a boundon the maximum magnitude of the motion of cataract.Although cataract regions are known to move, the mo-tion is 'small' in relation to the spatial extent of the com-plete cataract area for the time intervals involved. On aglobal scale, this constraint leads to the gross shape ofthe complete cataract area being maintained within animage sequence.

A final constraint is that motion boundaries describethe change in cataract. At this stage, we are primar-ily interested in the movement and spreading of smallerregions of cataract within larger lens areas.

of the algorithm, the initial estimates of the velocity fieldmay be set to arbitrary values, or alternatively, estimatesbased on the matching surface can be chosen. A weightedleast squares calculation is performed between the vari-ances of the neighbour constraint lines and the matchconstraint lines described above. This results in a new es-timate of velocity at each point. A set of new constraintlines may now be calculated and the process repeateduntil the new estimates of velocity are not significantlydifferent from the previously calculated values.

The presence of two orthogonal match constraint linesat a point, each with low variance, suggests that pointmotion has occurred. However, if one match constraintline has a high variance, this suggests that edge motionhas occurred. The procedure is effective at combiningdifferent smoothness assumptions depending on the dif-ferent types of data structure. The resultant flow-field isaccurate and undistorted at motion boundaries.

4. MOTION ALGORITHMSSeveral techniques may be employed to generate the mo-tion field. Individual cataract regions may be segmentedand matched using minimal mapping theory6. However,algorithms based on this technique are best suited forisolated cataract features. In many cases, cataracts aredensely clustered within single regions (see figure 1).

Algorithms founded on the continuity and smoothnessof motion 7 8 could be used to recover the flow field. How-ever, it was shown in the previous section that motionboundaries describe the change in cataract. Imposingthe two constraints of continuity and smoothness of theflow field will distort the true flow field at these motionboundaries, leading to inaccurate estimates of the truemotion.

An alternative approach is the four-line technique.This procedure is accurate in recovering a flow-fieldundistorted at boundaries.

5. THE FOUR-LINE METHODThe method is described fully elsewhere1, however, a briefoutline of the procedure is given below.

The first step in the algorithm is the calculation of amatching strength surface between features in a time se-ries of images. Scott's algorithm employed a correlationtechnique for this purpose. The second step in the algo-rithm is to perform a principal axes decomposition withinthe match surface for every point in one image. The re-sult is two orthogonal match constraint lines, each withan associated variance attached to it. This variance is ameasure of the spread of possible matching points. Thesetwo constraint lines now remain fixed throughout the pro-cedure. A second set of orthogonal constraint lines arethen calculated for every point, based on the estimatesof velocity of neighbouring points. In the first iteration

6. THE CATARACT MOTION RE-COVERY PROCEDURE

Feature selection

The four-line procedure requires the calculation of amatching strength surface between image features. Itwas shown in section 3 that absolute image intensitiesmay not be used as a basis for this calculation. Thissuggests the use of edge data as a source for the motionrecovery procedure.

It was also mentioned that the gross shape of the com-plete cataract is maintained over the time periods in-volved. These two points suggest a cataract motion re-covery procedure which uses edge data at low scales toregister the images, and edge data at higher scales to re-cover the true motion of individual cataract regions.

Edge data derived from the Canny algorithm2 was usedin this implementation. The choice of edge scale is cho-sen by the adjustment of the Gaussian smoothing oper-ator, which is the first step in the algorithm. Figure 3gives examples of the algorithm output when applied tothe lens regions corresponding to figure 2 . Subsets ofthe complete lens regions have been taken for illustrativepurposes only. The edge data has been superimposedsuch that black indicates an edge due to the first image,and grey an edge due to the second image. Each imageis from the same patient, but separated by a six monthinterval. The edge detection has been performed at a lowspatial scale (a = 12). Figure 2c shows the results ofedge detection at a higher scale (<r = 4).

The Canny algorithm returns data on edgel position,strength and orientation. Edgel orientation is used as thebasis of the calculation of the matching strength surface.

254

Recovering match constraint lines

The first step in the four-line procedure is the recoveryof the two orthogonal match constraint lines described insection 5.

One of the first problems to be tackled is the choiceof effective search area for a matching process. As men-tioned in the previous section, the complete motion re-covery procedure will result in large motion fields at lowscales, and relatively small motion fields at higher scales.This may suggest a hierarchy of search areas, rangingfrom the complete image area at gross scales to tens ofpixels at high scales. However, the selection of searcharea will be completely arbitrary and extremely imagedependent.

A further point is the implied separation of image reg-istration from true cataract motion in the procedure de-scribed above. This suggests that complete registrationmust be declared at one particular scale. Any motionrecovered at higher scales would then be due to truecataract motion. However, the choice of such a scalewould also be arbitrary and image dependent.

A partial solution to the problems described above isto use precisely the same algorithm at both low and highscales. Also, a technique can be devised for perform-ing continuous registration at both low and high scales.Gross registration will still be performed at the lowerscales, although a small registration vector may be re-turned at the higher scales.

Choosing candidate matches

It was shown in the previous section that a matchingprocedure must be devised which can operate at bothlow and high scales. One of the simplest adaptive pro-cedures is the choice of the nearest edgels with similarorientations as candidate matches. However, a problemhere occurs when part of a real edge within the cataractfails to be detected in both images of the sequence. Theprocedure could then choose incorrect candidate edgelsfrom elsewhere in the image. Also, because the effectivesearch space for the process must extend to the wholeimage, a small number of grossly incorrect matches couldalter the resultant flow-field dramatically.

The converse problem occurs when edgels of similarorientation are declared as candidate matches when thetrue correspondence is further away. The effect of thistype of incorrect match may not be as dramatic as thetype described above, however large numbers of incor-rect matches will result in an inaccurate estimate of themotion field.

Reducing match errors

The first of the two problems described above is addressedby performing an inverse matching procedure. A relatedtechnique was implemented by Spacek9.

Once the nearest candidate edgel in the second im-age has been chosen, the same procedure is performedbetween that edgel and the first image. If the nearestcandidate edgel in the first image is in close proximity tothe original point, then the original mapping is declaredto be valid. If the two points are not in close proximity,then the original point is eradicated from further analy-sis.

The choice of threshold for the proximity measure ischosen by estimating the size of the smallest cataractfeature to be tracked. The inverse tracking procedure isprimarily designed to prevent candidate matches betweenedgels from completely unrelated cataract features. Athreshold of 10 pixels was used in the implementationpresented here.

As mentioned above, the edge points which do not sat-isfy the inverse tracking procedure are not analysed fur-ther in the algorithm presented here. In this application,it is more important to have a moderate number of ac-curate match selections than a larger number of grosslyinaccurate match estimates. Qualitatively however, thishas not led to a dramatic reduction in edge data for thefour-line algorithm (see section 8).

As discussed above, the converse problem can occurwhen edgels of similar orientation are declared as candi-date matches when the true correspondence is furtheraway. Inaccurate match estimates due to this prob-lem cannot be avoided using the matching proceduredescribed above. However, the problem reduces as thecataract images are brought further into registration us-ing the multi-scale procedure described in section 7.

Calculating matching strengths

The procedure described above has chosen a single edgelas a candidate match. The inverse tracking techniquehas increased the likelihood that the candidate edgel isdue to part of the correct cataract feature. A simple tech-nique for calculating the match strength directly from theedge data is to analyse the spatial variance of edgels withsimilar orientations surrounding the candidate edgel. Aprincipal axes decomposition can be used to derive theorientation of the match constraint lines. Calculation ofthe second moments of area with respect to each of theseconstraint lines gives a measure of the spread or varianceabout the candidate edgel. A low value for both momentareas indicates point motion. A high value in one direc-tion with a low value in the other indicates edge motion.

These second moments of area can be used in heuristicformulae to return matching strengths for the four-lineprocedure.

Neighbour selection

The second part of the four-line procedure requires theselection of neighbour motion estimates. Analysing thesurrounding eight pixels will usually return only two

255

neighbour motion estimates from adjacent edgels withina linked collection of edgels. Choosing neighbour esti-mates by this technique prevents constraint propagationto other linked collections of edgels. A simple selectiontechnique which allows such propagation is the choice ofthe nearest neighbour motion estimate in each of the sur-rounding four quadrants.

Estimating registration vectorsIt was mentioned in section 6 that image registration andtrue cataract motion should not be separated. Therefore,once an estimate of the velocity field has been calculatedusing the four-line algorithm at any scale, it must alwaysbe described by rotational and translational registrationvectors. A current solution is to rotate the velocity fieldsynthetically until the variance of the new velocity fieldis minimised. The remaining velocity field is averagedand used as an estimate of the translation vector. Horn10

describes similar techniques for calculating rotational andtranslational components of a vector field.

7. SUMMARY OF PROCEDUREThe Canny algorithm is used to produce edge data ata large range of scales from a pair of successive retro-illumination images. Automatic lens/iris segmentationis then performed by separate algorithms not describedhere.

Beginning with the lowest scale, the four-line proceduredescribed in the previous sections is used to estimate theflow-field between the two images. Translational and ro-tational vectors are then recovered and used to bring allthe edge data at every scale into closer registration. Thisprocedure is repeated until the resultant translation andregistration vectors are negligible.

The next highest scale is now selected and the completeprocedure described above is repeated for all subsequentscales. The residual flow-field calculated at the highestscale is deemed to be due to cataract motion within theeye lens.

8. RESULTSTwo image sequences are presented. The first set of re-sults is from retro-illumination images taken during thesame day (Figure 2). Subsets of complete images havebeen taken for illustrative purposes only. Changes dueto experimental variation discussed in section 3 will stilloccur, however there should be no motion due to cataractchange.

Figure 3 shows superimposed Canny edge data at a lowscale (<r = 12) from both images. Figure 4 shows the in-verse tracking match vectors described in section 6. Thedark points indicate edge points with a valid candidatematch in the second image. The grey points representthe velocity vector end, and the lighter points representthe vectors themselves. Figure 5 shows the result of the

Figure 2: Subsets of images recorded on same day

four-line algorithm. Figure 6 shows the same Canny edgedata superimposed after the images have been broughtinto closer registration using the technique described insection 6. The edge data at this scale are brought com-pletely into registration after the last rotation and trans-lation procedure is performed.

Figure 7 shows superimposed Canny edge data at ahigh scale («r = 4) after the registration procedure hasbeen performed at the lower scale. Figure 8 shows theresult of the four-line algorithm. If the calculated vectorsare compared to figure 7, it can be seen that the calcu-lated motion field is in close correspondence to the actualmotion of well-defined features within the edge data.

The edge data are brought into complete registrationat this higher scale by further rotational and translationalprocedures described in section 6.

Figure 3: Superimposed edge data, a = 12 (lens subsets)

Figure 9 shows two subsets of cataract images takenfrom the same patient at a 6 month interval. Figure 10on the left shows superimposed Canny edge data at a lowscale (IT = 8) from both images after several registrationprocedures. Figure 10 on the right shows the final resultof the four-line algorithm at this scale.

Figure 11 on the left shows superimposed Canny edgedata at a higher scale (<r = 6) after the complete regis-tration procedure has been performed at the lower scale.

256

Figure 4: Match velocity estimates

Figure 8: Calculated velocity field (complete lens)

Figure 5: Calculated velocity field

Figure 6: Superimposed edge data (a = VI) after regis-tration (lens subsets)

:r^ ) /-

Figure 7: Superimposed edge data (cr — 4) after registra-tion at a lower scale (complete lens)

Figure 11 on the right shows the result of the four-line al-gorithm procedure. Qualitatively, the flow-field appearsto be converging in one direction, indicating the motionof cataract regions.

Figure 9: Subsets of images taken at a 6 month interval

CONCLUSIONThe cataract tracking procedure described above has pro-duced encouraging preliminary results on a small numberof image sequences. The procedure based on the four-line technique performs registration and can track well-defined cataract regions at higher scales.

However, the implementation in its present form hasseveral limitations. Firstly, only well-defined cataract re-gions can be tracked successfully. Regions which havegreatly changed shape or size between successive imagesare less likely to be tracked. Secondly, the procedure atpresent does not compensate for the very slight motion

257

• • • • • - ,

Figure 10: Superimposed edge data (a = 8) and calculatedflow-field

l \

Figure 11: Superimposed edge data (<r = 6) and calculatedflow-field

effects due to image foreshortening which occur due tovariations in the eye position. A third limitation of thefour-line algorithm in particular is the use of heuristicformulae in the estimation of weighting parameters. Theparameters chosen in this implementation have limitedtheoretical value.

Notwithstanding this, the results have produced pre-liminary data which were previously unavailable to theclinician.

ACKNOWLEDGEMENTSWe would like to thank Guy Scott for our invaluable con-versations concerning the implementation of his four-linemotion algorithm. We would also like to thank ProfessorJ.M. Brady for his enthusiasm and guidance in this work.Mr. A. J. Bron, Mr. G. A. Shun-Shin, Mr. J. Sparrowand Mr. N. Brown from the Oxford Eye Hospital haveall provided invaluable clinical information.

REFERENCES1. Scott G.L. Local and global interpretation of moving

imagery PhD thesis, University of Sussex, UK (1986)

2. Canny J. "Finding Edges and Lines in Images" MITAI Laboratory Technical Report No. AI-TR-720(1983)

3. Bron A.J., N. Brown, J. Sparrow, G.A. Shun-Shin"Medical Treatment of Cataract" Eye Vol 1 (1987)pp 542-550

4. Kawara, T., H. Obazawa, R. Nakano, M. Sasaki, T.Sakati "Quantitative evaluation of cataractous lensopacities with retro-illumination photography" Jpn.J. Clin. Ophthalmol Vol 33 (1979) pp 21-26

5. Kawara, T., H. Obazawa "A new method for retro-illumination photography of cataractous lens opaci-ties" Am. J. Ophthalmol. Vol 90 (1980) pp 186-189

6. Ullman S. The Interpretation of Visual Motion MITPress, Cambridge, MA, USA (1979)

7. Horn B.K.P, B.G. Schunck "Determining OpticalFlow" Artificial Intelligence Vol 17 (1981) pp 185-203

8. Hildreth, E.C. The measurement of Visual MotionMIT Press, Cambridge, MA, USA (1984)

9. Spacek, L.A. The Detection of Contours and their Vi-sual Motion PhD Dissertation, University of Essex(1985)

10. Horn B.K.P. Robot Vision MIT Press, Cambridge,MA, USA (1986)

258