eyes are the windows of the body: the analysis of corneal ... · need for a definition able to...

Eyes are the windows of the body: the analysis of corneal and retinal images

Alfredo Ruggeri

BioimLab – DEI, University of Padova, Padova, Italy

ICIAR 201815th International Conference on Image Analysis and Recognition

2

Eyes are window to the body

National Geographic on Youtube: https://youtu.be/BPAbANevTqM

3

Biomedical Image Analysis

• Over the years, our group developed several algorithms for the vascular analysis of retinal images.

• From “tools to perform image analysis” to “tools to provide diagnostic information”

• First (naive) attempt was to develop a complete diagnostic system for diabetic/hyper-tensive retinopathies, in cooperation with NidekTechnologies (Japan).

4

RET-H system

5

Biomedical Image Analysis

• Over the years, our group developed several algorithms for the vascular analysis of retinal images.

• From “tools to perform image analysis” to “tools to provide diagnostic information”

• Measurement of single clinical parameters with web-based tools

Image normalization

7

Inter

Intra

Image normalization

Image normalization

L(x,y)(x,y)IC(x,y)I(x,y) o +×=

),(ˆ),(ˆ),(),(ˆ

yxCyxLyxI

yxIo-

=

Acquired image

Contrast Luminosity

Original image

(Foracchia, Grisan, Ruggeri, Med Image Anal, 2005)

The vessel tracking engine

14

Tracking: pre-processing and seed-finding

• Luminance and contrast drifts removed (Foracchia et al, Medical Image Analysis, 2005)

• Equally-spaced 1-pixel rows and columns of the image are analyzed• From each line, the gray-level profile is

extracted• Using LoG filter at different scales,

patterns corresponding to candidate vessels are searched.

15

Tracking: multi-directional graph search

100 120 140 160 180 200 220

40

50

60

70

80

90

100

110

Seed points are identified and then connected by a multi-directional, lowest-cost graph search approach

16

Tracking: results

[Poletti, Fiorin, Grisan, Ruggeri; Proc. WC2009, Munich.Fiorin, Poletti, Grisan, Ruggeri; Proc. WC2009, Munich]

Optic disc identification

18

Geometrical model

• The course of the main vessels originating from OD can be modeled as two parabolas, having a common origin at OD.

( ){ }2, :x y ay xG = =

• Moving away from OD, vessels inside the parabolas bend towards the center of the image, while those outside bend towards the external edges of the image.

19

Geometrical model

÷÷÷÷

ø

ö

çççç

è

æ

-+-

----

+-

--=

=

)()(

||)sgn()(

||)sgn()sgn(arctan

);,(

21 ODOD

ODODOD

OD

ODOD

mod

xxcxxcaxxyyyy

xxasyyxx

pyxJ

On-parabola contribution Off-parabola contribution

20

[ ]{ }å=

-=N

iiimodiimeasi yxyxwRSS

1

2),(),( JJParameter estimation byminimizing cost function:(Simulated Annealing)

),( iimeas yxJModel of vessel direction: Measured vessel direction:

€

ϑ mod (xi,yi, p) = f (xi,yi;xDO,yDO,β)

Geometrical model

(Foracchia, Grisan, Ruggeri; IEEE TMI, 2004)

21

Objective function for xOD yOD estimation

21(Foracchia, Grisan, Ruggeri; IEEE TMI, 2004)

22

OD identification results

23

OD identification results

23

TorTNETGlobal Vessel Tortuosity Estimation

33

Introduction

In many retinopathies, vessel tortuosity is among the

first alterations appearing in the retinal vessel network

Need for a definition able to express in mathematical terms tortuosity

as perceived by retina experts

35

Tortuosity estimation

Proposed tortuosity index t :

When evaluating tortuosity of a line, human experts integrate information on:1. how many times a line changes curvature sign2. how large is the amplitude of each turn curve (the curve segment between

two changes in curvature sign)

(Grisan, Foracchia, Ruggeri; IEEE TMI, 2008)

38

1.05

3.87

6.86

9.15

12.78

10.13

Vessel tortuosity estimation

40

• Ground-truth derived as average on the 3 graders’ orderings.

New tortuosity estimation

Twist-based tortuosity:

Angle-based tortuosity:Based on difference between direction angles of adjacent samples

Based on arc-to-chord length ratio, for every twist(Grisan et al., IEEE TMI, 2008)

• 20 images (10 normal, 6 pre plus, 4 plus), 640x480 pxls 130° FOV, acquired with RetCam (Clarity Medical Systems, USA).

• Sorted by increasing tortuosity by 3 clinical graders and 3 ROP experts using TorTsorT(http://bioimlab.dei.unipd.it/TorTsorT.html)

41

IMAGE-LEVEL tortuosity 1

FinalIMAGE-

TORTUOSITY

VESSEL - LEVEL tortuosity 1

VESSEL-LEVEL tortuosity 8



aggregation ‶1

‶2

‶8

∑






aggregation

VESSEL -LEVEL tortuosity 2

Image-level tortuosity

42å= vesselall nn

tortuosityvesseltortuosityimage 1=n



FinalIMAGE-

TORTUOSITY





aggregation ‶1

‶2

‶8

∑






aggregation


5=n


43



FinalIMAGE-

TORTUOSITY





aggregation ‶1

‶2

‶8

∑






aggregation


FinalIMAGE-

TORTUOSITY

‶1

‶2

‶8

∑

Combination coefficients determined by REGRESSION on ground-truth ordering


44

Image Tortuosity = 3.5



Results: examples

45

CORNEAAnterior Chamber

Corneal image analysis

Anatomy of the cornea

46

Microscopic anatomy of cornea

OutsideInside

EPITHELIUMSTROMAENDOTHELIUM

Descemet’smembrane

Bowman’s membrane

47

• Corneal confocal microscopy allows the rapid and non-invasive acquisition of images of all corneal layers.

• Images are acquired with ConfoScan4 (Nidek Technologies, Italy) or HRTII-RCM (Heidelberg Engineering, Germany).

Confocal microscopy

Endothelial cells

Stromal keratocytes

Sub-basal nerve fibers

Epithelial cells

+

48

• Corneal specular microscopy allows acquisition of images only of the most reflective corneal layer, the endothelium.

• Konan, Tomey, Topcon are among the most common microscopes.

Specular microscopy

Endothelial cells

49

Automatic recognition and features measurement of

sub-basal nerve fibers

50

Analysis of subbasal layer images

OutsideInside

EpitheliumStromaEndothelium

Endothelial cells

Stromal keratocytes


Epithelial cells

51

Motivation

• Confocal microscopy allows the rapid and non-invasive acquisition of images of sub-basal nerve plexus (SNP) fibers.

• Their analysis appears to be an important clinical issue:• Malik Diabetologia 2003 (diabetes)• Midena J Refract Surg. 2006 (diabetes)• Tavakoli J Diabetes Sci Technol 2013 (diabetes)

• Calvillo, IOVS 2004 (LASIK)• Moilanen, IOVS 2003 (PRK)• Patel S., IOVS 2002 (contact lens)

• Patel D., IOVS 2005 (keratoconus)• Benitez del Castillo, IOVS 2004 (dry eyes)

• Rosenberg, Cornea 2002 (herpes keratitis)

• To provide useful information, quantitative morphometry should be available based on nerve tracing.

• To avoid long manual analysis, automated nerve tracing and quantification is required.

52

A first technique for corneal nerves tracing

• Corneal basal layer images are acquired and their luminosity and contrast are normalized.

• Seed points are detected and used as starting points for the nerve tracing procedure(based on our work on retinal vessel tracing).

• Nerve segments are recognized.

• Post-processing to:- remove false positive recognitions- increase sensitivity.

59

Some results

(Scarpa et al, Invest Ophthalmol Vis Sci, 2008)

60

N=90 images, with Nidek Technologies ConfoScan-4

Results


61

Nerve tortuosity

N=30 images, Heidelberg Engineering HRTII-RCM


r=0.783

62

New nerve tracing

• Images are first enhanced for uneven illumination and contrastby top-hat filtering.

• To enhance the corneal nerves, the corrected images are filtered with a bank of log-Gabor filters.

• Thresholding is applied to recognize all “elongated white structures”, which include corneal nerves.

• Recognition of true nerves is carried out with a Support Vector Machine classifier.

63

New nerve tracing

SENSITIVITY FALSE DETECTION RATEmean sd mean sd

Automatic Tracing 0.86 0.07 0.08 0.07Second Observer 0.92 0.05 0.08 0.05

• Nerve tracings were compared with ground-truth (first observer).

• 246 images from healthy subjects acquired with the Heidelberg HRTII-RCM. Nerve traced by two expert observers.

• 50 images used for training and optimization, 196 images for validation.

(Guimarães et al, Trans Vis Sci Tech, 2016)

64

New nerve tracing

• From several minutes to less than one second per image.

Correlation: 0.93

Density


65

New nerve tracing tortuosity

Tortuosity = 1.7 Tortuosity = 3.9 Tortuosity = 7.5

Tortuosity = 12.9 Tortuosity = 17.2

66

New nerve tracing

• From several minutes to less than one second per image.

Accuracy: 28/30Correlation: 0.93

Density Tortuosity


68

Automatic mosaicking of sub-basal nerve fibers images

69

Image mosaic

Architecture of the SNP on healthy[1] and keratoconus[2] subjects.

Mosaic manually built from ~500 images.Hugely time-consuming process (from 10–20 hours)

Patel DV, McGhee CNJ. Mapping of the normal human corneal sub-basal nerve plexus by in vivo laser scanning confocal microscopy. IOVS, 2005.Patel DV, McGhee CNJ. Mapping the corneal sub-basal nerve plexus in keratoconus by in vivo laser scanning confocal microscopy. IOVS, 2005.

72

Image mosaic

An algorithm for image mosaic / registration.

Problem: 100+ images in random order; how can I find the adjacent ones to register?

First step:Find the registration parameters between each pair of images

73

Mosaicking process

(xt, yt, r, score)

For each image find the closest other image using a “similarity score”

74

Mosaicking process

Build the best ordered sequence of pairs of images to be merged (MinimumSpanning Tree)

(xt, yt, r, score)

75

Mosaicking process

Move along the sequence merging the pairs of images (with rotation, translation and affine transform).

81

Image blending

82

Image blending

85Run-time 46 seconds.

…. back to nerve tracing

86

…. back to nerve tracing

Run-time 32 seconds.

87

Automatic analysis of endothelial cells

88

Analysis of endothelium images

OutsideInside

EpitheliumStromaEndothelium

Endothelial cells

Stromal keratocytes


Epithelial cells

89

• Normal corneal endothelium is a single layer of uniformly sized cells with a predominant hexagonal shape. This regular tessellation is affected by age and pathologies.

• Segmentation of a large number of endothelial cells required for a reliable estimation of clinical morphometric parameters:

endothelial cell density (ECD) = total cell area / nr of cells

pleomorphism = % of hexagonal cells polymegethism = fractional SD of cell areas

(last one requires contour detection)

• To avoid long manual analysis, automated cells contour segmentation is required.

Motivation

90

Cell contour recognition

91


Human visual processing is very powerful and complex …

Kanisza triangles Kanisza square

(Gaetano Kanisza, 1913-1993, psychologist and artist, University of Trieste)

92

Cell contours appear nice and clearon a broad view….

Human visual processing is very powerful and complex …

… but local gray-scale values do not give all the information necessary to identify all cell contours.

false contoursmissed contours


93

1. Gray-scale information

Þ contour extraction

2. Shape information

Þ contour completion and fixing

Two levels of information

94

Contour extraction: Artificial Neural Network with weight-filters arrays

Good as a �proof of concept�, but not usable in the clinicalroutine yet.


95

1. Gray-scale information

Þ contour extraction

2. Shape information

Þ contour completion and correction

Two levels of information

96

Connected boundariescorrect

not correct

Boundaries determined by the ANN based only on gray-scale values

Contour completion: connection of floating facing boundaries


97

False ContoursDetected by the excessively small sizeof the cell bodies

Missed ContourDetected by the excessively large sizeexcessively large aspect ratioof the cell body

Contour correction (from shape information)


98

missed contour

false contours

Back to floating boundaries connection

Contour correction (from shape information)


108

Nidek Technologies NAVIS-ENDO system

• The ENDO software is a module of the system for ophthalmology.

109

A little toy …

(Foracchia Ruggeri, EMBC, 2003)

110

A bigger toy …

(Foracchia Ruggeri, EMBC, 2007)

112

A new project for segmentation of cells contour• The contours of cells detected by a genetic algorithm.• A small set of vertices (individuals) forming regular hexagons is the

starting population. • At each step, the location of each vertex is randomly modified,

evolving into polygons with possibly different number and positions of vertexes.

• Each vertex is evaluated by considering both its correspondence with the actual image (pixels intensity) and the regularity of the resulting polygons.

115

Some results

normal subject subject with high polymegethism

subject with low ECD

116

Some results

normal subject subject with high polymegethism

subject with low ECD

ECD (cells/mm2) 2999

Pleomorphism (%) 83,3

Polymegethism (%) 24,4

ECD (cells/mm2) 2151



ECD (cells/mm2) 970



117

Results

Automated Manual 1 Manual 2 abs diff % abs diff %ECD cells/mm2 cells/mm2 cells/mm2 Auto vs M1 M1 vs M2

Mean 2562 2572 2575 0.60 % 0.46 %Sd 827 828 833 0.51 % 0.60 %

Min 457 468 465 0.09 % 0.00 %Max 3627 3653 3659 2.35 % 3.41 %

Pleomorphism % % %Mean 58.51 58.16 58.15 3.11 % 2.60 %

Sd 10.06 10.02 10.45 3.33 % 3.62 %Min 42.30 41.20 38.20 0.00 % 0.00 %Max 83.30 83.30 83.30 12.23 % 13.05 %

Polymegethism % % %Mean 34.85 36.68 37.35 5.33 % 2.89 %

Sd 5.92 6.54 6.65 2.94 % 1.67 %min 20.80 21.70 22.30 0.27 % 0.30 %max 46.30 49.00 50.40 11.82 % 6.67 %

• 30 images acquired with a Topcon SP3000 specular microscope.• Differences between automated and two manual assessments.

(Scarpa Ruggeri, Cornea, 2016)

118

The current team at BioImLab UniPD

http://bioimlab.dei.unipd.it

Marco ForacchiaEnrico GrisanMassimo De LucaLara TramontanDiego Fiorin

Alumni: Pedro GuimāraesJeff WigdahlEnea Poletti

119

A warning from the past …

“Things which we see are not by themselveswhat we see … It remains completely

unknown to us what the objects may be by themselves ... We know nothing but our manner of

perceiving them ...”

Immanuel Kant

eyes are the windows of the body: the analysis of corneal ... · need for a definition able to...

Documents