Automated Kidney Segmentation In 3D Ultrasound Imagery,

and its Application in Computer-assisted Trauma Diagnosis

Final PhD Oral Examination - Presentation SlidesSeptember 1st , 2016

Mahdi Marsousi

Supervisor: Prof. Konstantinos N. Plataniotis

Introduction

Abdominal trauma: injuries to abdomen, either blunt or penetrating, resulting

in severe blood loss.

Importance: massive internal bleeding quickly threatens patient’s life.

Cure: rapid diagnosis and surgery

[1]

[1] http://www.radiologyassistant.nl/en/p466181ff61073/acute-abdomen-role-of-ct-in-trauma.html

[2] http://www.wikidoc.org/index.php/Abdominal_trauma

[2]

Slide #1

Diagnosis by 3D ultrasound (3DUS) For hemodynamically unstable patients, ultrasound imagery is the preferred

imaging modality, because it is portable.

This is called: Focused Assessment with Sonography for Trauma “FAST”.

2D ultrasound is the popular diagnostic tool.

3DUS imagery is key to design computer-assisted diagnosis, because:

a) detecting and localizing internal organs in 3D space is possible by 3DUS;

b) 3DUS facilitates measuring the volume of internal bleeding.

Slide #2

:Focus of this presentation

FAST exam Six major abdominal views associated with FAST exam.

For computer-assisted trauma diagnosis, we focus on the

Morison’s pouch view (2),

• Because it is the most sensitive view to abdominal bleeding.

• An abdominal bleeding usually places between the right kidney and liver [1].

• Morison’s pouch view shows the right kidney and a portion of liver [1].

• The kidney has a unique shape among internal organs.

• By segmenting the kidney shape, we only look for an abdominal bleeding around

the top surface of the kidney shape.

[1] Heller et al. Ultrasound use in trauma: the fast exam. Academic Emergency Medicine, 14(6):525–525, 2007.

[2] Ingeman et al. Emergency physician use of ultrasonography in blunt abdominal trauma. Academic Emergency

Medicine, 3(10):931–937, 1996.

Slide #3

Motivation

Problem definition:

• Because of limited surgical resources in massive causalities, only true trauma patients should be identified and transported for rapid surgery.

• This importance necessitates the use of paramedics to conduct FAST exam.

• However, paramedics are not capable to do FAST exam with supervision.

Solution: a computer-assisted tool can be used to help paramedics to conduct

trauma diagnosis.

Slide #4

Contributions

Slide #5

3DUS dataset[1]

Slide #6

Training set: 6 with-kidney and 6 without-kidney images.

Evaluation set: 15 with-kidney and 15 without-kidney from healthy volunteers,

and 8 with-kidney images from abnormal patients.

Images of abnormal patients represent morphological changes to the kidney shape.

With-kidney image:

taken from Morison’s pouch view

Without-kidney image:

taken from other views.

[1] G. Sakas, Phase I: Ultrasound Imaging Data Collection and Applications," MedCom GmbH,

Tech. Rep., September 2012.

Contribution 1:Kidney detection

Overview on kidney detection Kidney detection: searching within 3D image domain to answer:

• whether kidney shape exists in a 3DUS image?

• if yes, what is its alignment?

State-of-the-art:

Proposed solutions:

Name Methodology Advantage Disadvantage

Noll et al. [1]

- applies volume enhancement - uses redial rays to find kidney shape mass-center

robust against speckle & low-contrast intensity

may wrongly detect other structures instead of kidney

[1] Noll et al. Automated kidney detection and segmentation in 3d ultrasound, in Clinical Image-Based Proc. Translational Research in Medical Imag. Springer, 2014, pp. 83–90.

Slide #7

Definition: mathematical representation of kidney shape to add shape prior in kidney detection

Proposed shape modeling: complex-valued implicit shape model (CVISM)

Ψ Ԧ𝑋 = 𝜓𝑃𝑆Ԧ𝑋 + 𝜓𝐾𝐶

Ԧ𝑋 − 𝑖 ⋅ 𝜓𝑅𝑀Ԧ𝑋 ,

𝜓𝑝𝑠, 𝜓𝐾𝐶, and 𝜓𝑅𝑀 are real-positive functions, defining voxel’s membership to

Pyelocalyceal System, Kidney Capsule, and Renal Medulla, respectively.

Generating regions: using ground truth data of training set of with-kidney images.

Kidney shape model

Slide #8

Kidney

Capsule

Renal

Medulla

Pyelocalyceal

System

Real values are used to

model bright regions:

𝐾𝐶: Kidney Capsule

𝑃𝑆: Pyelocalyceal System

Imaginary values are used

to model dark region:

𝑅𝑀: Renal Medulla

Objective: registering kidney CVISM on 3DUS images, to detect kidney shape and estimate its alignment.

Challenges:

• Partial occlusion of kidney shape

• Non-kidney structures with similar appearance to the kidney shape

Components:

• Global deformation,

• Similarity metric,

• optimization algorithm.

Global deformation: similarity transformation (shape-preserving)

Ԧ𝑌 = 𝑆𝑇 Ԧ𝑝𝑠𝑡,1Ԧ𝑋 + Ԧ𝑝𝑠𝑡,2

Shape-to-volume registration

Slide #9

Orientations & scaling

Ԧ𝑝𝑠𝑡,1 = 𝜃𝑥,𝜃𝑦,𝜃𝑧,𝑠𝑇

Translations

Ԧ𝑝𝑠𝑡,2 = 𝑡𝑥,𝑡𝑦, 𝑡𝑧𝑇

CVISM domain Image domain

Proposed similarity metric: Regularized complex normalized cross-correlation (RCNCC)

Formulation:

Γ Ԧ𝑝𝑠𝑡,1, Ԧ𝑝𝑠𝑡,2∗ = max

Ԧ𝑝𝑠𝑡,2∗ ∈ΩV

𝑠𝑢𝑏

1

Λ Ԧ𝑝𝑠𝑡,1, Ԧ𝑝𝑠𝑡,2⋅max 0,ℜ Σ𝐼 Ԧ𝑝𝑠𝑡,1, Ԧ𝑝𝑠𝑡,2

Σ𝐼𝐼ℜ Ԧ𝑝𝑠𝑡,1

⋅max 0,ℑ Σ𝐼 Ԧ𝑝, Ԧ𝑋,𝑉

Σ𝐼𝐼ℑ Ԧ𝑝𝑠𝑡,1

Λ Ԧ𝑝𝑠𝑡,1, Ԧ𝑝𝑠𝑡,2−1

increases as kidney CVISM aligns out of the ultrasound field of view.

RCNCC similarity metric

Regularization Term

Checking similarity of image data with Kidney Capsule and

Pyelocalyceal System

Checking similarity of image data with Renal Medulla

Robustness againstPartial kidney shape occlusion

Considering multi-regional structure of the kidney shape,

max 𝑎, 𝑏 : selects maximum of 𝑎 and 𝑏. “max”s are used to avoid multiplication of two negative values in RCNCC. ℜ{𝑎} and ℑ{𝑏} extract real part of 𝑎 and imaginary part of 𝑏.

Slide #10

Registration problem Objective: Finding Ԧ𝑝𝑠𝑡,1

∗ and Ԧ𝑝𝑠𝑡,2∗ which provides a maximum RCNCC for input image.

Optimization problem:

Γ∗, Ԧ𝑝𝑠𝑡,1∗ , Ԧ𝑝𝑠𝑡,2

∗ = maxԦ𝑝𝑠𝑡,1

Γ Ԧ𝑝𝑠𝑡,1, Ԧ𝑝𝑠𝑡,2∗

Optimization algorithm:

• initialization: searching for best seed point from a

set of seed points to initialize registration.

• iterative improvement: iteratively updating

registration parameters using Gradient Descent.

Figure shows RCNCC metric versus 𝜃𝑥 and 𝜃𝑧.Yellow regions correspond to desirable registration solutions. The gray stars are seed point.Red star is selected seed point.Black arrows show iterations of Gradient descent.

Slide #11

Processing pipeline of kidney detection

• Γ𝑡ℎ is threshold on similarity metric to decide whether the kidney shape exists or not.

• It is obtained using training set, as Γ𝑡ℎ = 3.5.

Slide #12

Experimentation Objective: to evaluate shape-based and atlas-based kidney detection, compared to Noll et al. [1].

Dataset:

• 15 with-kidney and 15 without-kidney images of healthy volunteers

• 8 with-kidney images abnormal patients

Comparison metrics:

• accuracy: 𝐴𝑐𝑐𝐾𝐷 = 100% ⋅𝑁𝑇𝑃+𝑁𝑇𝑁

𝑁𝑇𝑃+𝑁𝐹𝑁+𝑁𝑇𝑁+𝑁𝐹𝑃,

• sensitivity: 𝑆𝑒𝑛𝑠𝐾𝐷 =NTP

𝑁𝑇𝑃+𝑁𝐹𝑁,

• specificity: 𝑆𝑝𝑒𝑐𝐾𝐷 =𝑁𝑇𝑁

𝑁𝑇𝑁+𝑁𝐹𝑃.

𝑁𝑇𝑃, 𝑁𝑇𝑁, 𝑁𝐹𝑃, and 𝑁𝐹𝑁 are numbers of true positive, true negative, false positive, false negative detections.

[1] Noll et al. Automated kidney detection and segmentation in 3d ultrasound, in Clinical Image-Based

Proc. Translational Research in Medical Imag. Springer, 2014, pp. 83–90.

Slide #13

Kidney detection accuracy Using with-kidney and without-kidney images of healthy volunteers:

• Shape-based and atlas-based provide higher accuracy, compared to Noll et al., because:

1) higher specificity: involving multi-structural regions of kidney shape,

2) higher sensitivity: regularization factor provides robustness against kidney shape’s occlusion.

• Atlas-based shows higher accuracy than shape-based because of using texture information.

Using actual ultrasound volumes of abnormal patients:

Accuracy is reduced, because:

different device settings from healthy volunteers,

Morphological changes of RUQ view in abnormal patients.

Method 𝑵𝑻𝑷 𝑵𝑻𝑵 𝑵𝑭𝑷 𝑵𝑭𝑵 Accuracy (%) Sensitivity Specificity

Noll et al. 9 11 4 6 66.67 0.60 0.73

Shape-based kidney detection 14 12 3 1 86.67 0.93 0.80

Atlas-based kidney detection 15 13 2 0 93.33 1 0.87

Slide #14

Method 𝑵𝑻𝑷 𝑵𝑭𝑵 Accuracy (%)

Noll et al. [1] 5 3 62.5

Shape-based 5 3 62.5

Atlas-based 6 2 75.0

Examples of kidney detection

Blue region: Renal Medulla

Red region: Kidney Capsule/Pyelocalyceal System

Slide #15

Contribution 2:Automated kidney segmentation

Overview on automated kidney segmentation

Problem definition: automatically segmenting the kidney shape in 3DUS images.

State-of-the-art:

Proposed method: complex-valued regional level-set with shape prior (CVRLS-SP)

• Complex-valued structure is used to add multi-regional structure into segmentation.

• Adding multi-regional structure is important because it improves segmentation specificity.

Name Methodology Advantage Disadvantage

Noll et al. [1]

Radial ray trace + fast marching + edge-based level-set

- Robust against speckle and low-contrastintensity profile

- Automated initialization

- Not robust against kidney deformation- High-computational cost

MRF-AC[2]

2D active contour + Markov random field (MRF) + 3D reconstruction

- Low computational cost - Manual initialization- Discontinuity along z-axis

[1] Noll et al. Automated kidney detection and segmentation in 3d ultrasound, in Clinical Image-Based Proc.

Translational Research in Medical Imag. Springer, 2014, pp. 83–90.

[2] Martn-Fernndez et al., An approach for contour detection of human kidneys from ultrasound images using markov random fields and active contours, Med. Image Anal., 1–23, 2005.

Slide #16

CVRLS-SP representation

Mathematical representation of regions in CVRLS-SP:

𝑏𝑟 : 𝑋 ∈ 𝑝𝑦𝑒𝑙𝑜𝑐𝑎𝑙𝑦𝑐𝑒𝑎𝑙 𝑠𝑦𝑠𝑡𝑒𝑚 𝑜𝑟 𝑘𝑖𝑑𝑛𝑒𝑦 𝑐𝑎𝑝𝑠𝑢𝑙𝑒, 𝑖𝑓 ℜ 𝜙 𝑋 > 0,

𝑑𝑟 : 𝑋 ∈ 𝑟𝑒𝑛𝑎𝑙 𝑚𝑒𝑑𝑢𝑙𝑙𝑎, 𝑖𝑓 ℑ 𝜙 𝑋 > 0,

𝑏𝑔 : 𝑋 ∈ 𝑏𝑎𝑐𝑘𝑔𝑟𝑜𝑢𝑛𝑑, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒.

where 𝜙 is level-set function.

Prior shape: (𝜙𝑠) is generated by aligning CVISM on detected kidney shape:

𝜙𝑠 𝑋 = 2 Ψ 𝑆𝑇 Ԧ𝑝𝑠𝑡,1∗ 𝑋 + Ԧ𝑝𝑠𝑡,2

∗ > 0 − 1 + 𝑖 , ∀𝑋 ∈ Ω𝑉

where 𝑆𝑇 Ԧ𝑝𝑠𝑡,1∗ 𝑋 + Ԧ𝑝𝑠𝑡,2

∗ places kidney CVISM, Ψ, on detected kidney shape.

level-set function, 𝜙, is initialized by prior shape: 𝜙 𝑋; 𝑡 = 0 = 𝜙𝑠 𝑋 , 𝑡 is iteration.

Slide #17

CVRLS-SP evolution

[1] Chan and Zhu. Level set based shape prior segmentation. IEEE CVPR, vol 2, pp 1164–1170, 2005..

Slide #18

Conventional regional level-set with shape prior defines energy function as [1],

𝐹 𝑐, 𝜙, 𝜙𝑠 𝐴𝑇 Ԧ𝑝𝑎𝑓, 𝑉 = 𝜆 ⋅ 𝐹𝑆𝑃 𝜙,𝜙𝑠 𝐴𝑇 Ԧ𝑝𝑎𝑓

+ 𝜇 ⋅ 𝐹𝑖𝑛𝑡 𝜙 + 𝛾 ⋅ 𝐹𝑒𝑥𝑡(𝑐, 𝜙, 𝑉)

𝐹𝑆𝑃 𝜙,𝜙𝑠 𝐴𝑇 Ԧ𝑝𝑎𝑓: shape prior energy,

𝐹𝑖𝑛𝑡 𝜙 : internal energy, controlling smoothness.

𝐹𝑒𝑥𝑡(𝑐, 𝜙, 𝑉): external energy, pushing level-set toward region of interest.

𝑐: average intensity level of region of interest, 𝐴𝑇 Ԧ𝑝𝑎𝑓: affine transformation.

𝜆, 𝜇, 𝛾: Lagrange multipliers

Contribution: CVRLS-SP’s energy functional is defined as follows,

𝐹 𝑐𝑏𝑟 , 𝑐𝑑𝑟 , 𝜙, 𝜙𝑠 𝐴𝑇 Ԧ𝑝𝑎𝑓, 𝑉 = 𝜆 ⋅ 𝐹𝑆𝑃

𝑏𝑟 ℜ{𝜙},ℜ{𝜙𝑠 𝐴𝑇 Ԧ𝑝𝑎𝑓} + 𝜆 ⋅ 𝐹𝑆𝑃

𝑑𝑟 ℑ{𝜙}, ℑ{𝜙𝑠 𝐴𝑇 Ԧ𝑝𝑎𝑓}

+𝜇 ⋅ 𝐹𝑖𝑛𝑡 𝜙 + 𝛾 ⋅ 𝐹𝑒𝑥𝑡𝑏𝑟 (𝑐𝑏𝑟 , ℜ{𝜙}, 𝑉) + 𝛾 ⋅ 𝐹𝑒𝑥𝑡

𝑑𝑟 (𝑐𝑑𝑟 , ℑ{𝜙}, 𝑉)

• Multi-regional segmentation is added By dividing both FSP and Fext into br and dr terms.

CVRLS-SP segmentation procedure

Initialization: Creating prior shape and level-set initialization

Iterative evolution:

𝜙 is updated by reducing 𝐹 𝑐𝑏𝑟 , 𝑐𝑑𝑟 , 𝜙, 𝜙𝑠 𝐴𝑇 Ԧ𝑝𝑎𝑓, 𝑉 , using Euler-Lagrange equation.

convergence criteria: ΣΔ𝜙 𝑡 = ∭ 𝜙 𝑋, 𝑡 − 𝜙 𝑋, 𝑡 − 12𝑑𝑥𝑑𝑦𝑑𝑧

Slide #19

Experimentation Objective: to evaluate proposed kidney segmentation compared to Noll et al. [1] and MRF-AC. [2]

Dataset: evaluation set of actual ultrasound volumes

Comparison metrics:

• Dice’s similarity coefficient (DSC)=2𝑇𝑃

2𝑇𝑃+𝐹𝑁+𝐹𝑃,

• Accuracy metric (ACC)=𝑇𝑃+𝑇𝑁

𝑇𝑃+𝑇𝑁+𝐹𝑃+𝐹𝑁× 100% ,

• Sensitivity measure (SENS)=𝑇𝑃

𝑇𝑃+𝐹𝑁,

• Specificity measure (SPEC)=𝑇𝑁

𝑇𝑁+𝐹𝑃.

Parameter setting:

[1] Noll et al. Automated kidney detection and segmentation in 3d ultrasound, in Clinical Image-Based

Proc. Translational Research in Medical Imaging, Springer, 2014, pp. 83–90.

[2] Martin-Fernandez et al., An approach for contour detection of human kidneys from ultrasound images

using Markov random fields and active contours, Medical Image Analysis, 1–23, 2005.

Slide #20

Name 𝜆 𝜇 𝛾 𝑁𝑖𝑡𝑟 𝑡𝑚𝑎𝑥 𝜖

Value 1 0.1 0.05 5 50 10

Specificity of CVRLS-SP is higher because:

• capability of segmenting multi-regional structure,

• using shape prior.

Box-plot: shows statistical variation of DSC of the methods:

Kidney segmentation accuracy

Method𝑫𝑺𝑪 𝑨𝑪𝑪 𝑺𝑬𝑵𝑺 𝑺𝑷𝑬𝑪

𝜇 𝜎 𝜇 𝜎 𝜇 𝜎 𝜇 𝜎

CVRLS-SP 0.8143 0.0408 97.48 0.72 0.7863 0.0814 0.9890 0.0060

Noll et al. 0.4207 0.0795 85.48 2.77 0.7024 0.0912 0.8678 0.0326

MRF-AC 0.5921 0.1457 93.10 2.74 0.6947 0.2132 0.9490 0.0246

𝜇: mean, 𝜎:standard deviation

Slide #21

CVRLS-SP MRFAC Noll et al.

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Dic

e's

Coeffic

ient(D

SC

)

Examples of kidney segmentation

Fourth column:Yellow, red, and green colors are TP, FP, and FN regions. Slide #22

Original image Detected kidney Segmented

kidney regions

Comparing

automated segmentation

vs. ground-truth

3D view

Second and third columns:

Blue and red regions indicate Renal Medulla and

Pyelocalyceal System/Kidney Capsule, respectively.

Conclusions & Future Work

Conclusions:

• We discussed the importance of automated kidney segmentation in designing a computer-

assisted trauma diagnosis.

• Shape-based and atlas-based kidney detection methods were introduced.

• Automated kidney segmentation was introduced

• The accuracy of kidney detection and segmentation methods were evaluated and compared to

state-of-the-art

• Experimental results validate the superiority of the proposed methods.

Future work:

• Investigating the effect of motion artifact and interference using a bigger dataset

• Using the automated kidney segmentation to automatically detect free fluids.

Slide #23

Publications Journal Papers:

[1] M. Marsousi, K. Plataniotis, and S. Stergiopoulos, “An Automated Approach for Kidney Segmentation in Three-Dimensional Ultrasound Images”, IEEE Journal of Biomedical and Health Informatics, DOI: 10.1109/JBHI.2016.2580040, (Date of publication: June 13th 2016)

[2] M. Marsousi, K. Plataniotis, and S. Stergiopoulos, “Computer-Assisted 3D Ultrasound Probe Placement for Emergency Healthcare Applications”, IEEE Transactions on Industrial Informatics, DOI: 10.1109/TII.2016.2569522, (Date of publication: May 18th 2016)

[3] M. Marsousi, K. Plataniotis, and S. Stergiopoulos, “Kidney Detection in 3D Ultrasound Imagery Based on Shape and Texture Priors”, to be submitted at IEEE Transaction on Biomedical Engineering (T-BME) on September 2016.

Conference Papers:

[4] M. Marsousi, X. Lee, and K. Plataniotis, “Shape-included Label-Consistent Discriminative Dictionary Learning: An Approach to Detect and Segment Multi-Class Objects in Image”, IEEE international conference on Image Processing (ICIP), 2016, accepted.

[5] M. Marsousi, K. Plataniotis, and S. Stergiopoulos. “Atlas-based segmentation of abdominal organs in 3D ultrasound, and its application in automated kidney segmentation.”, IEEE International conference in Engineering in Medicine and Biology Society (EMBC), pp. 2001-2005, 2015.

[6] M. Marsousi and K. Plataniotis, “Binomial classification based on DLENE features in sparse representation: Application in kidney detection in 3D ultrasound.” IEEE International Conference in Acoustics, Speech and Signal Processing (ICASSP), pp. 1017-1021, 2015.

[7] M. Marsousi, K. Plataniotis, and S. Stergiopoulos, “Shape-based kidney detection and segmentation in three-dimensional abdominal ultrasound images”. IEEE International conference in Engineering in Medicine and Biology Society (EMBC), pp. 2890-2894, 2014.

[8] M. Marsousi, K. Plataniotis, and S. Stergiopoulos. “A multi-steps segmentation approach for 3D ultrasound images using the combination of 3D-Snake and Level-Set.” IEEE International Conference in Digital Signal Processing (DSP), pp. 1-4, 2013.

US patent:

[9] S. Stergiopoulos, P. Shek, K. Plataniotis, and M. Marsousi. “Computer aided diagnosis for detecting abdominal bleeding with 3D ultrasound imaging.” U.S. Patent Application 14/159,744, filed January 21, 2014.