General Approaches of Histopathological Image

Classification using Convolutional Neural

Networks

Jerome Cheng M.D.

Disclosures

none

Outline

Background on Convolutional Neural Networksdescriptionapplicationslayers

Image classification strategiestrain from scratch transfer learning

ToolsOrangeKeras

Convolutional Neural Networks

class of deep, feed forward artificial neural networks usually composed of convolution, pooling, fully connected layers

most commonly applied in pathology for image classification and segmentation tasks

Inspired by Dr. Hubel and Dr.Wiesel’s work on the cat visual system in the 50’s and 60’s that showed a hierarchical arrangement of neurons and some neurons were excited by lines in a particular orientation in a specific location

Artificial Neural Network Example

a1

a3

a2 𝚺 Output

w1

w2

w3

Rel

u

Activation

Function

+Bia

s

a1w1+a2w2+a3w3 + bias → Relu → Output

Convolutional

Layer

Convolutional

Layer

Convolutional

Layer

Pooling Layer

Pooling Layer

Fully Connected

Layer

Fully Connected Layer

Acceleration of CNN Progress and Adaptation

Faster PCs and GPUs

Publicly available image datasets

Open source CNN deep learning frameworks

ImageNetOnline database containing more than 14 million images belonging to 20,000 categories

Large training dataset is important in deep learning

- Several pre-trained CNN models where trained on ImageNet

Accelerated the progress of CNN research

Hosted yearly competitions

Many publicly available pre-trained CNN models were trained on ImageNet

http://www.image-net.org/

ImageNet - Images

Layers

Convolutional

Pooling

Fully connected

Activation layers

Dropout layer

Regularization layers

Low level

features

High level

features

Convolutional Neural Network Layers

Training

Loss function - measure the difference between the predicted value and the expected value

Weights are adjusted in the convolution and fully connected (dense) layers to minimize the loss function

Pre-trained CNN architectures

VGG-16

VGG-19

Inception

XCeption

MobileNet

VGG-16 architecture

Simonyan, Karen, and Andrew Zisserman. "Very

deep convolutional networks for large-scale image

recognition." arXivpreprint

Applications of CNN in Pathology

Image Classification

Image Segmentation

Object localization and classification

Image Classification

Dataset Source: Levenson RM, Krupinski EA, Navarro VM, Wasserman EA. Pigeons (Columba livia) as trainable observers of

pathology and radiology breast cancer images. PLoSOne 2015;10:e0141357.

Image Segmentation

Ground Truth Prediction

Object Detection and Classification

Hung J, Carpenter A (2017) Applying Faster R-CNN for Object Detection on Malaria Images, IEEE

Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 808–813

Classification Strategies (CNN)

Train from scratch

- Weights from all layers are updated

Transfer learning

- Freeze lower layers - Only weights from some layers are updated

- Combine pre-classification layer output with machine learning algorithms

- None of the weights are updated

Transfer Learning - Freezing Layers

Weights are updated only on selected layers

Information learned from a previous dataset (ImageNet) is preserved

Speeds up training

Transfer Learning - Combine Output with Another Machine Learning Algorithm

The final classification layer of the pre-trained network should not be included

Ex. an image processed through VGG 16 = 4096 features

4096 features + image label can be trained using various machine learning algorithms like Random Forest, logistic regression, and SVM

None of the weight parameters are updated during training

Steps – Classifying Images with CNN

Scan slides

Create and label tiles - ex. 224x224 or 299x299 pixels

Training and Validation set

Augmentation

Choose or Create a Model

Model Training

Prediction

Augmentation

Rotation

Shear

Channel Shift

Scaling

Height shift

Width shift

DemsarJ, CurkT, ErjavecA, GorupC, HocevarT, MilutinovicM, MozinaM, PolajnarM, ToplakM, StaricA, StajdoharM, UmekL, ZagarL, ZbontarJ,

ZitnikM, ZupanB (2013) Orange: Data Mining Toolbox in Python. Journal of Machine Learning Research 14(Aug):2349−2353.

Chollet F. Keras [Internet] San Francisco (CA): GitHub, Inc.; 2017. [cited 2019 Apr 26]. Available from: https://github.com/keras-team/keras.

Orange (Biolab) - https://orange.biolab.si

Data preparation

Create image tiles from the digital slide or image (224 x 224 for VGG 16)

Create a folder with subfolders for each image Class

Ex. Training Set

Cancer

Benign

Orange - loading images

Orange - Image Embedding

Orange - Hardware (image embeddings)

Cluster of 8 machines

6 machines- 2 processors with 4 cores each (Intel Xenon CPU), 32 GB RAM

2 machines - 2 processors with 8 cores (Intel Xeon e5-2660), 157 GB RAM

Embedders are automatically assigned to available CPU cores

Soon: switch to an improved backend with a combination of CPUs and GPUs, with the addition of 3 TitanX GPUs (12 GB ram each)

Orange - Test and Score

Orange - data visualization

Dimensional reduction + scatter plot

Hierarchical clustering

Keras

High level neural network library that works on top of TensorFlow, CNTK, or Theano

Written in Python

CNN models can be built and trained with very little code

Works with CPU and GPUs

Summary

Convolutional Neural Networks can be used to classify and segment histopathological images

Transfer learning applies knowledge learned from a different dataset onto a new image subject matter

Orange (Biolab) is a GUI based machine learning tool that can be used for transfer learning

Keras is a high level framework written in Python for deep learning

References

Chollet, François. Deep Learning with Python. Shelter Island, New York : Manning Publications Co., 2018.

Cheng J. Convolutional neural networks. PathologyOutlines.com website. http://www.pathologyoutlines.com/topic/informaticsconvnet.html. Accessed March 25th, 2019.

Levenson RM, Krupinski EA, Navarro VM, Wasserman EA. Pigeons (Columba livia) as trainable observers of pathology and radiology breast cancer images. PLoSOne 2015;10:e0141357.

Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXivpreprint

Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo,Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis,Jeffrey Ronneberger O., Fischer P., Brox T. (2015) U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab N., Hornegger J., Wells W., Frangi A. (eds) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. MICCAI 2015. Lecture Notes in Computer Science, vol 9351. Springer, ChamDean,Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow,Andrew Harp, Geoffrey Irving, Michael Isard, Rafal Jozefowicz, Yangqing Jia,Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Mike Schuster,Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Jonathon Shlens,Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker,Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas,Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke,Yuan Yu, and Xiaoqiang Zheng.TensorFlow: Large-scalemachine learning on heterogeneous systems,2015. Software available from tensorflow.org.

References

Wurtz RH. Recounting the impact of Hubel and Wiesel. J Physiol. 2009;587(Pt 12):2817–2823. doi:10.1113/jphysiol.2009.17020

HUBEL DH, WIESEL TN. Receptive fields of single neurones in the cat's striate cortex. J Physiol. 1959;148(3):574–591.

Olga Russakovsky*, Jia Deng*, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg and Li Fei-Fei. (* = equal contribution) ImageNet Large Scale Visual Recognition Challenge.

DemsarJ, CurkT, ErjavecA, GorupC, HocevarT, MilutinovicM, MozinaM, PolajnarM, ToplakM, StaricA, StajdoharM, UmekL, ZagarL, ZbontarJ, ZitnikM, ZupanB (2013) Orange: Data Mining Toolbox in Python. Journal of Machine Learning Research 14(Aug):2349−2353.

Chollet F. Keras [Internet] San Francisco (CA): GitHub, Inc.; 2017. [cited 2019 Apr 26]. Available from: https://github.com/keras-team/keras.