Deep Learning Based Automatic Immune Cell

Detection for Immunohistochemistry Images

Ting Chen and Christophe Chefd’hotel

Ventana Medical Systems, Inc. A Member of the Roche Group, USA, 94043

Abstract. Immunohistochemistry (IHC) staining is a widely used tech-nique in the diagnosis of abnormal cells such as cancer. For instance, itcan be used to determine the distribution and localization of the differen-tially expressed biomarkers of immune cells (such as T-cells or B-cells) incancerous tissue for an immune response study. Typically, the immuno-logical data of interest includes the type, density and location of theimmune cells within the tumor samples; this data is of particular inter-est to pathologists for accurate patient survival prediction. However, tomanually count each subset of immune cells under a bright-field micro-scope for each piece of IHC stained tissue is usually extremely tediousand time consuming. This makes automatic detection very attractive,but it can be very challenging due to the wide variety of cell appear-ances resulting from different tissue types, block cuttings, and stainingprocesses. This paper presents a novel method for automatic immune cellcounting on digitally scanned images of IHC stained slides. The methodfirst uses a sparse color unmixing technique to separate the IHC imageinto multiple color channels that correspond to different cell structures.Since the immune cell biomarkers that we are interested in are mem-brane markers, the detection problem is formulated into a deep learningframework using the membrane image channel. The algorithm is evalu-ated on a clinical data set containing a large number of IHC slides anddemonstrates more effective detection than the existing technique andthe result is also in accordance with the human observer’s output.

1 Introduction

Immunohistochemistry (IHC) slide staining has the advantage of identifying pro-teins in cells of a tissue section, and hence is widely used to study of the dis-tribution and localization of different cell types in a biological tissue such ascancerous cells and immune cells. For example, tumors often contain infiltratesof immune cells, which may prevent the development of tumors or favor theoutgrowth of tumors [1]. In this scenario, multiple biomarkers are used to tar-get different types of immune cells and the population distribution of each typeare compared with the clinical outcomes of the patients. As an emerging re-search topic of tremendous interest in pathology and immunology, an “immuneprofile” studies the correlation between the immune response and the growthand recurrences of human tumors. However, a prerequisite of an immune profile

G. Wu et al. (Eds.): MLMI 2014, LNCS 8679, pp. 17–24, 2014.c© Springer International Publishing Switzerland 2014

18 T. Chen and C. Chefd’hotel

study requires the human observer to manually locate and count the number ofdifferent immune cells within selected lymph node regions, which may containhundreds to thousands of cells. This is an extremely tedious and time consumingprocess and the results are also subject to intra- and inter-individual variability.In order to avoid the tedium involved in manual counting, a technique that isable to automatically and reliably detect different types of immune cells is ofgreat research and clinical interest.

Typically, a tissue slide is stained by an IHC diagnostic assay with a clusterof differentiation (CD) protein markers identifying the immune cells and the nu-cleus marker Hematoxylin (HTX) marking the nuclei. The stained slide is thenimaged using a CCD color camera mounted on a microscope or a scanner. Theacquired RGB color image is a mixture of the immune cell membrane and theuniversal cell nuclear biomarker expressions. Several techniques have been pro-posed in the literature to detect such cells. Most of the techniques are basedon image processing methods that capture the symmetric information of thecell appearance features. For instance, in [2] Pavin et al. proposed an iterativevoting method to cluster and group non-convex perceptual circular symmetriesalong the radial line of an object, and demonstrated its efficacy in the detec-tion of nucleus which presents a round blob shape. Xin et al. in [3] extendedPavin’s method by adding a shifted Gaussian kernel at the center of the vot-ing area and showed improved detection for overlapping cells. Machine learningtechniques have also been explored in literature for cell detection. A statisti-cal model matching method learned from structured SVM was proposed in [4]to identify the cell-like regions. However, all of the three aforementioned tech-niques are limited to nucleus rather than cell membrane detection. Since themost popular immune cell markers, such as CD3 and CD8 for universal T-cellsand cytotoxic T-cells respectively, are membrane markers, the stain appears asa ring instead of a blob. Another machine learning based system using SIFT,Random Forests, and Hierarchical Clustering was developed by Mualla et al. in[5] for unstained cell imaging which has the properties of maintaining sufficientcontrast of cell boundaries. In this work, the SIFT key-points are classified intocells and backgrounds, and all the key-points within each cell are linked togetherusing hierarchical clustering. The system was validated on a large data set, andhas shown robustness and stability. However, it is non-trivial to extend it todetect immune cells in IHC stained images.

Recently, an automatic CD8 cytotoxic T-cell counting algorithm was proposedby Niazi et al. in [6], wherein normalized multi-scale difference of Gaussian isused to detect the candidate regions, and the color and intensity informationis applied to fuse the results. This image processing and rule-based techniquehas a potential robustness concern due to the great diversity exhibited in termsof cell shape and size. Meanwhile, deep learning techniques, such as Convolu-tional Neural Networks (CNN) [7], have evidenced great success in mitosis celldetection from histology images stained with Hematoxylin and Eosin [8].Thisserves as a valuable source of inspiration for developing a new learning-basedimmune cell detection algorithm as the hard mitosis cell detection problem also

Deep Learning Based Automatic Immune Cell Detection for IHC Images 19

suffers from large appearance variation and high complexity. As a powerful pixelclassifier, CNN takes the image patch centered at each pixel as input and pro-vides the classification label for each patch. The advantage of CNN is that thefeature descriptors are automatically learned as the kernel matrices in the con-volution layers, therefore, less effort is needed in designing sophisticated featuresfor immune cells.

In this paper, we propose an automatic immune cell detection frameworkfor IHC images. The algorithm first unmixes the RGB image into different colorchannels corresponding to the different cell structures. CNN is then trained usingthe immune cell marker image channel and produces a probability response mapof the immune cell locations. Finally, we apply non-max suppression to obtainthe centroids of the cells. There is very little published literature for IHC imageanalysis due to the limited data availability. As the major contribution of thispaper, we are the first to propose an immune cell counting algorithm basedon sparse color unmixing and deep learning, and demonstrate a RGB imageunmixing algorithm potentially working for more than three stains as well as anew, important application of the CNN algorithm.

2 Methodology

Fig. 1. The framework of the algorithm

In this section, we presentthe methodology of our algo-rithm. We begin with illus-trating the basic frameworkin Fig.1. In the analysis ofthe cancerous tissues, differ-ent biomarkers are specifiedto one or more types of im-mune cells. For instance, CD3is a known universal markerfor all T-cells and CD8 onlystains the membranes of cy-totoxic T-cells. To detect theT-cells, the IHC image is firstunmixed into HTX and T-cellmarker channels. The pixelsin the T-cell channel are thenclassified into foreground and

background using CNN, the output of which is a probability map. Finally, weapply non-max suppression to the probability map and obtain the T-cell cen-troids.

2.1 Color Unmixing

There are several techniques available in the literature to unmix each pixel ofthe RGB image into multiple biologically meaningful channels corresponding to

20 T. Chen and C. Chefd’hotel

different stain colors. As the most widely used method in the digital pathologydomain, Ruifrok et al. developed an unmixing algorithm [9] to unmix the RGBimage with up to three stains in the converted optical density space. Given thereference color vectors xi ∈ R3 of the pure stains, the method assumes that eachpixel of the color mixture a ∈ R3 is a linear combination of the pure stain colorsand solves a linear system to obtain the combination weights b ∈ RM . The linearsystem is denoted as a = Xb, where X = [x1, . . . , xM ],M ≤ 3 is the matrix ofreference colors. Most of the effort in reported literature is for multi-spectralimage unmixing [10] which is not applicable for RGB images. To the best of ourknowledge, a technique that is able to reliably unmix more than three colors froma RGB image has never been reported in literature. In this paper, we extendedRuifrok’s method by providing the L1 norm constraint for b due to the fact thatonly a small number of stains exist at each pixel. With the sparse constraint,we obtain the following advantages: (1) the linear system is no longer deficientand thus can potentially unmix more than three colors, (2) background noise isgreatly suppressed in the unmixed channels due to the sparsity regularizationwhich leads to a larger signal noise ratio.

In a pre-processing step, the RGB image I is converted into the optical den-sity (OD) space using the formula Oc = − log( Ic

I0,c) derived from Beer’s law; this

is based on the fact that the optical density is proportional to the stain concen-tration. Here c is the index of the RGB color channels, I0 is the RGB value ofthe white points and O is the optical density image obtained. As in [9], O willbe the image to work with in the rest of the paper.

Let a be a pixel of O and it is a 3-dimensional column vector correspondingto the OD values converted from RGB. There are M biomarkers available in amultiplex IHC slide corresponding to M stain colors. Let b be the combinationweight vector of the stains; bm,m = 1, . . . ,M is the mth element of b. The sparseunmixing problem is then formulated as the following:

minb

||a−Xb||22 + λ||b||1. (1)

Each column ofX corresponds to a reference stain color sampled from the controlslide of pure stain. Existing techniques such as LASSO can be used to solve Eqn.1, and can achieve better unmixing results than in [9] in terms of robustness andaccuracy.

Note that instead of working in the unmixed immune cell marker channel, thedetection algorithm proposed in this paper is also applicable to the absorbanceimage of I. However, the noisy HTX channel ( Fig.5) may lead to a false positivedetection of immune cells. As accurate unmixing provides an accurate separationof the two stains with less cross-talk among different channels, we therefore workin the unmixed T-cell marker channel which has sufficient information about cellmembranes.

2.2 Cell Detection

The accurate detection of immune cells is a challenging task due to the largevariation of data caused by a variety of issues, such as different tissue types,

Deep Learning Based Automatic Immune Cell Detection for IHC Images 21

tissue section cuttings, chemical staining artifacts, and scanner focus problems,etc. Fig.2 shows some example fields of view (FOVs) from the whole slide imagesand the tissue cutting problem.

Fig. 2. Example FOVs demonstrate the data variations. The boxes show the stainingartifacts and the cell shape variations due to cutting. The figure on the right shows theblob and ring shape cells from different cuttings.

Fig. 3. The architecture of CNN for membrane detection

Given an input RGBimage I, sparse colorunmixing is used tounmix the image intoimmune cell markersand nucleus markerchannels, denoted asIdab and Ihtx, respec-tively. Idab is thenused as the input image for learning the detector. The immune cell detectionproblem is formulated as classifying each pixel of Idab into two classes, positivefor the centroids of the immune cells and negative for the rest. More specifically,let P be the training data and Y be the set of labels, where (pn, yn) are drawnrandomly from P × Y based on some unknown distribution. In this application,P is the set of patch images centered at each pixel of Idab and Y is a binaryset containing two labels {+1,−1}. We have ground truth immune cell locationsmanually labeled by the pathologist, wherein the coordinates of the cell centroidsare recorded. The positive class of training data consists of k by k-pixel imagepatches centered at the pixels within a d-pixel neighborhood of the recordedcoordinates. The non-immune cell class contains all the image patches centeredat pixels sampled from the boundaries of the cells and the background.

Fig. 4. The test image and its probabilitymap from CNN

A convolutional neural network(CNN) is trained given the trainingdata (patches and their correspondinglabels). CNN is basically a neural net-work with the sequence of alternatingconvolutional layers and sub-samplinglayers, followed by the fully connectedlayers, which can be trained by aback-propagation algorithm (Fig.3). CNN has the advantages of automaticallylearning the feature descriptors which are invariant to small translation and dis-tortion from the training image patches. The convolution layer convolves theinput patch with a kernel matrix; the output is passed to a continuous and dif-ferentiable activation function. The kernel matrix is part of the parameter set to

22 T. Chen and C. Chefd’hotel

be learned in CNN. The sub-sampling layer reduces the size of the image by acoarse sampling or max-pooling. The fully connected layer is similar to a typicalneural network designed to generate probabilistic labels for each class.

In the testing stage, we first unmix the test RGB image I to obtain Idab, thenapply the trained CNN classifier to the patches centered at each pixel of thetest image Idab. Let y = C(p) denote the CNN classifier that takes the patchp as input and produces the probabilistic label y for the patch, here y ∈ [0, 1].Hence, a probability map M as shown in Fig.4 is created for each test image, inwhich higher probability means that the pixel is more likely to be the centroidof the immune cell. Non-maximum suppression (NMS) [11] is typically used forthe local maximum search, i.e. finding the pixel with the value that is greaterthan all its neighbors. Therefore, we apply NMS to the probability map to yieldthe final detection.

3 Experiments

In this section, we empirically validate our immune cell detection algorithm, andcompare it to the existing technique and human observer’s output.

3.1 Data Set and Experiment Setting

Fig. 5. Unmixing of the RGB image. (a) input image I . (b) inputimage in absorbance space. (c) Idab from Ruifrok’s method [9].(d) Idab from sparse unmixing.

A clinical data setcontaining severaldifferent cancer tis-sue samples wasused to test theproposed approach.The data set con-tains 42 fields ofview (FOVs) whichwere scanned under 20X magnification. The tissues were stained with the follow-ing assay: DAB for the immune cell markers and HTX for the nucleus marker.We have two types of T-cell markers in our data set, the universal T-cell markerCD3 and the cytotoxic T-cell marker CD8. As both are membrane markers withDAB staining, they have a similar appearance. Hence, the same detection al-gorithm can be used for both cases. Among the 42 FOVs, 10 were randomlyselected by the pathologist for manual counting for quality control purposes,and these images were reserved for testing only. We first selected 3 images outof the remaining 32 FOVs, and manually annotated the positive and negativepixels. This yielded 491 positive samples and 539 negative annotated samples.To generate more training data, we sampled the patches around the 2-pixelneighborhood of the annotation, also flipped and rotated the patches, and fi-nally created 17355 training patches in all. For all the experiments, the sparseregularization parameter λ was set to be 0.5.

Deep Learning Based Automatic Immune Cell Detection for IHC Images 23

3.2 Comparisons of Detection Algorithms

We first compared the sparse unmixing with Ruifrok’s unmixing algorithm andshowed that the sparse unmixing leads to better results with less backgoundnoise. Fig.5 shows the input RGB image, its absorbance transformed image andthe unmixed immune cell channel from both methods.

To train a CNN classifier, we used the following network configuration. Thesize of the image patch was set to be 27 by 27. We used Idab as the input layerand set 2 convolution layers with both kernel sizes being 6 by 6. 10 epochswere used for training and we finally obtained the training error 0.008. Exampledetection results are shown in Fig.6. We compared with iterative voting basedmethod [2] for immune cell detection. The algorithm tends to accumulate votesaround the centroid of the ring, hence it has the potential to fail if the immunecell shows a solid elliptical shape due to the angle at which the tissue samplewas cut (Fig.2). It also has the disadvantage of being sensitive to local gradientchanges, since local votes will be accumulated near the boundary of the blob.See Fig. 6 for the examples. The iterative voting based method obtained twolocal maximums near the boundary of the elliptical blob. Note that we alsoimproved the iterative voting algorithm by filtering out false positive detectionsnear the ring boundaries caused by the local votes using the binary mask of theimmune cell marker image channel. As both blob and ring shape immune cellscan be included in the training of CNN, the proposed algorithm is more robustin detecting both types of cells.

Fig. 6. First row: manual annotation. Second row: detection using the proposedmethod. Third row: detection using iterative voting based method [2].

24 T. Chen and C. Chefd’hotel

3.3 Comparison to Human Observer’s Counts

We have 10 FOVs manually annotated by a human observer. To measure theaccuracy of the proposed detection algorithm, we compared the algorithm countswith the manual counts and obtained correlation coefficients as high as 0.9949.The average count difference is 17.19 cells per FOV; the number of cells in eachFOV is between 200 to 300. This demonstrates that the automatic immune celldetection algorithm proposed in this paper is in concordance with the human’soutput. As more annotations become available, we believe that it is possible totrain the algorithm to follow the human observer’s behavior even more reliably.

4 Conclusion

In this paper, we introduced an automatic immune cell detection algorithm forIHC images to assist the clinical immune profile studies. Sparse color unmixingalgorithm was proposed to unmix the RGB image into different biologicallymeaningful color channels. The cell detector was trained using a convolutionalneural network in the immune cell marker image channel. The experiments usingclinical data demonstrate the efficacy of the proposed algorithm in terms ofaccuracy, stability, and robustness when compared to the existing techniquesand human observer.

References

1. Galon, J., et al.: Type, Density, and Location of Immune Cells Within HumanColorectal Tumors Predict Clinical Outcome. Science 313(5795), 1960–1964 (2006)

2. Parvin, B., et al.: Iterative Voting for Inference of Structural Saliency andCharacter-ization of Subcellular Events. IEEE Trans. Image Processing 16(3), 615–623 (2007)

3. Xin, Q., et al.: Iterative Voting for Inference of Structural Saliency and Character-ization of Subcellular Events. IEEE Trans. Biomedical Engineering 59(3), 754–765(2011)

4. Arteta, C., Lempitsky, V., Noble, J.A., Zisserman, A.: Learning to Detect CellsUsing Non-overlapping Extremal Regions. In: Ayache, N., Delingette, H., Golland,P., Mori, K. (eds.) MICCAI 2012, Part I. LNCS, vol. 7510, pp. 348–356. Springer,Heidelberg (2012)

5. Mualla, F., et al.: Automatic Cell Detection in Bright-Field Microscope ImagesUsing SIFT, Random Forests, and Hierarchical Clustering. IEEE Trans. MedicalImaging 32(12), 2274–2286 (2013)

6. Niazi, M.K.K., et al.: An Automated Method for Counting Cytotoxic T-cells fromCD8 Stained Images of Renal Biopsies. In: SPIE, vol. 8676 (2013)

7. LeCun, Y., et al.: Gradient-based Learning Applied to Document Recognition.Proceedings of the IEEE 86(11), 2278–2324 (1998)

8. Ciresan, D.C., Giusti, A., Gambardella, L.M., Schmidhuber, J.: Mitosis Detec-tion in Breast Cancer Histology Images with Deep Neural Networks. In: Mori, K.,Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013, Part II. LNCS,vol. 8150, pp. 411–418. Springer, Heidelberg (2013)

9. Ruifrok, A.C., et al.: Quantification of Histochemical Staining by Color Deconvo-lution. Anal. Quant. Cytol. Histol. 23, 291–299 (2001)

10. Kesheva, N.: A Survey of Spectral Unmixing Algorithms. Lincoln Laboratory Jour-nal 14(1), 55–78 (2003)

11. Lindeberg, T.: Edge Detection and Ridge Detection with Automatic Scale Selec-tion. In: CVPR, pp. 465–470 (1996)