36

ReviewPathology

Advanced deep learning applications in diagnostic pathologyDaisuke Komura1 and Shumpei IShIKawa1*1 Department of Preventive Medicine, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-0033, Japan

Abstract. For many years, pathologists have performed histopathological diagnoses by observing tissue specimens under a microscope. Recently, however, it has become possible to scan whole slides of tissue specimens using a dedicated slide scanner and store the resultant high-resolution digital images, i.e., whole slide images. This has led to the emergence of digital pathology, a field in which whole slide images are used for histopathological diagnoses. This field is gradually expanding, especially in large hospitals such as university hospitals. In addition, dramatic advancements in image recognition technology have been made since 2012 when deep learning won the general image recognition competition ILSVRC with overwhelming accuracy. Subsequently, deep learning has been applied to various medical images, including X-ray, ocular fundus, and skin images, and is reported to have achieved generalist- or even professional-level diagnostic accuracy in each field. Similarly, the use of deep learning, directed towards digital histopathological images, for assistance with pathological diagnoses is gradually becoming practicable, especially for diseases in which many cases occur. Recently, advanced applications have been developed such as searching for similar cases, predicting genetic mutations from histological images, and generating special stained images from hematoxylin and eosin-stained images. These emerging applications have the potential to greatly expand the field of diagnostic pathology and contribute to the further development of medicine. In this review, we introduce the use of deep learning technology in the field, detail the current advanced applications, and speculate on future perspectives.Key words: diagnostic pathology, deep learning, whole slide image

highlights

There have been recent advances in digital pathology and image recognition technology using deep neural networks. Accordingly, the clinical applications of computer-assisted histopathological diagnosis, such as tumor detection and mitosis cell count, are becoming a reality. Although such traditional applications have been well discussed so far, more advanced applications, such as content-based image retrieval, prediction of genetic mutations from histological images, and virtual staining or stain-to-stain transformation, has not been well documented. Because these applications can have a huge impact on diagnostic pathology, this review introduces the current advanced applications and future perspectives.

Introduction

Human tissues are composed of various cell types. As dis-ease often changes the morphology, structure (e.g., glands), or composition of cells from normal tissues in target organs, disease diagnosis can be performed by observing stained tissue slides under a microscope. This practice is known as diagnostic pathology.

For many years, pathologists have observed tissue slides under a microscope. In recent years, however, some hos-pitals have begun to use a special slide scanner to capture and then store the entire glass slide as a digital image, known as a whole slide image (WSI), which can be used for remote diagnosis (Fig. 1). With the digitization of tissue

images, information on slides can be easily sent to other hospitals; thus, remote diagnosis is becoming practicable. Importantly, with the advancement of digital image analy-sis technology, such as machine learning, computer-assisted diagnosis (CAD) is now also becoming a possibility.

Machine learning for images has a long history; however, tremendous progress has especially been made in a class of deep neural networks known as convolutional neural networks (CNNs) since the 2012 ILSVRC general image rec-ognition competition was won with overwhelming accuracy [1]. CNNs include many layers of multiple convolutional kernels to extract specific local structures from images (e.g., a line or a circle) and the weight of these convolutional kernels is optimized through training with labeled images (a process known as supervised learning). CNNs are now widely used in image classification, object detection, and *Correspondence to: Ishikawa, S.: Ishum-prm@m.u-tokyo.ac.jp

Received: Mar. 15, 2021; Accepted: May 11, 2021

Translat Regulat Sci. 3(2): 36–42, 2021; doi: 10.33611/trs.2021-005

©2021 Catalyst Unit

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-Commercial No Derivatives (by-nc-nd) License. (CC-BY-NC-ND 4.0: https://creativecommons.org/licenses/by-nc-nd/4.0/)

Deep learning applications in pathology

37

segmentation because of their superior performance over conventional methods. In histopathological image analysis, CNNs have been applied in the detection or segmentation of tumor cells [2] (preprint), detection of mitotic cells [3, 4], segmentation of glands [5], [6] (preprint), subtyping of tumors (such as lymphoma subtypes) [7], and grading of cancers (such as Gleason scoring of prostate cancer samples) to facilitate routine pathological diagnoses [8]. These traditional applications are important for reducing the burden on pathologists and equalizing the quality of diagnosis; however, the outstanding image recognition and feature extraction abilities of CNNs make it possible to real-ize more advanced and clinically important applications. These emerging applications include prediction of somatic mutations or survival time from histopathology images as well as content-based image retrieval (CBIR).

In 2014, generative adversarial networks (GANs), another important class of deep neural network, were invented by Goodfellow et al. [9] (preprint). GANs consist of two deep neural networks: a generator and a discriminator. Genera-tors attempt to generate as realistic an image as possible, whereas discriminators (which have similar structures to CNNs) aim to discriminate between real images and the image generated by the generator. Through the com-petitive optimization of these two networks, generators are ultimately able to generate images that closely resemble the real images. GANs can not only create real images de novo but also transform images in one domain into those in another domain. Examples of GAN applications in general image analysis include generation of real face images with arbitrary features [10], coloring monochrome pictures [11] (preprint), and converting sketches to color photographs [12] (preprint). In diagnostic pathology, GANs are now being used to increase the resolution of microscopic im-ages and generate special stained images from unstained or hematoxylin and eosin (H&E)-stained tissue images.

In this review, we introduce the various research ap-

plications and the limitations of such applications, i.e., the tasks that pathologists do not (which does not always mean “cannot”) do, in the standard workflows of pathological diagnosis using CNNs or GANs.

Advanced Deep Learning Applications

Content-based image retrievalWhen pathologists encounter a case of which they have

no prior knowledge, they often ask for the opinion of other pathologists or try to find similar images in reference ma-terials such as pathology atlas books. This, however, can be a time-consuming process. If they could instead quickly find similar cases in a pathology image database, the prob-ability and time for performing a correct diagnosis could be substantially improved. In image analysis, the technique of retrieving similar images by using images as queries is known as CBIR (briefly mentioned in section 1). In CBIR, each image is converted to numerical representations, such as high-dimensional vectors, that represent the character-istics of the image. Representations of two images should be closely related when the content of the images is similar and vice versa. Previously, many image representations have been used for CBIR in the field of diagnostic pathol-ogy including local binary patterns or their variants [13] and scale invariant feature transform [14]. Additionally, some methods have used unsupervised or self-supervised machine learning models to optimize image features [15] [16] (preprint).

More recently, image features extracted from deep neural networks have been shown to outperform other features in terms of retrieval accuracy [17] [18] (preprint) [19]. Al-though CBIR is a promising technique, CBIR systems for diagnostic pathology have been unavailable until recently, possibly due to insufficient retrieval performance and the lack of an image database with sufficient quality. Therefore, we developed and published a web service, Luigi (https://

Fig. 1. Whole slide images in digital pathology. (A) Slide scanner (Hamamatsu Photonics NanoZoomer S60). (B) A hematoxylin and eosin-stained tissue slide and (C) the scanned image of the tissue slide (i.e., the whole slide image; scale bar: 2.5 mm). (D) Magnified image of the rectangular area in (C); scale bar: 50 µm.

38

Translational and Regulatory Sciences

luigi-pathology.com), which allows users to search for similar images and related genomic abnormalities in over 7,000 cases from 32 cancer types included in The Cancer Genome Atlas dataset [20] [21, 22] (preprint) (Fig. 2). Luigi uses 1,024 dimensional image features, which we call “deep texture representation,” from a special type of CNN, a bilinear CNN, with positional invariance [23] (preprint). Positional invariance is important because histopathological images differ from ordinary general images, e.g., images of dogs and cats, in that they have a texture-like structure. We found that the texture information, which is extracted from a middle layer of a CNN pretrained on numerous general images, accurately expresses the characteristics of histopathological images of tumors; hence, it can be ap-plied to retrieve similar images. We have also developed a smartphone version of Luigi; therefore, using software with a microscope lens adapter for smartphones (sold for a few dozen dollars at most), CBIR can be used in small hospitals that do not have expensive histology imaging equipment such as slide scanners and microscope digital cameras.

Recently, Fuchs et al. [24] developed a freely-available CBIR system that searches similar cases across social media and PubMed.

Although CNNs have improved retrieval accuracy, CBIR-related challenges still exist in relation to pathology images. For example, there remains a need to improve robustness against stain variations. Furthermore, differences in sample preparation and equipment can affect the process. For instance, H&E stains are often used to stain pathological tissues, but color distribution varies slightly depending on the composition of the staining solution and the washing conditions (including the type of water used). The thickness of the tissue and the scanner used can also affect the color distribution and image quality of WSIs. Such differences can lead to differences in representation, which can in turn reduce the accuracy of CBIR. A simple solution to these problems is the use of image transformation techniques such as color normalization [25] (preprint) [26, 27], which adjusts the color distribution of tissue images to a reference image. However, given that the selection of a reference

Fig. 2. A content-based image retrieval system: Luigi (https://luigi-pathology.com). (Left) Query image, (middle) retrieved similar images, and (right) predicted gene mutations with confidence levels. Receiver operating characteristics of the classifier on test data and histology images of mutation-positive and -negative cases are also shown.

Deep learning applications in pathology

39

image can affect quality and because other differences in image quality can create biases, further research is required to refine CBIR.

Prediction of clinical biomarkersSupervised machine learning can be used to predict not

only existing disease classifications but also features that are usually outside the scope of pathological diagnosis, e.g., patient prognosis and genomic features. For example, survival or recurrence of colon cancer [28], melanoma [29], or mesothelioma [30] can be predicted from H&E-stained images using CNN. Furthermore, Li et al. [31] incorporated topological structures of WSIs using graph CNNs to predict the survival of patients with lung or brain cancer.

Recent advances in parallel sequencers have made it pos-sible to comprehensively examine cancer genome informa-tion, which has led to genome-based diagnosis and drug selection. Although cancer genome analysis is a powerful tool, it cannot realistically be used for every patient because of its high costs. In contrast, histopathological diagnosis is an essential tool that is routinely used. If it becomes possible to predict genomic aberrations from histology images alone, histopathological diagnosis will represent an inexpensive screening tool with particular relevance to developing countries where genome analysis is not easily accessible. Even before the digitization of histopathological images, the association between genetic mutations and histological morphology in various cancers has been studied by patholo-gists [32–34]. However, because multiple combinations of driver genes and cancer types exist, it remains difficult for pathologists to identify such associations on a large scale. To overcome these challenges, supervised learning models that predict mutations from histopathology data are a use-ful application. For example, Coudray et al. [35] showed that the presence or absence of six somatic gene mutations, including mutations for TP53, EGFR, and KRAS, could be accurately estimated from WSIs of lung cancer using CNNs. Subsequently, it has been shown that SPOP mutations in prostate cancer [36] (preprint) and BRAF and NRAS muta-tions in melanoma [37] (preprint) can be estimated from histopathological images. Prediction of actionable genomic aberrations from H&E images alone has received substan-tial attention because of its potential clinical utility. For example, some researchers have reported that patients with microsatellite instability tumors in some cancer types, such as stomach or colon adenocarcinoma, respond to immune checkpoint inhibitors and that microsatellite instability status can be predicted with high accuracy when assessing H&E-stained images using a deep neural network [38, 39]. Furthermore, in some studies, including our own, predic-tion models have been developed from tens of clinically actionable mutations [22, 40, 41]. Some of our models are implemented in the abovementioned Luigi system.

Other attempts have been made to predict prognosis with greater accuracy using the combination of cancer genome information and histological images. For example, Mobadersany et al. [42] used a specialist CNN known as the genomic survival convolutional network to analyze histopathological images of brain tumors and related genomic abnormality information. In addition, Cheerla et al. [43] showed that the prognosis of brain tumors could be

accurately estimated using such methods, albeit to a lesser extent than by analyzing genomic abnormalities alone.

Virtual stainingDepending on the purpose, tissue specimens are stained

in various ways before being examined under a microscope. Although these steps are essential for histopathological diagnosis, they are time consuming, labor intensive, and expensive especially for special stains. Additionally, with small tissue specimens, it may not be possible to obtain suf-ficient tissue sections for multiple staining. In recent years, a technology has been developed by which unstained speci-mens can be virtually stained; this technology learns the correspondence between unstained and stained specimens through image conversion technologies such as GANs [44, 45]. Another technique has been proposed in which images that have already been stained using one method (mostly with H&E) are converted to appear stained by other special methods [46] (preprint) [47]; this is known as stain-to-stain transformation [48]. GANs are usually trained using paired images. When unstained images are virtually stained by GANs, paired images for training can be prepared for the exact same tissue; however, for stain-to-stain transforma-tions, images of serial sections that differ slightly are usually prepared and two sections are each stained with different stains. It is relatively easy to align unstained specimens with stained specimens because their positions are perfectly matched; thus, accurate methods for paired-images, such as pix2pix [49] (preprint), can be used (Fig. 3A). In contrast, during the conversion from H&E staining to special staining, the two images from serial sections differ slightly. Although the accuracy is lower than that obtained when using paired images, the CycleGAN [50] (preprint) technique can be ap-plied to such unpaired images (Fig. 3B).

Although these techniques seem promising, current evaluations are mainly based on reconstruction accuracy; indeed, few studies have evaluated the improvements in the quality of diagnoses [51] (preprint). Therefore, the clinical utility of the techniques in terms of final diagnostic accuracy and time/cost reduction in various applications should be assessed in future research.

Limitations of deep neural networks in diagnostic pathology

Although deep neural networks are promising technolo-gies, they have some limitations when applied to diagnostic pathology. First, the number of training samples is often insufficient, especially with rare diseases for which few training samples are available; thus, it is difficult to develop a machine learning model with good performance. There-fore, techniques that can learn accurately even from a small number of samples such as few-shot learning or few-shot GAN, are often necessary [52, 53] (preprint).

Interpreting the resultant models can also be a problem. For instance, if a category, e.g., mutation status, is predict-able, it is important to know the features that were used to predict it to determine whether the model is biased and whether it makes biological sense. Algorithms for present-ing the regions that contribute to the decision making of deep neural networks have been proposed, e.g., GradCAM [54] and integrated gradients [55] (preprint). However, it

40

Translational and Regulatory Sciences

is sometimes difficult to interpret the pathological images by presenting only specific areas of the image, which can usually be achieved with general images. Such problems from deep neural networks also exist for CBIR and virtual staining. Thus, a visualization method that is specifically suitable for pathological images should be investigated further.

Future Perspectives

Previous research has proved that deep learning algo-rithms used with histopathology images are effective. In future research, it will be important to explore the possibility of using these models in actual pathological diagnoses for their ability to improve the accuracy of the diagnoses and reduce costs when incorporated into a system. To improve

the usefulness of the applications themselves, the accumula-tion of data, including training data, will be important. The establishment of a system by which to accumulate data from many medical institutions and make it available to many researchers and doctors is expected to bring these applica-tions closer to practical use.

Conflict of InterestThe authors declare that they have no conflict of interests.

acknowledgmentsThis work was supported by the AMED Practical

Research for Innovative Cancer Control [Grant Number JP 20ck0106400h9903 and 21ck0106640 to S.I.] and JSPS KAKENHI Grant-in-Aid for Scientific Research (B) [Grant Number 21H03836 to D.K].

Fig. 3. Generative adversarial networks (GANs) for virtual staining. (A) GANs for paired images (e.g., pix2pix [49]). (B) GANs for unpaired images (e.g., cycleGAN [50]). G: generator; D: discriminator; H&E: hematoxylin and eosin.

Deep learning applications in pathology

41

references

1. Krizhevsky, A., Sutskever, I. and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25: 1097–1105.

2. Kanavati, F., Ichihara, S., Rambeau, M., Iizuka, O., Arihiro, K. and Tsuneki, M. 2020. Deep learning models for gastric signet ring cell carcinoma classification in whole slide images. arXiv. Preprint posted Nov 18, 2020. doi. arXiv:2011.09247.

3. Cireşan, D. C., Giusti, A., Gambardella, L. M. and Schmidhuber, J. 2013. Mitosis detection in breast cancer histology images with deep neural networks. pp. 411–418. In: Medical Image Computing and Computer-Assisted Intervention International conference on medical image computing and computer-assisted intervention, Springer, Berlin, Heidelberg.

4. Veta, M., van Diest, P. J., Willems, S. M., Wang, H., Madabhushi, A., Cruz-Roa, A., Gonzalez, F., Larsen, A. B., Vestergaard, J. S., Dahl, A. B., Cireşan, D. C., Schmidhuber, J., Giusti, A., Gambardella, L. M., Tek, F. B., Walter, T., Wang, C. W., Kondo, S., Matuszewski, B. J., Precioso, F., Snell, V., Kittler, J., de Campos, T. E., Khan, A. M., Rajpoot, N. M., Arkoumani, E., Lacle, M. M., Viergever, M. A. and Pluim, J. P. 2015. Assessment of algorithms for mitosis detection in breast cancer histopathology images. Med. Image Anal. 20: 237–248. [Medline] [CrossRef]

5. Chen, H., Qi, X., Yu, L. and Heng, P. A. 2016. DCAN: Deep contour-aware networks for accurate gland segmentation. arXiv. Preprint posted Apr 10, 2016. doi. arXiv:1604.02677.

6. Binder, T., Tantaoui, E. M., Pati, P., Catena, R., Set-Aghayan, A. and Gabrani, M. 2019. Multi-organ gland segmentation using deep learning. Front. Med. (Lausanne) 6: 173. [Medline] [CrossRef]

7. Li, D., Bledsoe, J. R., Zeng, Y., Liu, W., Hu, Y., Bi, K., Liang, A. and Li, S. 2020. A deep learning diagnostic platform for diffuse large B-cell lymphoma with high accuracy across multiple hospitals. Nat. Commun. 11: 6004. [Medline] [CrossRef]

8. Wang, D., Foran, D. J., Ren, J., Zhong, H., Kim, I. Y. and Qi, X. 2015. Exploring automatic prostate histopathology image Gleason grading via local structure modeling. Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference, 2015: 2649–2652.

9. Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. and Bengio, Y. 2014. Generative adversarial networks. arXiv. Preprint posted Apr 10, 2014. doi. arXiv:1406.2661.

10. He, Z., Zuo, W., Kan, M., Shan, S. and Chen, X. 2019. AttGAN: Facial attribute editing by only changing what you want. IEEE Trans. Image Process. 28: 5464–5478. [Medline] [CrossRef]

11. Nazeri, K., Ng, E. and Ebrahimi, M. 2018. Image colorization with generative adversarial networks. arXiv. Preprint posted Mar 14, 2018. doi. arXiv:1803.05400.

12. Chen, W. and Hays, J. 2018. SketchyGAN: Towards diverse and realistic sketch to image synthesis. arXiv. Preprint posted Jan 9, 2018. doi. arXiv:1801.02753.

13. Erfankhah, H., Yazdi, M., Babaie, M. and Tizhoosh, H. R. 2019. Heterogeneity-aware local binary patterns for retrieval of histopathology images. IEEE Access 7: 18354–18367. [CrossRef]

14. Mehta, N. Raja’s A, Chaudhary, 2009. Content based sub-image retrieval system for high resolution pathology images using salient interest points. In Proceedings of 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. doi. 10.1109/IEMBS.2009.5334811.

15. Zheng, Y., Jiang, Z., Shi, J. and Ma, Y. 2014. Retrieval of

pathology image for breast cancer using PLSA model based on texture and pathological features. In Proceedings of 2014 IEEE International Conference on Image Processing (ICIP). doi. 10.1109/ICIP.2014.7025467.

16. Gildenblat, J. and Klaiman, E. 2020. Self-supervised similarity learning for digital pathology. arXiv. Preprint posted May 20, 2019. doi. arXiv:1905.08139.

17. Hegde, N., Hipp, J. D., Liu, Y., Emmert-Buck, M., Reif, E., Smilkov, D., Terry, M., Cai, C. J., Amin, M. B., Mermel, C. H., Nelson, P. Q., Peng, L. H., Corrado, G. S. and Stumpe, M. C. 2019. Similar image search for histopathology: SMILY. NPJ Digit. Med. 2: 56. [Medline] [CrossRef]

18. Kalra, S., Choi, C., Shah, S., Pantanowitz, L. and Tizhoosh, H. R. 2019. Yottixel—An image search engine for large archives of histopathology whole slide images. arXiv. Preprint posted Nov 20, 2019. doi. arXiv:1911.08748.

19. Kalra, S., Tizhoosh, H. R., Shah, S., Choi, C., Damaskinos, S., Safarpoor, A., Shafiei, S., Babaie, M., Diamandis, P., Campbell, C. J. V. and Pantanowitz, L. 2020. Pan-cancer diagnostic consensus through searching archival histopathology images using artificial intelligence. NPJ Digit. Med. 3: 31. [Medline] [CrossRef]

20. Weinstein, J. N., Collisson, E. A., Mills, G. B., Shaw, K. R., Ozenberger, B. A., Ellrott, K., Shmulevich, I., Sander, C., Stuart J. M., Cancer Genome Atlas Research Network 2013. The cancer genome atlas pan-cancer analysis project. Nat. Genet. 45: 1113–1120. [Medline] [CrossRef]

21. Komura, D., Fukuta, K., Tominaga, K., Kawabe, A., Koda, H., Suzuki, R., Konishi, H., Umezaki, T., Harada, T. and Ishikawa, S. 2018. Luigi. Large-scale histopathological image retrieval system using deep texture representations. bioRxiv. Preprint posted Jul 19, 2018. doi.org/ 10.1101/345785.

22. Komura, D., Kawabe, A., Fukuta, K., Sano, K., Umezaki, T., Koda, H., Suzuki, R., Tominaga, K., Konishi, H., Nishida, S., Furuya, G., Katoh, H., Ushiku, T., Fukayama, M., Ishikawa, S. 2020. Deep texture representations as a universal encoder for pan-cancer histology. bioRxiv. Preprint posted Jul 29, 2020. doi.org/ 10.1101/2020.07.28.224253.

23. Lin, T.Y., Roychowdhury, A. and Maji, S. 2015. Bilinear CNNs for fine-grained visual recognition. arXiv. Preprint posted Apr 29, 2015. doi. arXiv:1504.07889.

24. Schaumberg, A. J., Juarez-Nicanor, W. C., Choudhury, S. J., Pastrián, L. G., Pritt, B. S., Prieto Pozuelo, M., Sotillo Sánchez, R., Ho, K., Zahra, N., Sener, B. D., Yip, S., Xu, B., Annavarapu, S. R., Morini, A., Jones, K. A., Rosado-Orozco, K., Mukhopadhyay, S., Miguel, C., Yang, H., Rosen, Y., Ali, R. H., Folaranmi, O. O., Gardner, J. M., Rusu, C., Stayerman, C., Gross, J., Suleiman, D. E., Sirintrapun, S. J., Aly, M. and Fuchs, T. J. 2020. Interpretable multimodal deep learning for real-time pan-tissue pan-disease pathology search on social media. Mod. Pathol. 33: 2169–2185. [Medline] [CrossRef]

25. Ciompi, F., Geessink, O., Bejnordi, B. E., de Souza, G. S., Baidoshvili, A., Litjens, G., Van Ginneken, B., Nagtegaal, I. and Van Der Laak, J. 2017. The importance of stain normalization in colorectal tissue classification with convolutional networks. arXiv. Preprint posted Feb 20, 2017. doi. arXiv:1702.05931.

26. Khan, A. M., Rajpoot, N., Treanor, D. and Magee, D. 2014. A nonlinear mapping approach to stain normalization in digital histopathology images using image-specific color deconvolution. IEEE Trans. Biomed. Eng. 61: 1729–1738. [Medline] [CrossRef]

27. Sano, K., Komura, D. and Ishikawa, S. 2019. Fast and stable color normalization of whole slide histopathology images using deep texture and color moment matching. MICCAI 2019 Workshop Co. https://openreview.net/forum?id=Bkgwe3GnZB [accessed March 15, 2021].

28. Kather, J. N., Krisam, J., Charoentong, P., Luedde, T., Herpel, E., Weis, C. A., Gaiser, T., Marx, A., Valous, N. A., Ferber,

42

Translational and Regulatory Sciences

D., Jansen, L., Reyes-Aldasoro, C. C., Zörnig, I., Jäger, D., Brenner, H., Chang-Claude, J., Hoffmeister, M. and Halama, N. 2019. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS Med. 16: e1002730. [Medline] [CrossRef]

29. Kulkarni, P. M., Robinson, E. J., Sarin Pradhan, J., Gartrell-Corrado, R. D., Rohr, B. R., Trager, M. H., Geskin, L. J., Kluger, H. M., Wong, P. F., Acs, B., Rizk, E. M., Yang, C., Mondal, M., Moore, M. R., Osman, I., Phelps, R., Horst, B. A., Chen, Z. S., Ferringer, T., Rimm, D. L., Wang, J. and Saenger, Y. M. 2020. Deep learning based on standard H&E images of primary melanoma tumors identifies patients at risk for visceral recurrence and death. Clin. Cancer Res. 26: 1126–1134. [Medline] [CrossRef]

30. Courtiol, P., Maussion, C., Moarii, M., Pronier, E., Pilcer, S., Sefta, M., Manceron, P., Toldo, S., Zaslavskiy, M., Le Stang, N., Girard, N., Elemento, O., Nicholson, A. G., Blay, J. Y., Galateau-Sallé, F., Wainrib, G. and Clozel, T. 2019. Deep learning-based classification of mesothelioma improves prediction of patient outcome. Nat. Med. 25: 1519–1525. [Medline] [CrossRef]

31. Li, R., Yao, J., Zhu, X., Li, Y. and Huang, J. 2018. Graph CNN for survival analysis on whole slide pathological images. Lect. Notes Comput. Sci. 11071: 174–182. [CrossRef]

32. Nishino, M., Klepeis, V. E., Yeap, B. Y., Bergethon, K., Morales-Oyarvide, V., Dias-Santagata, D., Yagi, Y., Mark, E. J., Iafrate, A. J. and Mino-Kenudson, M. 2012. Histologic and cytomorphologic features of ALK-rearranged lung adenocarcinomas. Mod. Pathol. 25: 1462–1472. [Medline] [CrossRef]

33. Fecteau, R. E., Lutterbaugh, J., Markowitz, S. D., Willis, J. and Guda, K. 2014. GNAS mutations identify a set of right-sided, RAS mutant, villous colon cancers. PLoS One 9: e87966. [Medline] [CrossRef]

34. Lakhani, S. R. 1999. The pathology of familial breast cancer: Morphological aspects. Breast Cancer Res. 1: 31–35. [Medline] [CrossRef]

35. Coudray, N., Ocampo, P. S., Sakellaropoulos, T., Narula, N., Snuderl, M., Fenyö, D., Moreira, A. L., Razavian, N. and Tsirigos, A. 2018. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat. Med. 24: 1559–1567. [Medline] [CrossRef]

36. Schaumberg, A. J., Rubin, M. A. and Fuchs, T. J. 2017. H&E-stained whole slide image deep learning predicts SPOP mutation state in prostate cancer. bioRxiv. Preprint posted May 12, 2017. doi.org/ 10.1101/064279.

37. Kim, R. H., Nomikou, S., Coudray, N., Jour, G., Dawood, Z., Jour, G., Donnelly, D., Moran, U., Weber, J.S., Razavian, N., Snuderl, M., Shapiro, R. and Berman, R.S. 2020. A deep learning approach for rapid mutational screening in melanoma. bioRxiv. Preprint posted Aug 19, 2020. doi.org/ 10.1101/610311.

38. Kather, J. N., Pearson, A. T., Halama, N., Jäger, D., Krause, J., Loosen, S. H., Marx, A., Boor, P., Tacke, F., Neumann, U. P., Grabsch, H. I., Yoshikawa, T., Brenner, H., Chang-Claude, J., Hoffmeister, M., Trautwein, C. and Luedde, T. 2019. Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nat. Med. 25: 1054–1056. [Medline] [CrossRef]

39. Yamashita, R., Long, J., Longacre, T., Peng, L., Berry, G., Martin, B., Higgins, J., Rubin, D. L. and Shen, J. 2021. Deep learning model for the prediction of microsatellite instability in colorectal cancer: a diagnostic study. Lancet Oncol. 22: 132–141. [Medline] [CrossRef]

40. Fu, Y., Jung, A. W., Torne, R. V., Gonzalez, S., Vöhringer, H., Shmatko, A., Yates, L. R., Jimenez-Linan, M., Moore, L. and Gerstung, M. 2020. Pan-cancer computational histopathology reveals mutations, tumor composition and prognosis. Nat.

Can. 1: 800–810. [CrossRef] 41. Kather, J. N., Heij, L. R., Grabsch, H. I., Loeffler, C., Echle,

A., Muti, H. S., Krause, J., Niehues, J. M., Sommer, K. A. J., Bankhead, P., Kooreman, L. F. S., Schulte, J. J., Cipriani, N. A., Buelow, R. D., Boor, P., Ortiz-Brüchle, N. N., Hanby, A. M., Speirs, V., Kochanny, S., Patnaik, A., Srisuwananukorn, A., Brenner, H., Hoffmeister, M., van den Brandt, P. A., Jäger, D., Trautwein, C., Pearson, A. T. and Luedde, T. 2020. Pan-cancer image-based detection of clinically actionable genetic alterations. Nat. Can. 1: 789–799. [Medline] [CrossRef]

42. Mobadersany, P., Yousefi, S., Amgad, M., Gutman, D. A., Barnholtz-Sloan, J. S., Velázquez Vega, J. E., Brat, D. J. and Cooper, L. A. D. 2018. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Natl. Acad. Sci. USA 115: E2970–E2979. [Medline] [CrossRef]

43. Cheerla, A. and Gevaert, O. 2019. Deep learning with multimodal representation for pancancer prognosis prediction. Bioinformatics 35: i446–i454. [Medline] [CrossRef]

44. Rivenson, Y., Wang, H., Wei, Z., de Haan, K., Zhang, Y., Wu, Y., Günaydın, H., Zuckerman, J. E., Chong, T., Sisk, A. E., Westbrook, L. M., Wallace, W. D. and Ozcan, A. 2019. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nat. Biomed. Eng. 3: 466–477. [Medline] [CrossRef]

45. Zhang, Y., de Haan, K., Rivenson, Y., Li, J., Delis, A. and Ozcan, A. 2020. Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue. Light Sci. Appl. 9: 78. [CrossRef]

46. Levy, JJ., Jackson, CR., Sriharan, A., Christensen, B. and Vaickus, LJ. 2020. Preliminary evaluation of the utility of deep generative histopathology image translation at a mid-sized NCI Cancer Center. bioRxiv. Preprint posted Jan 17, 2020. doi.org/ 10.1101/2020.01.07.897801.

47. Levy, J. J., Azizgolshani, N., Andersen, M. J. Jr., Suriawinata, A., Liu, X., Lisovsky, M., Ren, B., Bobak, C. A., Christensen, B. C. and Vaickus, L. J. 2021. A large-scale internal validation study of unsupervised virtual trichrome staining technologies on nonalcoholic steatohepatitis liver biopsies. Mod. Pathol. 34: 808–822. [Medline] [CrossRef]

48. Rivenson, Y., de Haan, K., Wallace, W. D. and Ozcan, A. 2020. Emerging advances to transform histopathology using virtual staining. BME Frontiers 2020: 11. [CrossRef]

49. Isola, P., Zhu, J.Y., Zhou, T. and Efros, A. A. 2018. Image-to-image translation with conditional adversarial networks. arXiv. Preprint posted Nov 26, 2018. doi. arXiv:1611.07004.

50. Zhu, J.Y., Park, T., Isola, P. and Efros, A. A. 2020. Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv. Preprint posted Aug 24, 2020. doi. arXiv:1703.10593.

51. de Haan, K., Zhang, Y., Liu, T., Sisk, A. E., Diaz, M. F. P., Zuckerman, J. E., Rivenson, Y., Wallace, W. D. and Ozcan, A. 2020. Deep learning-based transformation of the H&E stain into special stains improves kidney disease diagnosis. arXiv. Preprint posted Aug 20, 2020. doi. arXiv:2008.08871.

52. Wang, Y., Yao, Q., Kwok, J. and Ni, L. M. 2020. Generalizing from a few examples: A survey on few-shot learning. arXiv. Preprint posted Mar 20, 2020. doi. arXiv:1904.05046.

53. Robb, E., Chu, W.S., Kumar, A. and Huang, J.B. 2020. Few-shot adaptation of generative adversarial networks. arXiv. Preprint posted Oct 22, 2020. doi. arXiv:2010.11943.

54. Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D. and Batra, D. 2016. Grad-CAM: Visual explanations from deep networks via gradient-based localization. Int. J. Comput. Vis. 128: 336–359. [CrossRef]

55. Sundararajan, M., Taly, A. and Yan, Q. 2017. Axiomatic attribution for deep networks. arXiv. Preprint posted Jun 13, 2017. doi. arXiv:1703.01365.