Development and Validation of an Imaging Mass Cytometry and Multiplex Immunofluorescence Biologically-guided Cellular Segmentation Strategy

STTARR Innovation Centre, MaRS Discovery District, Princess Margaret Cancer Research Tower, 101 College Street, 7th floor, Toronto, ON, M5G 1L7, Canada

University of Toronto, Department of Immunology, Toronto ON Canada

Fred Fu MASc, Veronica Cojocari MSc, Zaldy Balde, Albert Nguyen, Valeria Ramaglia PhD, Salma Sheikh-Mohamed, Mark Zaidi, Jade Bilkey, Susan Done MD, Jen Gommerman PhD, Trevor D. McKee PhD

MS: Manual Classifier

Spatio-Temporal Targeting and Amplification of Radiation Response Program

http://www.sttarr.com STTARR.Image.Analysis@rmp.uhn.ca

Multiplex IF/IMC AnalysisImaging Mass Cytometry and Multiplex

Fluorescence staining approaches, require more stringent approaches for immune

cell image segmentation & classificationLarge panels of antibodies have been common in flow cytometry staining of cells in suspension, allowing for deep cell surface marker interrogation of the immune system.

Novel techniques (stain / strip immuno-fluorescence, or heavy metal-based antibody tagging, followed by imaging mass cytometry) allow highly multiplexed staining to be applied to the analysis of tissue sections.

Segmentation of these high multiplexed image sets to produce single cell counts for downstream analysis is more challenging than single marker studies, due to a strong desire to avoid mixing of signals from neighboring cells, and obtain “pure” readouts of biomarker abundance on a single cell level.

We have adopted a hybrid segmentation approach that utilizes biological domain knowledge to assist in the segmentation and classification of immune cell types in IMC and MxIF images

Two example methodologies are described here, first to adjust segmentation to account for biological differences, and the second to adjust the classification strategy used to identify cell types, overcoming bleedthroughof neighboring cells using a supervised feature-based random forest machine learning approach.

Selection of manually annotated cells to help guide supervised training allows for the reporting of quality metrics with respect to both segmentation (overall number of manually-scored cells versus segmented cells), and classification (per-class proportion of machine-scored immune cell subsets relative to gold standard of manual annotation)

Future work could utilize this approach in a generalized segmentation and classification schema to improve analysis of immune subsets in highly multiplexed immunostainedbiopsies or tissue samples.

bands

Image SegmentationCell Segmentation in IMC assists direct

evaluation of platinum deposition in cisplatin treated pancreatic tumor xenografts

• Imaging mass cytometry with heavy metal tagged antibodies allows for:• highly multiplexed tissue immunostaining • Integrated tissue & cell segmentation utilizing epithelial, stromal markers• Highly quantitative single cell analytics: intensity, spatial relationships

• Finding: Majority of injected platinum actually binds collagen, not tumor cellsPublished: Chang et al, Scientific Reports 2016; 6:36641

γH2AX PanCK EF-5 Collagen DNA IdU EF-5 cisPt Histone-H3E-Cadherin cisPt Histone-H3 α-SMA Epithelial nuclei Stromal nuclei EF-5 (Hypoxia) intensity per cell

Pt deposition intensity per-cell

Multiple views of marker combinations in 28 marker panel. Segmentation & Intensity

Quantitation of per-cell proliferation (IdU), DNA Damage (gH2AX), and cisPt uptake

Laboratory of Dr. David Hedley, Princess Margaret Cancer Centre. Performed in collaboration with Fluidigm Canada Inc., manufacturers of Hyperion IMC system.

Segmenting into bands of equal distance from blood vessels / stromal regions in pancreatic cancer xenografts highlights tumor microenvironment biomarker relationships

Epithelium Stroma Lumen Distance from stroma EF5 (Hypoxia) per-cell EF5 in stromal dtstance bands

0

0.5

1

1.5

2

2.5

3

0 50 100 150 200 250 300

EF5

Sign

al

Distance from Stroma

Hypoxia Distance Map

0

0.5

1

1.5

2

2.5

3

0 50 100 150 200 250 300

Prol

ifera

tion

Sign

al

Distance from Stroma

Proliferation Distance Map

IdU Intensity KI67 Intensity

of Mean IdUU_I127Di of Mean CK7_Dy164Di

of Mean Ki67_Er168Di of Mean p53_Nd150Di

f Mean Pt196_Pt196Di of Mean Vimentin_Gd156Di

Mean Pan_keratin_Dy162Di Mean DNA191_Ir191Di of Mean HH3_Yb176DiBeta_catenin_Ho165Din E_cadherin_Gd158Di of Mean EF5_Tb159Di

0

0.5

1

1.5

2

2.5

3

3.5

4

Closest to stromafurthest from

stroma

Stromal Distance Analysis of Hypoxia

This segmentation method allows for comparison amongst multiple biomarkers to highlight any putative microenvironmental relationships. Combining distance with cellular segmentation can also allow for multidimensional interrogation of biomarker relationships with distance, such as the “tissue cytometry” plot showing individual cell hypoxia and proliferative markers, with color scale denoting distance from each cell to nearest vessel.

Multi-marker stromal distance relationship Tissue Cytometry scatterplot

Zaidi et al 2019

Pure DNA based segmentation can result in combination of biomarkers from neighboring cell types, resulting in implausible immune cell types (e.g. mixed T and B cell markers). By adopting an approach to combine DNA segmentation with biological domain knowledge (i.e. immune surface marker subtypes), we can improve segmentation.

Utilizing Immune Markers to guide Segmentation

Random Forest Classifier

CD3 CD8DNA

IgKIgLDNA

HLA CD68DNA

Putative T Cell

Putative T Cell Putative B Cell

Putative T Cell Putative B CellPutative Macrophage/ Microglia / Other Cell

Biologically guided segmentation

Ramaglia et al, Elife 2019

Immune Non-Immune

T Cytotoxic T Helper

B Cell T Cell

A

B

C

A

B

C

Adapted from Cudrici et al JRRD 2006

MS: Immune cell blood vessel distance

Putative T CellPutative Macrophage

Putative B CellDNA / Other Cell

CD31+ Vessel Distance from Vessel

Immune influx from blood vessels plays an important role in progression of multiple sclerosis. Creation of a distance map utilizing CD31+ vessels permits direct measurement of distance to nearest (in-plane) vessel on a per-cell basis.

Gating strategy using manual annotationsClassification of distinct immune cell subsets was performed by first manually labeling these subsets, then determining manually identified “gates” / thresholds by inspection of 2D scatterplots of markers of interest, to identify positivity cutoffTraining data: Manual labeling 2D Tissue Cytometry plot: CD8+ T cells Proliferative T Cells (manual labels)

Pandas dataframe logical gating strategy to classify T Cell subsets

MS Lesion Classification Schema: Discrete Biological Activity within Lesions

The resulting heatmap was obtained by first segmenting every cell within specific lesions (corresponding to regions of interest with discrete biological activity), then applying the gating strategy determined by manual labeling of discrete immune cell types. The cell subtype count is reported in units of cells / mm2 across each region of interest.

In addition to the immune cell density, the average distance to nearest blood vessel for each immune cell subset is reported here.

Immune Cell Density Heatmap

Ramaglia et al, Elife 2019

Breast cancer: Immune infiltrate classificationKnowledge of the immune content of the tumor microenvironment, and the proportions of distinct regulatory and cytotoxic immune cell components surrounding and within the tumor, is critical for the delivery of optimal therapy. Multiplexed immunofluorescence, using sequential staining, stripping and re-staining, can highlight immune cell subsets. However, challenges remain with segmentation and classification of these cell types. The higher resolution of these images, and greater density of immune cells, made segmentation adjustments more challenging; we optimized the segmentation to balance under- and over-segmented cells; and utilized a Random Forest machine learning strategy to “train” an immune classifier using supervised labeled training data.

Red – CD8 Intensity (THelper Cells)Green – CD4 intensity (TCyt Cells)

■ B cells based on threshold

■Actual B cells (based on manual investigation)

■Actual TC cells (based on manual CD8 investigation)

■ Actual TH cells (based on manual CD4 investigation

■ and ■ Non-immune cells

Red – CD3 Intensity (T Cells)Green – CD20 intensity (B Cells)Yellow – Segmented Cell Border

■B cells (based on gated CD20threshold)

■T cells (based on gated CD3threshold)

■Poorly segmented cells (double positive for both markers)

■Non-immune cells (double negative / below both thresholds)

DNA (Nuclear) and Na-K ATPase (Membrane) Classical Computer Vision Segmentation

Original Image Segmented ImageGrey – DAPI (DNA Intercalator)Green – Na-K ATPase

■Membrane

■ Cytoplasm

■ Nucleus

Poorly Segmented Cells B Cell / T Cell Scatterplot Thelper vs TCytotoxic Scatterplot

Original Image Manually Labeled Cells

Random Forest-based Immune Cell Classifier Computer vision-based segmentation was tuned to balance under- and over-segmented cells (utilizing thresholds for DAPI and membrane markers). For each detected cell, ~50 features were exported, corresponding to pixel intensities for sub-regions and histogram-based intensity subsets. These were used with manually labeled data to train a Random Forest classifier to properly identify immune cell subsets.

Trained Random Forest Cell Classifier Manually Labeled Cells

Relative Feature Importance for Classification Scatterplot of Classified Cells Validation: Confusion Matrix

Trained classifier achieves an accuracy of 0.87, f1-score (macro) of 0.84; Precision of: [0.86, 0.88, 0.74, 0.89], and recall of [0.88 0.88 0.74 0.88] when trained on labeled cells from representative tumor microarray cores. This approach is useful when segmentation is challenging; allowing for extraction of immune cell density data for a number of classes, for better evaluation of the tumor microenvironment.

Red – CD3 Intensity (T Cells) Green – CD20 intensity (B Cells)

Manual Classifier

B Cell 145 165

T Helper 102 126

T Cytotoxic 11 2

Non-Immune 175 141

Total Counts in this ROI

Classifier per-class confidence