High-plex immunofluorescence imaging and traditional histology of the same tissue section for discovering image-based biomarkers

1The microanatomy of fixed and stained tissues has been studied using light microscopy for over two centuries,, and histopathology review of hematoxylin and eosin (H&E) stained tissue sections, complemented by IHC and exome sequencing, remains the primary approach for diagnosing and managing many diseases, particularly cancer. More recently, machine learning and artificial intelligence (ML/AI) approaches have been developed to automatically extract information from H&E images, leading to progress in computer-assisted diagnosis. However, H&E and IHC images generally lack the precision and depth of molecular information needed to optimally predict outcomes, guide the selection of targeted therapies, and enable research into mechanisms of disease.

2The transition of histopathology to digital approaches is concurrent with the introduction, in research settings, of methods for obtaining 10–100-plex imaging data from tissues (e.g., MxIF, CyCIF, CODEX, 4i, mIHC, MIBI, IBEX, and IMC)–. Such approaches combine subcellular resolution morphological analysis with spatially resolved molecular data and are ideal complements to dissociative single cell methods, such as scRNA sequencing. Methods compatible with the type of specimens universally acquired for diagnostic purposes –– formaldehyde-fixed and paraffin-embedded (FFPE) specimens –– also make it possible to tap into large archives of human biopsy and resection specimens. Many high-plex imaging studies performed to date on human cohorts involve tissue microarrays (TMAs; arrays of many 0.3 to 1 mm specimens on a single slide) or the small fields of view characteristic of mass-spectrometry based imaging,, but whole-slide imaging is required for clinical research and diagnosis, both to achieve sufficient statistical power and as an FDA requirement.

3During histopathology review of H&E images, a human expert draws on implicit and explicit knowledge about the abundances and morphologies of cellular and acellular structures prognostic of disease or predictive of drug response. This prior knowledge, summarized in resources such as the American Joint Committee on Cancer’s staging manual, is based on thousands of clinical research papers and numerous clinical trials. In contrast, research using highly multiplexed imaging relies primarily on spatial statistics, which recognizes recurrent patterns in a data-driven manner, but has not yet been subjected to rigorous validation in a clinical setting. An opportunity therefore exists to combine deep knowledge of tissue anatomy (acquired from H&E images) with newly acquired insights into single cell types and states. We reasoned that an ideal instrument for achieving this would perform whole-slide imaging (WSI), have sufficient plex and resolution to distinguish tumor, immune and stromal cell types, and enable reliable and efficient data acquisition with minimal human intervention. In current practice, combining immunofluorescence (IF) and H&E imaging requires the use of different tissue sections. However, collection of same-cell multi-modal images would enable one-to-one comparison of cell morphologies and molecular properties and also facilitate integration with ML/AI approaches being developed for H&E data.

4The relative complexity of existing highly multiplexed imaging assays has slowed their adoption in the clinic; the current standard in clinical research is 5 to 6-plex imaging using a Perkin Elmer Vectra Polaris™ (now Akoya PhenoImager HT™). However, a first-principles analysis suggests that a minimum of 16–20 molecular (IF) channels are required for tumor profiling (Supplementary Table 1): 10–12 to subtype major immune cell types, 2–3 to detect and subtype tumor cells and states, 2–4 to identify relevant tissue structures, 1–3 to examine tumor cells states or therapeutic mechanisms, plus a nuclear stain to locate cell nuclei. Achieving this would benefit from acquisition of many fluorescent channels in parallel (one-shot imaging) rather than the sequential process developed by Gerdes et al. and subsequently extended by our group and others.

5In this paper, we describe the development of an approach to one-shot, whole-slide, 16 to 18-channel immunofluorescence (IF) imaging, followed by H&E staining and imaging of the same cells. We compare the performance of the “Orion™” approach and a commercial-grade instrument that implements it, with established IHC and cyclic data acquisition by CyCIF. Using both human inspection and machine learning (ML) on multi-modal Orion images we demonstrate beneficial transfer of information from H&E images to high-plex IF data (e.g., to distinguish normal tissue from a tumor) and also the other way round (e.g., to subtype immune cells that are indistinguishable in H&E data). In a proof of principle study, we use two independent 30–40-patient human colorectal cancer (CRC) cohorts (n = 74 patients total) to identify spatial biomarkers prognostic of tumor progression having Hazard Ratios of 0.05 to 0.15 (i.e., up to a 20-to-1 discrimination of rapid vs. slow progression). Thus, the Orion method makes multi-modal data accessible and compatible with cohort studies and eventual use in diagnosis.

6We investigated multiple approaches for achieving one-shot high-plex IF followed by H&E imaging of the same tissue section. Overlap in the excitation and emission spectra of most widely used fluorophores limits the number of separable fluorescence channels (typically five to six) that can be accommodated within the wavelengths useful for antibody labeling (~350 to 800 nm). This can be overcome using tuned emission and excitation filters and spectral deconvolution (e.g., of 6 to 10 channels) or by dispersing emitted light using a diffraction grating and then performing linear unmixing,. However, unmixing complex spectra has historically resulted in a substantial reduction in sensitivity and has not been widely implemented. Simultaneous high-plex imaging of tissue specimens therefore required innovation in the optical platform as well as careful selection of fluorophores.

7With support from an NCI SBIR grant, a commercial-grade Orion instrument was developed that utilizes seven lasers (Fig. 1a and Extended Data Fig. 1a) to illuminate the sample and collect emitted light with 4X to 40X objective lenses (0.2 NA to 0.95 NA; Orion data is this paper were collected with a 20X 0.75 NA objective) followed by multiple tunable optical filters that use a non-orthogonal angle of incidence on thin-film interference filters to shift the emission bandpass. These filters have 90–95% transmission efficiency and enable collection of 10 – 15 nm bandpass channels with 1 nm center wavelength (CWL) tuning resolution over a wide range of wavelengths (425 to 895 nm). Narrow bandpass emission channels improve specificity but substantially reduce signal strength; we overcame this problem by using excitation lasers that are ~10 times brighter than conventional LED illuminators and by using a sensitive scientific CMOS detector (camera). Raw image files were then processed computationally to correct for system aberrations such as geometric distortions and camera non-linearity, followed by spectral extraction to remove crosstalk and isolate individual fluorophore signals (and thus, the antibodies to which they were conjugated). The features of single cells and regions of tissue were then computed using MCMICRO software.

8We tested >100 chemical fluorophores from different sources and identified 18 ArgoFluors™ that were compatible with spectral extraction enabled by discrete sampling. Key criteria were: (i) emission in the 500 – 875 nm range; (ii) high quantum-efficiency; (iii) good photostability; and (iv) compatibility with each other in high-plex panels (Extended Data Fig. 1b, Supplementary Tables 1 & 2). ArgoFluor dyes were covalently coupled to commercial antibodies directed against lineage markers of immune (e.g., CD4, CD8, CD68), epithelial (cytokeratin, E-cadherin), and endothelial (CD31) cells as well as immune checkpoint regulators (PD-1, PD-L1), and cell state markers (Ki-67), to generate panels suitable for studying the microenvironment and architecture of epithelial tumors and adjacent normal tissue (Extended Data Fig. 1c; the logic underlying Orion panels is shown in Extended Data Fig. 1d). An accelerated aging test demonstrated excellent reagent stability, estimated to be >5yr at −20°C storage (Extended Data Fig. 1e).

9Because eosin fluoresces strongly at 530 – 620 nm, it proved impractical to perform H&E staining prior to IF (although alternatives to H&E compatible with IF have been described). However, we found that H&E staining could be performed using industry-standard slide stainers after one or a small number of IF cycles. No established methods exists for evaluating the quality of these or other digital H&E images and comparison is complicated by variation in color intensity across platforms. We therefore acquired H&E images using an Aperio GT450 microscope (Leica Biosystems), which is a gold standard for diagnostic applications, rather than the integrated Orion brightfield mode. Four practicing pathologists were shown images of tissue sections that had been subjected to one or more IF staining cycles followed by fluorophore bleaching and asked whether they could distinguish these images from serial section controls that had been stained with H&E in the standard manner in a clinical facility (Fig. 1a). They found the two sets of images to be indistinguishable and “diagnostic grade” (Extended Data Fig. 2a). We conclude that good quality post-Orion H&E imaging can be obtained although further study of additional tissues will be required to fully assess whether they are adequate for use in diagnosis.

10To test the Orion approach, three types of data were collected: (i) whole slide images of human tonsil, a standard tissue for antibody qualification, and human lung cancer, a particularly common cancer type; (ii) images of a TMA that contains 30 different types of normal, non-neoplastic disease, as well as tumor samples from 18 tissues (iii); whole-slide images of 74 stage I-IV colorectal cancer (CRC) resections obtained from the archives of the Brigham and Women’s Hospital Pathology Department (these resections were split into two cohorts with 40 and 34 patients each as indicated in Supplementary Table 3). We tested and optimized the antibody panel on tonsil tissue and then applied it successfully to the lung cancer specimen (Extended Data Fig. 2b), TMA (Extended Data Fig. 2c, Supplementary Table 4), and CRC cohort. We also collected data from a dedicated autofluorescence channel (445 nm excitation / 485 nm emission, CWL) both to extract natural fluorescence from the IF channels and improve biomarker signal to noise ratio (SNR), and to provide information on naturally fluorescent structures such as connective tissues and components of blood vessels (Fig. 1b). In each case, we performed 18–20 plex imaging (16–18 antibody channels, autofluorescence and a nuclear stain) plus H&E. Exploratory studies suggest that it should be possible to add 2–4 additional antibody channels to the method following further optimization of fluorophores and optical systems (see Methods).

11Inspection of whole-slide images of lung, tonsil, and CRC confirmed error-free imaging and stitching of 1,000 or more adjacent tiles (area up to 35 by 20 mm; Fig. 1c) including bright in-focus staining of cellular and cellular substructures within each tile (Fig. 1d). To quantify the effectiveness of spectral extraction, we imaged serial sections of human tonsil tissue each stained with a single antibody conjugated to a different ArgoFluor and then recorded data in all channels. Under these conditions, crosstalk between adjacent channels averaged ~35%. Spectral extraction reduced this to <1% (Fig. 1e). As a result, when a tissue section was subjected to multiplexed antibody labeling, we observed correlated signals only for antibodies that stain targets co-localized on the same types of cells (e.g., co-staining of T-cell membranes anti-CD3e and anti-CD4; Extended Data Fig. 2d).

12The staining patterns observed with ArgoFluor-antibody conjugates were similar to those obtained by conventional IHC performed on the same specimen using the same antibody clones (as described in Du et al., one-to-one comparison of IF and IHC is not possible given fundamental differences in imaging modalities; Fig. 2a and Extended Data Fig. 3a). We also compared Orion data to data acquired from a serial tissue section using a well-established cyclic immunofluorescence method (CyCIF). We found that the fractions of cells scoring positive for the same markers across the two methods were highly correlated (Fig. 2b, 2c shows four examples with ρ = 0.8 to 0.9) except when marker positive cells were rare and cell counts were subject to statistical fluctuation from one serial section to the next (e.g., ρ = 0.55 for FOXP3 positivity; Extended Data Fig. 3b). Moreover, projections of high dimensional CyCIF and Orion data using t-SNE successfully resolved the same immune and tumor cell types (Fig. 2d and Extended Data Fig. 3c).

13To test the repeatability of the method, sample processing and imaging of CRC Cohort 1 (n = 40 specimens) was performed at RareCyte in Seattle WA, and processing and imaging of Cohort 2 (n = 34 specimens from different patients) was performed at HMS in Boston MA on different instruments by different operators; six specimens from Cohort 1 were imaged at both RareCyte and HMS. Corresponding pairs of images from these six specimens looked very similar and when cell count data from all 12 images was subjected to unsupervised clustering, batch effects were not observed (Extended Data Fig. 4a–c). Thus, the Orion method reproducibly generates results that are qualitatively similar to those obtained using conventional IHC and quantitative marker intensities are similar between Orion and CyCIF.

14There are situations in which data from 16–20 fluorescent channels is likely to be insufficient for identifying cell types of interest. We therefore asked whether multiple rounds of Orion data collection could be performed on the same cells using a cyclic approach,. We stained tonsil tissue with 16 ArgoFluor-conjugated antibodies and collected IF, autofluorescence, and nuclear (Hoechst-stained) images. Slides were then subjected to oxidation with hydrogen peroxide (bleaching), stained with 13 additional antibodies (a number based on reagent availability), followed by IF imaging, and then subjected to H&E staining and brightfield imaging. We found that crisp, high SNR second-round images could be obtained using a cyclic approach, yielding a 32-plex Orion image (if same-cell H&E is included; Fig. 2e and Extended Data Fig. 5a). We confirmed that the inter-cycle bleaching step reduced ArgoFluor intensity by >95% and that crosstalk from one cycle to the next was therefore low (Extended Data Fig. 5b). We also established that it was possible to perform multiple rounds of CyCIF after one round of Orion (Extended Data Fig. 5c); multi-cycle CyCIF is slower than second cycle Orion but potentially more flexible. Moreover, although many cycles of IF staining and bleaching reduced H&E image quality, our pathology team judged H&E images collected after two IF and photobleaching steps to be indistinguishable from controls and therefore diagnostic grade (Extended Data Fig. 5d, e). We conclude that two-cycle Orion imaging retains IF and H&E image quality, opening the door to efficient 32–36 plex multi-modal imaging. Exploratory studies suggest room for further development of cyclic and high-plex Orion imaging.

15When same-cell H&E and IF data were compared, we found that molecular labels obtained from IF enabled more complete enumeration of lymphocytes than inspection of H&E images; for example, CD4 and CD8 T cell and B cell lineages look similar by H&E but were distinguishable by IF (arrows in Fig. 3a). We also identified many cell types and cell states that were more readily defined in H&E images based on morphologic features than by immunofluorescence staining; this included eosinophils and neutrophils with distinctive H&E morphology but no lineage markers in our Orion panels, as well as the prophase, metaphase, anaphase, and telophase stages of mitosis (arrows and dashed lines in Fig. 3b). A wide variety of acellular structures such as basement membranes, mucin pools, necrotic domains, etc. were also more readily scored in H&E than IF images. To begin to quantify the amount of complementary information in H&E and IF images, we computed the fraction of all cells (as identified by nuclear segmentation) in the 40-specimen CRC Cohort 1 that could not be assigned a clear identity using IF images; we found that this varied from 6.5 to 42% of total nuclei (median = 16%) (Fig. 3c). We have previously observed a similar fraction of “unidentifiable” cells following 40–60 plex CyCIF imaging and surmised that these cells were either negative for all antibody markers or difficult to segment.

16To identify cells missing labels in Orion IF data, we used a previously published ML model trained on H&E images (see Methods for details of this model and its performance). We found that >50% were predicted to be smooth muscle, stromal fibroblasts, or adipocytes (Fig. 3d); these predictions were confirmed by visual inspection of the H&E images (Fig. 3e). We also examined specimens (e.g., from patient 26, Fig. 3f and Extended Data Fig. 5f) in which a subset of epithelium was difficult to identify by IF because it stained weakly with pan-cytokeratin, E-cadherin, and immune markers. Inspection of H&E images showed that weakly-staining cells corresponded to a serrated adenoma that was distinct from nearby domain of invasive low-grade adenocarcinoma (in which tumor cells stained strongly for pan-cytokeratin and E-cadherin). Differential staining of cytokeratin isoforms in serrated adenoma and adenocarcinoma has been described previously and we speculate that in specimen C26, it reflects clonal heterogeneity. Low staining intensity in the serrated adenoma interferes with IF-based cell type calling when strongly staining adenocarcinoma is also present. From these findings we conclude that the availability of H&E and IF images of the same set of cells substantially increases the fraction of cell types and states that can be identified as compared to either type of data alone. This is particularly true of cell types for which specific molecular markers do not exist (e.g., stromal fibroblasts) or are not included in the panel (e.g., neutrophils) and markers that are lost due to tumor sub-clonality (e.g., specific cytokeratin isoforms). Cells that are highly elongated or have multiple nuclei and are difficult to segment (e.g., muscle cells) are also commonly lost to computational analysis of IF data but highly distinctive in H&E images.

17The classification of cancers for diagnostic purposes using American Joint Committee on Cancer (AJCC/UICC-TNM classification) criteria is based primarily on tumor-intrinsic characteristics (tumor, lymph node, and metastases, the TNM staging system). However, the extent and type of immune infiltration also plays a major role in therapeutic response and survival. In colorectal cancer (CRC) this has given rise to a clinical test, the Immunoscore®, that quantifies features of the intratumoral and tumor-proximal immune response to predict CRC progression as measured by progression-free survival (PFS) or overall survival (OS). The Immunoscore has been validated in multicenter cohort studies and shown to predict time to recurrence in stage III cancers in a Phase 3 clinical trial. The Immunoscore uses IHC to evaluate the number of CD3 and CD8-positive T cells at the tumor center (CT) and the invasive margin (IM; for Immunoscore this is defined as a region encompassing 360 μm on either side of the invasive boundary; in our work we set this to ± 100 μm from the boundary). The hazard ratio (HR; the difference in the rate of progression) between patients with tumors containing few immune cells in both the CT and the IM (Immunoscore = 0) and patients with tumors containing many cells in both compartments (Immunoscore = 4) has been reported to be 0.20, (95% CI 0.10–0.38; p < 10−4) in a Cox regression model, with increasing score correlating with longer survival. This is a clinically significant difference that can be used to inform key treatment decisions: for example, whether or not to deliver adjuvant therapy (chemotherapy after surgery). Because chemotherapy is associated with significant adverse effects, requires infusion or injection in a healthcare setting, and is expensive, it is desirable that patients who are unlikely to experience disease recurrence be spared the burden of adjuvant therapy.

18Using Orion data, we developed software scripts to recapitulate key aspects of the Immunoscore using Progression Free Survival (PFS) as an outcome measure. First, we detected the tumor-stromal interface and generated masks that matched the criteria for CT and IM (± 100 μm around the tumor boundary; Fig. 4a). CD3 and CD8 positivity in single cells was determined by Gaussian Mixture Modeling with the median positive fraction for each marker (CD3 or CD8) in each region (CT or IM) across all 40 CRC cases used as the cutoff for assigning a subscore of 0 or 1; the sum of the four subscores was used as the final score for Image Feature Model 1 (IFM1; Fig. 4b). Parameters for computing IFM1 such as the size of the invasive margin and the staining threshold for scoring cells positive and negative were set a priori (naively) without any parameter tuning to reduce the risk of over-training; IFM1 nonetheless yielded a hazard ratio similar to Immunoscore itself on Cohort 1 (HR = 0.14; 95% CI 0.06–0.30; p = 7.63 × 10−5) (Fig. 4c), Next, we used the underlying logic of Immunoscore to leverage multiple Orion channels. A total of 13 immune focused markers were used to generate ~15,000 marker combinations (IFMs), each composed of four markers within the CT and IM domains (Fig. 4d). Scores for each CRC case were binarized into high and low scores based on median intensities (again, without any parameter tuning). When HRs were calculated we found that nearly 600 IFMs exceeded IFM1 in performance (Extended Data Fig. 6a–c). The top 10 IFMs were insignificantly different from each other, and we chose one (IFM2) for further analysis; it exhibited an HR = 0.05 (95% CI: 0.02–0.10, p = 5.5 × 10−6) (Fig. 5a) and comprised the fractions of α-SMA+ cells in the CT, and CD45+, PD-L1+, and CD4+ cells in the IM. Leave-one-out resampling showed that IFM2 was significantly better than IFM1 with respect to HR (adjusted p value based on the Benjamini-Hochberg Procedure padj = 7.3 × 10−21; Fig. 5b, Extended Data Fig. 6d). To determine whether IFM2 is generalizable, we tested the performance of this model created using Cohort 1 on specimens in Cohort 2. Once again, we observed a statistically significant discrimination between progressing and non-progressing tumors (HR = 0.17; 95% CI: 0.05 to 0.56, p = 6.9 × 10−3; Fig. 5c). We conclude that multiplexed immunoprofiling data extracted from Orion images of CRC resections can be used to generate performant prognostic biomarkers.

19Inspection of images from IFM2 tumors exhibiting slow progression (e.g., patient C34) revealed high-levels of PD-L1+ cells (Fig. 5d, Extended Data Fig. 6e) adjacent to pan-cytokeratin positive tumor cells; based on overlap of PD-L1+, CD68 and CD45 staining we conclude that PD-L1+ cells are likely myeloid in origin, as described previously. In C34, α-SMA stained tumor proximate stromal cells – most likely fibroblasts – were also well-infiltrated with CD4+ T cells. By contrast, in a patient with rapid progression (i.e., patient C09), PD-L1 levels were below the level of detection and CD4+ cells were less abundant in the stroma. By H&E, IFM2-high tumors exhibited extensive lymphohistiocytic chronic inflammation including large lymphoid aggregates and tertiary lymphoid structures (TLS) at the tumor invasive margin, whereas IFM2-low tumors had relatively few lymphoid aggregates and no TLS (Fig. 5d and Extended Data Fig. 6e). Although IFM2-low tumors were also more invasive than IFM2-high tumors, IFM score was independent of histologic subtype (e.g., conventional vs. mucinous morphology) and did not correlate with histologic grade (low vs. high grade carcinoma). Thus, IFM2 is likely to capture activity of the immune microenvironment around the invasive tumor margin as well changes in tumor-associated fibroblasts. However, deeper phenotyping of more specimens will be required to identify which molecular features of IFM2 are important for predicting progression.

20As an unbiased means of identifying new progression models, we used spatial Latent Dirichlet Allocation (Spatial-LDA). Spatial LDA can reduce complex assemblies of intermixed entities into distinct component communities (“topics”) while accounting for uncertainty and missing data; it has performed well on other multiplexed tissue imaging datasets,. We separated CRC specimens in Cohort 1 into tumor and adjacent normal tissue using H&E data and a ML/AI model and then performed spatial LDA at the level of individual IF markers on cells in the tumor region (Fig. 6a). This yielded 12 spatial features (topics) that recurred across the dataset (the number of topics was optimized by calculating the perplexity; see Methods for details) (Extended Data Fig. 7a). Visual inspection of images by a pathologist confirmed that marker probabilities matched those computed for different topics and that the frequency distribution of each topic varied, sometimes substantially, among CRC samples (Fig. 6b and Extended Data Fig. 7b). The strongest correlations between topics and PFS for Cohort 1 were found to be −0.52 (p < 0.001) for Topic 7, comprising pan-cytokeratin and E-cadherin positivity (with little contribution from immune cells) and +0.57 (p < 0.001) for Topic 11, comprising CD20 positivity with minor contributions from CD3, CD4, and CD45 (Fig. 6b–6f and Extended Data Fig. 7a). In contrast, topics involving the proliferation marker Ki-67+ (Topic 6), PD-L1 positivity (Topic 9), or immune cells markers (CD45+ or CD45RO+; Topics 3 and 10) exhibited little or no correlation with progression-free survival (Extended Data Fig. 7a).

21Given the correlation of Topic 7 with PFS, we constructed a Kaplan-Meier curve for tumors having a proportion of Topic 7 below the 50th percentile versus those above this threshold (including all cells in the specimen; note that the value of the threshold was not critical over the range of 50% – 75% (Fig. 7a and Extended Data Fig. 8a). Imposing a 50th-percentile threshold on Topic 7 yielded model IFM3 which, on Cohort 1, resulted in HR = 0.26 (Fig. 7a; CI 95%: 0.11 – 0.63; p = 2.98 × 10−4). When we tested IFM3 on Cohort 2 we observed even better performance (HR = 0.07; CI 95%: 0.02 – 0.24; p = 5.6 × 10−4; Fig. 7b), suggesting that the model had not been over trained. We conclude that spatial-LDA had discovered – via unsupervised analysis of high-plex IF data – a tumor-intrinsic feature distinct from immune infiltration that was significantly associated with poor patient survival.

22Poor interpretability is a common limitation of models generated using ML methods such as spatial LDA. In the case of Topic 7, the primary molecular features were pan-cytokeratin and E-cadherin positivity, but Topic 8 was similar in composition while exhibiting no correlation with PFS (r = 0.01; Fig. 6c, 6f and Extended Data Fig. 7a). To identify the tumor histomorphology corresponding to these topics, we transferred labels from IF to the same section H&E images, trained a convolutional neural network (CNN) on the H&E data, and inspected the highest scoring tumor regions (Extended Data Fig. 8b). Topic 7 was readily identifiable as poorly differentiated/high-grade tumor with stromal invasion (Fig. 7c and 7d). In contrast, Topic 8 consisted predominantly of nearly normal intestinal mucosa with some areas of well-differentiated tumor (Fig. 7c and Extended Data Fig. 8c). When we inspected Orion and CyCIF images of specimens with a high proportion of Topic 7 (e.g., patient C06, Extended Data Fig. 9) we found that the E-cadherin to pan-cytokeratin ratios were low relative to normal mucosa or Topic 8 (expression of Na,K-ATPase, another protein found on the plasma membranes of colonic epithelial cells, was also low). These are features of cells undergoing an epithelial-mesenchymal transition (EMT), which is associated in CRC with progression and metastasis. However, follow-on CyCIF imaging showed that some features of EMT, such as low proliferation and increased expression of EMT-associated transcriptional regulators (e.g., ZEB1), were not generally observed in Topic 7-positive cells. For example, the proliferation index in tumor cells was high (40–50% Ki67 and PCNA positivity) and staining for ZEB1 was low (even though ZEB1 was easily detected in nearby stromal cells with mesenchymal differentiation – compare yellow and white arrows; Extended Data Fig. 9). Thus, even though the molecular and morphological features of Topic 7 were consistent with each other, H&E morphology was more readily interpretable with respect to long established features of CRC progression. Interpretability increases confidence in a potential biomarker and substantially improves its chances of clinical translation.

23Only about one-third of patients in Cohort 1 scored high for IFM1 and low for IFM3 (the combination correlated with the longest PFS; Fig. 7e) arguing for use of a composite model (IFM4; HR = 0.12; 95% CI = 0.05 to 0.28; p = 6.7 × 10−7) (Fig. 7f). Statistically significant results were also obtained from Cohort 2 using a model trained on Cohort 1 (Fig. 7g). This demonstrates that immunological and tumor-intrinsic features of cancers can be combined to generate prognostic models with high predictive value. Although no parameter tuning (e.g., setting thresholds for positivity) was involved in the generation of IFMs 1–4, experience with Immunoscore shows that parameter tuning using larger cohorts of patients can further boost performance.

24In this paper, we describe an approach to multimodal tissue imaging that combines one cycle (single-shot) high-plex, subcellular resolution IF with imaging of endogenous fluorescence and subsequent H&E imaging of the same cells. We show that such multimodal tissue imaging is reproducible across performance sites and has substantial benefits for human observers and ML models. Most obviously, it facilitates the use of historical knowledge about tissue microanatomy (derived from H&E images) for the interpretation of molecular data derived from multiplexed molecular imaging. We find that human experts and ML algorithms can exploit H&E images to classify cell types and states that are not readily identifiable in multiplexed data given inevitable limitations in antibody variety. H&E and autofluorescence imaging are also effective at characterizing acellular structures that organize tissues at mesoscales (e.g., the elastic lamina of the vessel wall). Moreover, by overlaying molecular data on H&E images we show that it is possible to discriminate cell types that have similar morphologies but different functions. The ability of molecular data to label cell types in H&E images is expected to be advantageous in supervised learning for ML/AI modeling as well as the use of H&E data to unpack “black box” ML models trained on molecular data. The interpretability of AI models is thought to be important in medicine in as means of promoting uptake, increasing generalizability across cohorts, and ensuring compliance with ethical standards.

25The Orion instrument described here supports up to 18-plex simultaneous data acquisition (including DNA and one or more autofluorescence channels), but it is likely that several additional channels can be added as fluorophores are more optimally matched to available lasers and optical elements. Our data demonstrate 18-plex Orion imaging of 30 types of cancer, diseased tissues, and normal tissues available as TMA cores or whole slide specimens, demonstrating that the Orion method is widely applicable. Of course, the antibody panel used for CRC will not be optimal for all tissues, but substitution of a few antibodies is expected to yield panels usable with many cancers of epithelial origin. The only practical limitation to assembling additional Orion panels using commercial antibodies developed for IHC and IF imaging of tissues is the time needed to prepare ArgoFluor-labelled antibodies and test panel performance and stability.

26We show that it is possible to perform cyclic data acquisition using the Orion approach as well as Orion followed by CyCIF, thereby increasing the number of molecular channels dramatically. Cyclic Orion is particularly well suited to discovery research in which 20–40 plex imaging is increasingly common. However, H&E staining must be performed after all IF is complete, and we find that H&E image quality goes down as IF data acquisition extends beyond 2–4 cycles. For diagnostic applications our data suggest that image-based prognostic tests may require only a subset of the channels available to Orion (speculatively 8–14 channels) with attendant reductions in test complexity and cost.

27It is not surprising that multiplexed IF data add information to H&E images. More surprising are the many cell types and structures that are difficult to identify in multiplexed images and readily identified in H&E images by histopathologists or ML algorithms. This includes acellular structures, cell types for which good markers are not readily available, highly elongated and multi-nucleated cells that are difficult to segment with existing algorithms (e.g., muscle cells), and – most remarkably – tumor cells themselves. Many tumor types lack a definitive cell-type marker, and even when such markers are available, some cells in a tumor are observed to express these markers poorly or not at all, likely due to sub-clonal heterogeneity. In contrast, pathologists are skilled at identifying dysplastic and transformed cells in H&E images based on morphology and this is potentially more reliable than IF imaging using molecular markers for the identification of some types of tumor cells. Conversely, many immune cell types cannot be reliably differentiated using H&E images, and their presence can also be difficult to discern when cells are crowded; the use of IF lineage markers provide critical new information in these cases.

28There are multiple ways to exploit the complementary strengths of H&E and IF imaging using ML approaches. ML models trained on H&E data can increase the number of identifiable cells in multi-modal images relative to multiplexed IF data alone. Conversely, IF images can be used to automatically label structures in H&E images (e.g., immune cell types) to assist in supervised learning on these images. This is a significant development because the labor associated with labeling of images – currently by human experts – is a major barrier to the development of better ML models. Finally, multi-modal data can provide molecular insight into the features of tissue images to which ML models “attend”. We anticipate many opportunities for joint learning from H&E and IF data using adversarial, reinforcement, and other types of ML/AI modeling for research purposes, development of biomarkers, and analysis of clinical H&E data at scale.

29A surprising number of pathology workflows involve staining serial sections of a specimen with one IHC biomarker each; the Orion approach can simplify such workflows to single section imaging. For example, Immunoscore is a pathology-driven clinical test that uses H&E and multiple IHC sections to determine the distribution of specific immune cell types at the tumor margin and predict outcome (time to recurrence) for patients with CRC. In this paper, we reproduced the logic of Immunoscore with Orion data plus automated scripts and show that it is possible to improve upon it using additional immune markers (as measured by Hazard Ratios computed from PFS data; see limitations section below). In a distinct but complementary approach, we show that IF data and spatial modeling (LDA) can be used to identify cell neighborhoods significantly associated with CRC progression. The top-performing feature in this case is tumor-cell intrinsic and based on the distributions of cytokeratin and E-cadherin, two epithelial cell markers. Precisely why this feature is prognostic is unclear from IF data alone: other features involving similar markers are not predictive. However, inspection of corresponding H&E data (and training of an ML model) showed that LDA had identified local tumor morphologies typical of poorly differentiated/high-grade tumor with stromal invasion, increasing our confidence in the model. The immediate availability of Orion as a commercial platform and our use of open-source software and OME (Open Microscopy Environment) and MITI (Minimum Information about Tissue Imaging) compliant data standards makes further development of these approaches straightforward.

30Although the images in this paper represent a large dataset by the standards of high-plex whole-slide IF imaging, the number of specimens and the composition of the cohort is insufficient for IFMs to be considered validated biomarkers or clinical tests. Systematic meta-analysis has identified a range of factors that negatively impact the reliability and value of prognostic biomarkers, particularly those based on new technology and multiplexed assays. In the current work, specific limitations include a relatively small cohort size, the absence of pre-registration, and the acquisition of specimens from a single institution. As a result we do not fully control for all relevant covariates (e.g., depth of invasion, sex, age, race, clinical stage, etc.), and more progressors were included in our cohort than would be observed in an unselected population, biasing the cohort to more serious disease (~50% two-year disease-free survival for Stage III colon cancer in our cohort v. an accepted value of ~80%). These and others are addressable with more diverse sets of tissue blocks, and we anticipate that it will be feasible to progress in a few years to validated clinical tests that can be added to colorectal cancer treatment guidelines, substantially improving opportunities for personalized therapy.

31The research described in this manuscript complies with all relevant ethical regulations and was reviewed and approved by the Institutional Review Boards (IRB) at Brigham and Women’s Hospital (BWH), Harvard Medical School (HMS), and Dana Farber Cancer Institute (DFCI). Formaldehyde-fixed and paraffin-embedded (FFPE) tissue samples were used after diagnosis and informed written patient consent under Dana-Farber Cancer Institute IRB protocol 17–000. Two cohorts from same biobank were assembled, the first with 40 patients with state II-IV CRC, then the second with 34 patients with no consideration for the sex of the participants. All samples were collected at the time of initial diagnosis.

32Blocks of FFPE tonsil (AMSBIO, cat# 6022CS) and lung adenocarcinoma (AMSBIO, cat# 28004) as well as colorectal adenocarcinomas from the BWH Pathology Department archives were cut at 5 μm thickness using a rotary microtome and the sections were mounted onto Superfrost™ Plus microscope glass slides (Thermo Fisher, Catalog No.12-550-15). The slides were dried at 37°C overnight and baked at 59°C for one hour. Slides were stored at 4°C until use.

33The Orion instrument is designed to work with an optimized ArgoFluor dyes (RareCyte Inc.) whose emission peaks cover the spectrum from green to far-red (Supplementary Table 2). Although the instrument can also be used with other commercially available dyes, ArgoFluor dyes have been strategically chosen based on resistance to photobleaching, narrow excitation and emission spectra, and high quantum efficiency. To date, RareCyte has optimized 18 ArgoFluor dyes, with others in development.

34Antibodies (listed in Supplementary Table 2) were obtained from vendors in carrier-free PBS and conjugated directly to either biotin for α-SMA, digoxygenin for pan-cytokeratin, or ArgoFluor™ dyes using amine conjugation chemistry. Labeling efficiency was determined using absorbance spectroscopy and the conjugated antibodies were diluted in PBS-Antibody Stabilizer (CANDOR Bioscience GmbH, Catalog No. 130050) to a concentration of 200 μg/mL.

35Slides were de-paraffinized and subjected to antigen retrieval for 5 minutes at 95°C followed by 5 minutes at 107°C, using pH8.5 EZ-AR 2 Elegance buffer (BioGenex, Catalog No. HK547-XAK). To reduce tissue autofluorescence, slides were placed in a transparent reservoir containing 4.5% H2O2 and 24 mM NaOH in PBS and illuminated with white light for 60 minutes followed by 365 nm light for 30 minutes at room temperature as previously described. Slides were rinsed with surfactant wash buffer (0.025% Triton X-100 in PBS), placed in a humidified stain tray, and incubated in Image-iT™ FX Signal Enhancer (Thermo Fisher, Catalog No. I36933) for 15 minutes at room temperature. After rinsing with surfactant wash buffer, the slides were placed in a humidity tray and stained with the panel of fluor- and hapten-labeled primary antibodies in PBS-Antibody Stabilizer (CANDOR Bioscience GmbH, Catalog No.130 050) containing 5% mouse serum and 5% rabbit serum for 2 hours at room temperature. Slides were then rinsed again with surfactant wash buffer and placed in a humidified stain tray and incubated with Hoechst 33342 (Thermo Fisher Catalog no. H3570), ArgoFluor™ 845 mouse-anti-DIG, and ArgoFluor™ 875-conjugated streptavidin in PBS-Antibody Stabilizer containing 10% goat serum for 30 minutes at room temperature. The slides were rinsed a final time with surfactant wash buffer and PBS, coverslipped with ArgoFluor™ Mounting Media (RareCyte, Inc.) and dried overnight.

36Antibody accelerated-aging studies were performed to determine ArgoFluor™-antibody conjugation stability. Reagent stability was measured using the ratio of quantitative metrics obtained with the accelerated condition (21.6°C) to those obtained with the storage condition (−20°C). For testing on tissues, single-cell mean fluorescence intensity (MFI) data obtained by imaging FFPE tonsil stained with an ArgoFluor™ conjugate was gated using a Gaussian mixture model to obtain the percent of positive cells and S:B values (S and B refer to the MFI of cells with values above (S, Signal) and below (B, Background) the gated threshold). To perform fluorophore stability assessment, bead MFI data was obtained by imaging Ig-capture beads incubated with (S) or without (B) the ArgoFluor™ conjugate. For binding stability assessment, data from peripheral blood mononuclear cells (PBMC) stained with the ArgoFluor conjugated antibody was assessed in a fluorescence activated cell sorter and then manually gated to obtain % Positive and S:B values.

37In the Orion instrument ArgoFluor™-conjugated antibodies along with Hoechst dye and tissue autofluorescence were excited by seven laser lines, ranging from 405 to 730 nm (Supplementary Table 2). To separate the overlapping emission spectra, images were captured through a set of nine bandpass filters, which can achieve a tunable narrow band detection window (10 – 15 nm) throughout the spectrum from 425 nm to 894 nm. For a specific sample, the detection bands were chosen to optimize color separation, implemented with RareCyte Inc.’s Artemis™ software. Tuning of emission filters is based on the well-known fact that the spectrum of a thin-film interference filter shifts toward shorter wavelengths when the angle of incidence shifts away from 0 degrees (orthogonal to the filter surface). The filters were motorized such that any narrow band of 10 – 15 nm can be achieved across the entire fluorescence spectrum. Narrow bandpass emission channels improve specificity; the resulting lower signal is overcome by using high power excitation lasers, which yield power at the sample plane ranging from 270 to 600 mW, more than 10 times greater than a typical fluorescence microscope.

38High-plex imaging exploits the fact that the greater the number of features collected, the greater the ability to distinguish lineages and states at a single-cell level. The ability of the Orion imaging platform to discriminate among multiple antibody-fluorophore conjugates is dependent on the degree of spectral overlap among the fluorophores, the intensity and spectral profile of overlapping autofluorescence or background signals, and the difference between the most intense staining of highly expressed proteins and the weakest stain of low abundance proteins. Panel design with the Orion platform involves assigning biomarkers to channels with the appropriate sensitivity ranges while managing spectral overlap between markers that are co-localized. Orion imaging technology is compatible with 20-plex one-shot fluorescence image acquisition (19 antibodies plus Hoechst nuclear stain) and the necessary research into ArgoFluor is ongoing to achieve this on a routine basis. In the current work we found that 17-plex antibody panels were easier to achieve at an acceptable SNR given the endogenous fluorescence of tonsil and CRC tissue. We anticipate that, with relatively few additions and substitutions, the panel we developed for CRC will work well other common tumor types (e.g., lung, breast, melanoma). In cases in which more precise immunophenotyping is desired, a second cycle Orion panel of similar complexity is possible. The prognostic image feature models we describe in this paper could be acquired using as few as 8–12 channels. Thus, optimal Orion imaging and staining strategies in both a research and clinical setting are likely to rely on the use of both pre-set, high-plex, and lower-plex, lower-cost, “mix and match” panels.

39Whole slides were scanned using the Orion instrument using acquisition settings optimized for the specific antibody panels. Briefly, acquisition channel parameters were defined for each biomarker plus an additional channel dedicated to tissue autofluorescence, and included excitation laser, emission center wavelength (CWL), and exposure times. The nuclear channel was scanned at low resolution to identify tissue boundaries, followed by surface mapping at 20X to find the tissue in the z-axis. Whole tissue was acquired at 20X following the surface map within the specified tissue boundaries by collecting all channels for a single field of view (FOV) before proceeding to the next partially overlapping FOV. Raw image files were processed to correct for system aberrations, and signals from individual targets were then isolated using the Spectral Matrix obtained with control samples, followed by stitching of FOVs to generate a continuous open microscopy environment (OME) pyramid TIFF image.

40After Orion imaging was complete, slides were de-coverslipped by immersion in 1x PBS at 37°C until the coverslips fell away from the slide. Slides were rinsed in distilled water for 2 minutes, then stained by a routine regressive H&E protocol using Harris Hematoxylin (Leica, Catalog No. 3801575) and alcoholic eosin Y (Epredia, Catalog No. 71211). Coverslipping was performed with toluene-based mounting media (Thermo Scientific, Catalog No. 4112). After drying for 24 hours, slides were scanned on an Orion system in brightfield mode, using the same scan area used for IF image acquisition. H&E images were also acquired using an Aperio GT450 microscope (Leica Biosystems), and the H&E images were registered to the IF images using ASHLAR and PALOM v2022.3 (https://github.com/labsyspharm/palom) software.

41H&E images were annotated by a board-certified anatomic pathologist (SC and SS) in OMERO PathViewer software on whole slide images according to morphologic criteria that included: normal mucosa, hyperplastic mucosa, adenomatous mucosa (tubular or serrated), invasive adenocarcinoma (tumor), lymphovascular invasion (LVI), peri-neural invasion (PNI), secondary lymphoid structures/Peyer’s patches (SLS), tertiary lymphoid structures (TLS), lymphoid aggregates (without identifiable germinal center formation), lymph nodes. Tertiary lymphoid structures were morphologically defined by the presence of a lymphoid aggregate with germinal center formation and an anatomic distribution and appearance inconsistent with a secondary lymphoid structure (Peyer’s patch or lymph node).

42Tissue-based cyclic immunofluorescence (CyCIF) was performed as previously described following the methods available in protocols.io (dx.doi.org/10.17504/protocols.io.bjiukkew). Data from specimens C01-C17 was acquired as previously reported and computed cell counts were compared in this study with cell counts derived from Orion images of adjacent sections from the same specimens. A BOND RX Automated Slide Stainer (Leica Biosystems Inc.) was used to bake FFPE slides at 60°C for 30 minutes. Dewaxing was performed using Bond Dewax solution at 72°C, and antigen retrieval was performed using BOND Epitope Retrieval Solution 1 at 100°C for 20 minutes. Slides then underwent multiple cycles of antibody incubation, imaging, and fluorophore inactivation to perform the CyCIF process. All antibodies were incubated overnight at 4°C in the dark. Slides were stained with Hoechst 33342 for 10 minutes at room temperature in the dark following antibody incubation in every cycle. Coverslips were wet-mounted using 200μL of 10% Glycerol in PBS prior to imaging. Images were taken using a 20X objective (0.75 NA) on a CyteFinder™ slide scanning fluorescence instrument (RareCyte Inc.). Fluorophores were inactivated by incubating slides in a 4.5% H2O2, 24mM NaOH in PBS solution and placing under an LED light source for 1 hr. To perform CyCIF following Orion imaging, slides were immersed in 1x PBS at 37°C until the coverslips fell away from the slide. The standard CyCIF method was subsequently performed on these slides.

43FFPE sections were de-paraffinized, dehydrated, and endogenous peroxidase activity was blocked. Antigen retrieval was performed for 20 minutes at 100°C, at pH9, using BOND Epitope Retrieval Solution 2 (Leica Biosystems). Detection was achieved using a Bond Polymer Refine Detection® DAB chromogen kit and counterstained with hematoxylin. Slides were scanned using a RareCyte CyteFinder instrument. Primary antibodies used in immunohistochemistry are listed in Supplementary Table 2.

44Stitching, channel registration, illumination and geometric distortion correction was performed with Artemis software on the Orion platform and single-cell data analysis then performed using MCMICRO modules including UNMICST2 with cell masks that involved 5-pixel dilation of the nucleus mask. Mean intensity of each channel and morphological features were quantified for each cell.

45Tonsil FFPE sections stained with single antibody-ArgoFluor underwent standard acquisition and extraction process using the Orion instrument. The pixel intensities of all 18 channels from 17 samples were used to quantify bleed through of a given antibody-ArgoFluor complex to the other channels before and after spectral extraction. 18-plex tonsil image. Pearson’s correlation coefficients between all channel pairs were computed using pixel intensities in the 18-plex tonsil image before and after spectral extraction.

46IFM1 was designed to replicate the logic of the Immunoscore method and was calculated in a semi-automated manner using Orion data. In brief, quantitative data of tumor and immune markers (pan-cytokeratin, CD3e, and CD8a) were gated for high and low cells using a Gaussian Mixture Model (GMM) and confirmed by inspection. After gating, the pan-cytokeratin+ cells were then used to generate tumor masks using a K-Nearest Neighbor (KNN) model (kernel size = 25 cells). The tumor margins were derived from tumor masks by expanding 100 microns in either direction from the point of stroma-tumor contact. The CD3+ and CD8+ fraction, defined as marker positive cells divided by the total of all successfully segmented cells of all types in either the tumor center (TC) or invasive margin (IM). Tumor and margins were enumerated independently in each sample. The median values of all samples were used as a cutoff to defined a subscore as follows: below the median scored as 0 and above the median scored as 1. The final IFM1 value was calculated by adding together the subscores for CD3 and CD8 positive cells in the TC and IM regions (see Fig. 4b for a flow diagram). The IFM1 score therefore ranged from 0 (CD3+ and CD8+ low in both regions) to 4 (CD3+ and CD8+ high in both regions). Similar logic was used to generated other combinations of IFMs. 13 selected immune markers (CD3, CD8, CD45, CD45RO, CD68, CD163, CD4, CD20, α-SMA, FOXP3, PD-1, PD-L1) were gated as described above, and 26 parameters (each marker in the tumor or tumor/stromal interface regions) were generated. The complete combination of 4 out 26 parameters was tested against PFS days for Hazard Ratio (HR). IFM2 was the 3rd best IFM among those combinations, excluding the 1st and 2nd best combinations which had some of the same markers as IFM1 (i.e., CD3 and CD8); the difference in performance between the top performing models was insignificant.

47In the LOO test, the ranks of IFM1 and IFM2 were recalculated with the 40 set of samples (n = 39); each set left out one sample from the original cohort. The collections of ranks from IFM1 and IFM2 were then tested with pairwise t-test. For bootstrapping, the 500 set of randomly selected samples were used to recalculate the hazard ratios of IFM1 and IFM2 as described above. The collections of hazard ratios from IFM1 and IFM2 were then tested with the pairwise t-test. To adjust for multiple hypotheses, the Benjamini-Hochberg Procedure was used with FDR = 0.1.

48Latent Dirichlet Allocation (LDA) was used to compute spatial neighborhoods as described. First, each sample was divided into “grids” of 200 microns, and marker frequency was calculated in each grid. The summarized probabilities of all samples were then used to generate the LDA model with 12 topics using collapsed Gibbs sampling in MATLAB. The optimal topic number was determined via varying numbers (between 8 to 16) of topics and evaluating the goodness-of-fit by calculating the perplexity of a held-out set. After fitting a global LDA model, the individual samples were then applied with the same models to assign topics at the single-cell level.

49A publicly available DenseNet161 model (https://doi.org/10.1101/2021.12.23.474029) trained with the 100K CRC H&E dataset (https://doi.org/10.5281/zenodo.1214456) was used to classify the post-Orion H&E image patches (112 μm2) for all the CRC samples. WSI patch prediction was performed with TIAToolbox v1.1.0 (https://doi.org/10.1101/2021.12.23.474029) on a Windows PC with Nvidia GeForce GTX 1080 graphics card and using batch size = 32. Model performance was reported as F1 = 0.992. The model has nine output classes: adipose (ADI), background (BACK), debris (DEB), lymphocytes (LYM), mucus (MUC), smooth muscle (MUS), normal colon mucosa (NORM), cancer-associated stroma (STR), colorectal adenocarcinoma epithelium (TUM). Scripts for reproducing the inference results can be found at https://github.com/labsyspharm/orion-crc). The transfer learning of a GoogLeNet model was done as follows. First, image patches 224 × 224 pixels were generated from post-Orion H&E images, and the LDA topics were assigned to each patch using Orion data. To exclude low confidence training data, only patches having more than 20 cells and a dominant topic of >60% were used in the analysis. The selected patches were than separated into training, validation, and test sets as the ratio 0.6:0.2:0.2. The training was done with MATLAB (version 2019b) and the results are shown in Extended Data Fig. 8b. Training parameters are listed at Supplementary Table 5.

50Each multi-modal image of a tissue section is unique because H&E stained specimens cannot be re-analyzed by IF. This is a common limitation of high-plex tissue imaging methods in which destructive data collection or tissue damage limits re-acquisition of data. Reproducibility of the Orion approach was therefore estimated by performing imaging on serial sections using two instruments in two different locations (Seattle and Boston) as shown in Extended Data Fig. 4 and by comparing Orion with CyCIF images of serial sections as shown in Extended Data Fig. 4. Image analysis assumed multi-modal log-normal intensity distributions separable using Gaussian mixture models but this was not formally tested for all specimens. Image feature models were trained on a 40-specimen cohort and tested on an independent of cohort of 34 specimens from different patients. Specimen sample number was limited by the availability of tissue blocks from the BWH pathology archives. No statistical method was used to predetermine sample size and no data were excluded from the analyses. The experiments were not randomized and the investigators were not blinded to allocation during experiments and outcome assessment. These and other limitations of this study are described in the main text under “limitations.” Further information on research design is available in the Nature Research Reporting Summary linked to this article.

51For all survival analyses, we used a combined survival endpoint of progression-free survival that encompasses both time to disease recurrence for patients who underwent curative-intent resections (disease-free survival; PFS) and time to progression for patients with measurable disease (progression-free survival; PFS); we used PFS in this paper because it is more familiar. Outcome analysis was performed using Kaplan-Meyer estimation and log-rank test utilizing the MatSurv function in MATLAB. Cutoffs for IFM1, IFM2, and IFM3 were selected at the median value of the entire cohort, and cutoff for IFM4 were selected based on IFM1 & IFM3 as described. Hazard ratios and confidence intervals were calculated with the log-rank approach: HR = (Oa/Ea)/(Ob/Eb), where Oa & Ob are the observed events in each group and Ea & Eb are the number of expected events.