Machine learning and image-based profiling in drug discovery

1Using complex phenotypes to predict functions of genes has been a powerful approach in genetics and clustering mutations by their organismal phenotypes has yielded important insights into functions of many genes and pathways. Visual phenotypes in genetic screens were usually manually scored and phenotypes clustered by similarity. These approaches yielded fundamental insights into the components and architecture of many conserved signaling pathways that have later been found to be recurrently mutated in many diseases,,.

2More recently, automated microscopy-based phenotyping has become a powerful method to infer functions and functional relationships of genes, to investigate cellular or organismal structure, behavior and disease mechanisms,,. Automated imaging-based methods comprise a broad range of qualitative and quantitative strategies to measure phenotypes in various systems ranging from unicellular organisms to cell lines and whole animals,,,,,, however, challenges are often similar, from standardization of imaging conditions to an automated approach to analyze very large data sets.

3In drug discovery, automated imaging was used in a number of different applications in pre-clinical development and has been shown to allow scalable and systematic phenotypic profiling of small molecules, establishing itself as a complementary method to target-based in-vitro screening,,. In this review, we will summarize the current state of the art with a focus on recent publications that highlight novel analysis methodologies for image-based screening. While most methods are applicable in any perturbation-based screens, this review in particular highlights applications in chemical genetics and drug discovery.

4For image-based phenotyping of perturbations, one can distinguish two approaches to experimental design that fundamentally differ with regards to probes as well as image analysis,,.

5The first approach includes screening applications (often called high-content or phenotypic screening) that are focused on pre-defined, specific phenotypes with the aim to identify drugs or drug targets that modulate it (Figure 1A). Such image-based phenotypic screens have, for example, been successfully used to identify targets and compounds that modulate phenotypes like the subcellular localization of specific proteins,.

6The second application of high-throughput imaging in drug discovery is the more global profiling of perturbations. This approach profiles cells upon exposure to genetic, pathogenic or chemical perturbations and is complementary to techniques like transcriptional profiling (Figure 1B,,, ). Subcellular structures are stained with multiplexed fluorescent dyes and fluorescently labeled antibodies to visualize or ‘paint’ cells and subcellular structures. Automated image acquisition and analysis are subsequently used to profile phenotypes of cells in an unbiased manner (Figure 2, Box 1,, ). Computer vision can extract multivariate feature vectors of cell morphology such as cell size, shape, texture and staining intensity without further human intervention (Figure 2A). All large-scale studies reported so far employed segmentation approaches to accurately define cellular outlines prior to feature extraction. The derived profiles of single cells or cell populations are then scored to find relationships within data sets comprising up to tens of thousands of perturbations (Figure 2B,,, ).SegmentationHigh-throughput small molecule profiling is typically based on the staining of subcellular structures and fluorescence microscopy. A selective staining of abundant subcellular structures such as DNA or actin and the subsequent imaging in distinct channels allows the identification of single cells as objects. An accurate segmentation of nuclei and cell bodies can be achieved by intensity-based thresholding and region growing approaches,,. For more complex cell shapes or multicellular objects such as organoids, more sophisticated models are required to segment objects of interest. A number of computational applications for segmentation-free analysis in image-based profiling have been developed,. Recently, deep learning approaches based on segmentation-free strategies, in some cases in combination with single-cell segmentation, have been developed for image-based small molecule profiling (see main text for further details).Feature extractionFollowing segmentation, computer vision is used to extract morphology, intensity and texture features of single cells,. Morphological features describing cellular size and shape are calculated based on the cell outlines. Intensity and texture features are calculated on pixel values of raw or pre-processed images within those outlines. These features are defined based on human knowledge. Hundreds to thousands of features are extracted to achieve an unbiased character of the studies. In contrast, segmentation-free approaches provide a hypothesis-free approach for feature extraction independent of segmentation outlines. Recently, several approaches that use deep neural networks for segmentation-free feature extraction were developed,,,. While the features extracted by deep neural networks cannot be readily interpreted by humans, results from pilot studies indicate that those features are complementary to those extracted by segmentation-based strategies.Profile generationSegmentation-based image analysis results in collections of single cell feature vectors for each condition (e.g. per small molecule treatment). Various strategies have been explored to process those single cell feature vectors into profiles for downstream analyses. A common method relies on the aggregation of single cell features by summarizing the mean and standard deviation over all cells of a perturbation and other methods using machine learning approaches to generate profiles are reported. Furthermore, features are often highly correlated and deliver very similar information content. This inflates data complexity and impairs downstream analysis. Strategies to either select features carrying redundant information or to reduce data dimensionality by transformations are thus commonly applied before proceeding with downstream analyses.ClassificationThe aim of many profiling experiments is a classification of compounds. Based on a set of reference compounds and their profiles, classification strategies are used to infer putative MOA or toxicity relationships of formerly unseen compounds. This process is a bona fide machine learning task. Approaches include hierarchical clustering of compounds into groups that share high profile similarity, training support vector machines on profiles for MOA prediction or correlation network analysis,,. More recently, advanced classifiers such as random forests or deep neural networks were employed in order to achieve higher accuracy in predicting MOAs,,. The latter however with an increased complexity for analysis and demand more in-depth knowledge of neural network-based machine learning.Alt-text: Box 1

7Segmentation

8High-throughput small molecule profiling is typically based on the staining of subcellular structures and fluorescence microscopy. A selective staining of abundant subcellular structures such as DNA or actin and the subsequent imaging in distinct channels allows the identification of single cells as objects. An accurate segmentation of nuclei and cell bodies can be achieved by intensity-based thresholding and region growing approaches,,. For more complex cell shapes or multicellular objects such as organoids, more sophisticated models are required to segment objects of interest. A number of computational applications for segmentation-free analysis in image-based profiling have been developed,. Recently, deep learning approaches based on segmentation-free strategies, in some cases in combination with single-cell segmentation, have been developed for image-based small molecule profiling (see main text for further details).

9Feature extraction

10Following segmentation, computer vision is used to extract morphology, intensity and texture features of single cells,. Morphological features describing cellular size and shape are calculated based on the cell outlines. Intensity and texture features are calculated on pixel values of raw or pre-processed images within those outlines. These features are defined based on human knowledge. Hundreds to thousands of features are extracted to achieve an unbiased character of the studies. In contrast, segmentation-free approaches provide a hypothesis-free approach for feature extraction independent of segmentation outlines. Recently, several approaches that use deep neural networks for segmentation-free feature extraction were developed,,,. While the features extracted by deep neural networks cannot be readily interpreted by humans, results from pilot studies indicate that those features are complementary to those extracted by segmentation-based strategies.

11Profile generation

12Segmentation-based image analysis results in collections of single cell feature vectors for each condition (e.g. per small molecule treatment). Various strategies have been explored to process those single cell feature vectors into profiles for downstream analyses. A common method relies on the aggregation of single cell features by summarizing the mean and standard deviation over all cells of a perturbation and other methods using machine learning approaches to generate profiles are reported. Furthermore, features are often highly correlated and deliver very similar information content. This inflates data complexity and impairs downstream analysis. Strategies to either select features carrying redundant information or to reduce data dimensionality by transformations are thus commonly applied before proceeding with downstream analyses.

13Classification

14The aim of many profiling experiments is a classification of compounds. Based on a set of reference compounds and their profiles, classification strategies are used to infer putative MOA or toxicity relationships of formerly unseen compounds. This process is a bona fide machine learning task. Approaches include hierarchical clustering of compounds into groups that share high profile similarity, training support vector machines on profiles for MOA prediction or correlation network analysis,,. More recently, advanced classifiers such as random forests or deep neural networks were employed in order to achieve higher accuracy in predicting MOAs,,. The latter however with an increased complexity for analysis and demand more in-depth knowledge of neural network-based machine learning.

15Alt-text: Box 1

16High-throughput imaging has been successfully used for small molecule profiling in human cancer cells to group small molecules with similar activity and mechanism of action (MOA). The concept of clustering small molecules based on the similarity of phenotypic profiles was further exploited to correlate phenotypic responses with chemical structure similarity (Figure 3A and B,, ). The basic principle of grouping small molecules based on similarity of measured phenotypic profiles has further been successfully applied in a variety of drug discovery applications. For example, annotated libraries of pharmacologically active small molecules could be profiled to hypothesize off-targets and to suggest drug-target pairs with possibilities for repurposing,. Chemical genetics approaches also allow a systematic survey for drug repurposing by linking the effect of small molecules with the absence or presence of loss- or gain-of-function alleles,. Further strategies include the association of small molecules to a new target by comparing its induced phenotype to that of a reference molecule and the unbiased testing of small molecules for candidates that reverse a disease-associated cellular phenotype to the wild-type phenotype,,. Other applications include MOA profiling of natural product libraries and the enrichment of large compound libraries for effective and diverse subsets,. Studies also demonstrated the application of image-based profiling for toxicity profiling, for example, by using model systems such as 3-D liver cell culture or induced stem cell-derived cardiomyocytes to assess small molecule toxicity,. Additional applications include the unbiased identification of toxic small molecules in large natural product libraries,,. A further optimization of image-based profiling assays and analysis concepts made the technology accessible to more complex drug discovery studies as demonstrated by a recent study in which 50,000 compounds were profiled in pluripotent stem cells to complement a targeted antibody readout.

17Image-based phenotypic profiling has also been recently applied to characterize small molecule function in patient-derived primary cells or 3-D cell culture models. A recent study by Booij et al. used image-based profiling to quantify the phenotypic effects of kinase inhibitors in a 3-D cell culture model of Polycystic kidney disease to facilitate novel hypotheses for small molecule treatments (Figure 3C, ). Another study deployed image-based profiling in primary chronic lymphatic leukemia (CLL) cells to identify small molecules effective in overcoming resistance against the BCL2 inhibitor Venetoclax.

18Chemical genetics approaches have been pioneered in model organisms to discover drug MOA as well as gene functions,,,,. In such screens, libraries of loss-of-function or gain-of-function mutant strains were screened against chemical libraries to measure genotype-dependent changes in fitness. Comparisons of phenotypic responses of mutant strains across chemical perturbations allowed to map genes with shared function and, by assessing the response across a panel of mutants, the bioactivity and cellular targets of small molecules.

19In model organisms like yeast or bacteria, sophisticated experimental and analysis frameworks have been established for chemical-genetic studies which might provide further strategies for profiling in human cells,. As an example, a recent study in Saccharomyces cerevisiae used an algorithm to define a set of query mutant strains from genetic-interaction data. Several chemical libraries were then screened against the selection of deletion strains with the aim to functionally annotate small molecules across relevant biological processes without screening the full deletion strain library.

20This principle has been adopted to pharmacogenomics studies in human cancer cell lines by integrating drug response and the genetic background of cell lines with the aim to identify cancer-specific vulnerabilities,,. Several large-scale pharmacogenomics studies relied on univariate phenotypic readouts based on cell viability assays that can be applied to a large panel of cell lines, however, not all univariate cell variability measurements are equivalent,,.

21Microscopy-based profiling experiments are typically carried out in microtiter plates, which enables a rapid scaling of experiments for a large number of perturbations, i.e. to profile chemical libraries in a large number of genetic perturbations. As such, image-based chemical genetics offers a great potential to systematically characterize small molecules in human cell-based systems and to probe chemical-genetic interactions in complex and monogenic human diseases,. To systematically map chemical-genetic interactions by image-based profiling, smaller panels of human cancer cell lines have been used (Figure 3B,,,. The unbiased measurement of chemical genetic interactions across a wide phenotypic space can identify chemical-genetic interactions, small molecule off-target effects and drug repurposing opportunities. The establishment of image-based assays is often more labor intensive and not every cell line is suitable for high-throughput image-based profiling experiments.

22High-throughput microscopy generates large collections of phenotypic data. The size, complexity and multiparametric nature of such data sets make automated analysis strategies necessary to identify cellular phenotypes that differ from wild type phenotypes and discover novel relationships between phenotypes and the perturbagens which induce them. Simple statistical inference methods such as mean values are often insufficient to comprehensively describe the complexity of data sets. As a consequence, many profiling studies use supervised or unsupervised machine learning strategies to capture and evaluate phenotypic changes of single cells or populations of cells (Box 2, Figure 3D,,, ).Machine learning strategies can generally be separated in unsupervised and supervised approaches.Unsupervised machine learning approaches interpret and learn an abstract representation of the intrinsic structure of data sets and can be applied without a priori definition of labels (phenotypes) to be classified. A commonly applied unsupervised machine learning strategy is clustering of data points into patterns to ultimately derive biologically meaningful information,,. Commonly applied unsupervised clustering algorithms include hierarchical or k-means clustering.By contrast, supervised machine learning strategies are typically applied to classification problems and thus rely on pre-defined classes. Annotated training sets with data points representing those classes are used to train a classifier,. This way, the classifier learns how to distinguish between the classes based on the measured variables (also called features). Linear and non-linear supervised machine learning classifiers exist, and the choice of the classifiers depends on the classification problem. In many cases, a comparison of the classifier performance is used to identify the algorithm that is best suited for the problem. The training of classifiers requires representative training data sets of sufficient size and quality. To avoid issues such as overfitting and assess classification accuracy statistics, the resulting classifier needs to be evaluated carefully using test data (not used for training).Recently, deep learning has been exploited as a supervised machine learning technology for biological classification problems. Deep learning uses artificial neural networks, which consist of multiple layers of interconnected linear or non-linear transformations that are applied to the data with the goal to solve a classification problem. The networks are trained over multiple epochs, each time comparing the predicted and true class labels to optimize network parameters with automated algorithms. The actual network architecture relies on complex mathematical concepts. Strategies such as transfer learning or generic deep neural networks provide possibilities for non-experts to apply deep learning without full knowledge of the mathematical details,,.The high dimensionality and complexity of image data sets typically make analysis strategies with multiple steps and combinations of machine learning algorithms necessary. Here, deep learning strategies provide great potential to automate analysis at many stages from segmentation, feature extraction to classification to reduce manual tuning of analysis pipelines,.Alt-text: Box 2

23Machine learning strategies can generally be separated in unsupervised and supervised approaches.

24Unsupervised machine learning approaches interpret and learn an abstract representation of the intrinsic structure of data sets and can be applied without a priori definition of labels (phenotypes) to be classified. A commonly applied unsupervised machine learning strategy is clustering of data points into patterns to ultimately derive biologically meaningful information,,. Commonly applied unsupervised clustering algorithms include hierarchical or k-means clustering.

25By contrast, supervised machine learning strategies are typically applied to classification problems and thus rely on pre-defined classes. Annotated training sets with data points representing those classes are used to train a classifier,. This way, the classifier learns how to distinguish between the classes based on the measured variables (also called features). Linear and non-linear supervised machine learning classifiers exist, and the choice of the classifiers depends on the classification problem. In many cases, a comparison of the classifier performance is used to identify the algorithm that is best suited for the problem. The training of classifiers requires representative training data sets of sufficient size and quality. To avoid issues such as overfitting and assess classification accuracy statistics, the resulting classifier needs to be evaluated carefully using test data (not used for training).

26Recently, deep learning has been exploited as a supervised machine learning technology for biological classification problems. Deep learning uses artificial neural networks, which consist of multiple layers of interconnected linear or non-linear transformations that are applied to the data with the goal to solve a classification problem. The networks are trained over multiple epochs, each time comparing the predicted and true class labels to optimize network parameters with automated algorithms. The actual network architecture relies on complex mathematical concepts. Strategies such as transfer learning or generic deep neural networks provide possibilities for non-experts to apply deep learning without full knowledge of the mathematical details,,.

27The high dimensionality and complexity of image data sets typically make analysis strategies with multiple steps and combinations of machine learning algorithms necessary. Here, deep learning strategies provide great potential to automate analysis at many stages from segmentation, feature extraction to classification to reduce manual tuning of analysis pipelines,.

28Alt-text: Box 2

29Clustering image-based profiles to identify small molecules with shared MAO based on the ‘guilt-by-association’ rule relies on unsupervised machine learning strategies,,,. In the simplest case, morphological profiles are generated by aggregating all single-cell measurements per treatment condition and several statistical methods have been applied to generate morphological profiles. When used to classify a set of compounds by their MOA, a comparison of profiling methods showed only minimal differences between them.

30For large data sets, hierarchical (unsupervised) clustering algorithms are commonly used to cluster small molecule profiles based on their profile similarity. Often, matrices of all pairwise similarities between small molecule profiles are used to derive a distance measure (i.e. 1 – similarity) rather than absolute distance measures such as the Euclidian distance. Furthermore, phenotypic similarity matrices correlate in some cases with other similarities of small molecules such as chemical similarity,.

31Hierarchal clustering and the corresponding visualization as heatmaps are relatively easy to implement and require only few computational resources, even for large data sets. Furthermore, the concept of similarity-based clustering is frequently applied in other fields such as gene expression profiling or genetic interaction studies making those analyses and visualizations accessible to a broad audience,. Depending on the size and characteristics of the data set, alternative cluster algorithms and visualizations as for example two-dimensional maps might be favorable to identify relevant groups of cells or perturbations,,.

32In contrast to unsupervised approaches, supervised machine learning algorithms are commonly applied when cells or populations of cells need to be classified in discrete phenotypic classes. Supervised machine learning has been successfully used within image-based genetic screening experiments to classify single cells into pre-defined, biologically meaningful classes based on their phenotypic profiles (Figure 3D,,, ). Supervised classifiers are trained to learn the relevant phenotypes from training data to distinguish different phenotypes. This approach has been applied by using tool compounds, genetic perturbation or extracellular stimuli to create reference profiles which were further used to train classifiers,,,,,. In some cases, supervised machine learning strategies were used to calculate profiles of meta features which were then applied to various downstream analysis strategies. As an example, Loo et al. developed a strategy using support vector machines to separate treated from untreated cells in the high-dimensional feature space. In subsequent steps, profiles were calculated using classification accuracy and the orientation of support-vector hyperplanes. The derived meta-profiles incorporate multiphasic responses over dose ranges and retain human interpretable profiles for analyses.

33Supervised machine learning requires curated training data, e.g. manually labeled cells or defined experimental measurements. The size and quality of the training data is crucial for the performance of the corresponding classifier and manual annotation of samples may be biased by subjective decisions. As for many published studies that used supervised machine learning, training data sets need to be reviewed and possibly re-generated if only small experimental parameters change,. Thus, the choice of the machine learning algorithm and its proper implementation are key factors to avoid issues such as overfitting. Whilst such challenges might put unsupervised and simple statistical inference methods in favor for large-scale profiling experiments, supervised machine learning is being used with success to classify complex phenotypes,,.

34A recent study that used high-throughput microscopy and automated image analysis with deep convolutional neural networks classified proteins by their subcellular localization in S. cerevisiae and thereby demonstrated the power of supervised machine learning for image-based phenotyping (Figure 3E, ). The deep neural network that was used not only outperformed a previously implemented ensemble of support vector machines at a complex automated image analysis task, but could also be adapted to new, divergent data sets using transfer learning.

35Transfer learning aims to adopt a machine learning classifier to a new problem (i.e. new data sets, generated with e.g. different experimental parameters) by fine-tuning the classifier with training data from unseen data sets. Examples show that better classification performance can be achieved by combining deep- and transfer learning when compared to the setup of a classifiers trained from scratch,. This saves computational resources as the fine-tuning is computationally less expensive than training a deep neural network from scratch and high classification accuracy might be achieved with smaller training data sets.

36Recently, a number of studies reported the application of supervised machine learning approaches with deep neural networks to classify small molecules by their MOA using annotated image data sets from the Broad Bioimage Benchmark Collection,,,,,. Various strategies with applications of deep neural networks at the level of feature extractions and/or classifications were reported, in some cases with previous traditional single cell segmentation. As an example, one study used labeled full resolution images to train a deep neural network which gave slightly better treatment level classification results compared to previously reported predictions using segmentation and factor analysis. Notably, a relatively low number of images (25 images per treatment) were used to train the network and no previous segmentation and labeling of single cells was required.

37Nevertheless, labeled training data sets imply the a priori knowledge of phenotypes, which can contradict the unbiased strategy of image-based profiling. Two recent studies propose to use generic deep neural networks that were pre-trained on millions of ‘consumer’ images for image-based profiling tasks,. The approaches are based on the assumption that generic neural networks learned general properties of natural images and are thus capable of extracting biologically meaningful information without additional training. Both studies report better results compared to traditional feature extraction when predicting small molecule MOA and provide a proof-of-concept for the applicability of generic deep neural networks for image-based small molecule profiling. As noted by the authors, additional studies with larger data sets across conditions to sample broader biological and technical space will be required for further validation.

38Another recently explored application of supervised learning in image-based profiling, particularly deep neural networks, is a novelty detection framework to identify unexpected phenotypes. Label-free profiling and the prediction of targeted drug screening assays are also future approaches exploiting image-based profiling data,,.

39Image-based profiling studies demonstrated the capability to improve the pre-clinical development of small molecules at almost any step of the pipeline from target identification over mechanism of action prediction to toxicity profiling. Increasing the throughput and extending more complex analysis methods of image based phenotypic screens and profiling approaches will help to increase the methodological portfolio of cellular screens to support the drug development process. Community efforts to create annotated datasets that can be shared across laboratories will be required to test and optimize the potential of strategies such as transfer learning to improve discovery science,.

40Furthermore, large-scale chemical-genetics approaches inspired from successful studies in model organisms might harbor great potential to characterize drugs and drug-gene interactions in a systematic manner. Particularly, image-based profiling approaches in pre-selected informer panels of human cell lines might be a scalable and versatile tool to deprioritize compounds harboring adverse effects, asses compound efficacy and to generate hypotheses for drug synergism and repurposing.

41This work was in part supported by an ERC Advanced Grant (SYNGENE) of the European Commission.

42None.