DeepLoc 2.1: multi-label membrane protein type prediction using protein language models

1Membrane proteins are essential for a wide range of cellular functions. These functions range from assisting the cell in mediating communication through signaling pathways to facilitating the transportation of macromolecules and maintaining ion gradients across membranes. For these reasons, membrane proteins often pose as targets in drug discovery, highlighting their biological importance for proteomics research.

2On a high level, membrane proteins can be categorized into three major classes: peripheral membrane proteins, transmembrane proteins and lipid-anchored proteins. This categorization of a protein into one of these classes often provides insight into its functionality and biological properties. Similar to the task of predicting the subcellular localization of eukaryotic proteins, which was the aim of DeepLoc 2.0, having access to tools that can accurately determine the membrane association of a protein in addition to its subcellular localization is highly useful from a medical and biotechnological standpoint.

3In DeepLoc 1.0, a binary membrane classification was included in the server’s output, where a protein would be predicted as either membrane-bound or soluble, but in DeepLoc 2.0, this functionality was not included. With DeepLoc 2.1, we not only reinstate the capability of predicting membrane proteins, but expand it considerably by also distinguishing between the three types of membrane proteins. Due to the high similarity between these multi-label prediction tasks, the models developed for the prediction of membrane association have been built upon a similar data structure and model architecture. We use sequence-based embeddings from the ProtT5 and ESM-1B protein language models and implement the same model architecture used by DeepLoc 2.0, performing interpretable attention pooling over sequence embeddings with discrete cosine transform regularization.

4Various homology and machine learning-based tools exist for the prediction of membrane protein types. The majority of these tools can be categorized as homology-based models, that utilize sequence alignment to search for gene ontology (GO) terms within a database of experimentally annotated proteins to reach a conclusion about the membrane association of a query protein. This property often poses the limitation of homology-based models as high-quality experimental annotations of proteins are often expensive and laborious. Additionally, the databases used for searching GO-terms must be continuously updated to ensure acceptable accuracy of the models as novel proteins and functionality are discovered.

5Among the tools that have gained the most attention for the task of predicting membrane protein types are Mem-ADSVM, MemPype and MemType-2L. Mem-ADSVM is a multi-label homology-based predictor that uses a support vector machine to infer its membrane type predictions exclusively based on the GO-terms of homologous accession numbers from a compact database named ProSeq-GO. The MemPype server is a multi-class predictor developed for eukaryotic proteins. The server provides two outputs, i.e. one that is solely based on various machine learning tools and sequence-based inference along with a seperate homology-based output that acts to support the prediction of the ML-based output. The ML-based sequence profiling uses multiple other tools for the prediction of signal peptides, GPI-anchors, and transmembrane domains to arrive at a decision about the membrane type of the query protein. MemType-2L is another multi-class predictor that utilizes sequence similarity. Instead of searching for homologous sequences using alignment-based methods, it represents known proteins using a pseudo position-specific scoring matrix (Pse-PSSM) and performs k-nearest neighbor classification on a Pse-PSSM encoded query protein.

6We propose a multi-label prediction tool that relies purely on sequence-based predictions utilizing the advances made in recent years using transformer-based protein language models. DeepLoc 2.1 builds upon the same template architecture as the previous DeepLoc models for the prediction of subcellular localization. The architecture is defined by three stages. Firstly, a pre-trained protein language model is used to create a feature representation for each amino acid in the sequence. Secondly, the feature representation is passed to an attention-based pooling stage, which generates a single representation for the entire sequence. Lastly, the attention-based representation is passed to a classifier that predicts the membrane type labels.

7The web server of DeepLoc 2.1 enables the prediction of the four membrane protein types, Peripheral membrane protein, Transmembrane protein, Lipid-anchored protein, Soluble protein (i.e. non-membrane), along with the original output of the DeepLoc 2.0 web server that includes prediction of ten subcellular locations and nine sorting signals of a query protein. Additionally, an attention plot of the sequence is provided. Like its predecessor, DeepLoc 2.1 is free and publicly available with no login requirements. Input to the server is provided in FASTA format and the server takes a maximum of 500 sequences per submission. The predicted membrane protein types are displayed in a table highlighted with shades of green for positive labels. The intensity of the green color indicates the certainty of the predicted label to be a true label relative to the threshold value for a positive label for the given membrane protein type, similar to DeepLoc 2.0. DeepLoc 2.1 will always infer at least one positive label for its membrane type prediction, which implies that if the model did not predict any positive label, the inferred membrane protein type will be the one closest to its associated threshold value. Figure 1 showcases an example output of the web server. Estimates of prediction times are provided in Supplementary Table S1.

8We curated a dataset with experimental evidence for membrane protein type labels containing 25 240 eukaryotic protein sequences from the UniProt database. All data were extracted from UniProt release 2022_10 and were filtered using the following criteria: Eukaryota (eukaryotes) [2759], not fragments, >40 amino acids and any experimental assertion. From this data the sequences can be categorized into one or multiple of the following four membrane protein types: Peripheral, Transmembrane, Lipid-anchor and Soluble (for non-membrane association). Details of the curation process are given in Supplementary Section 1 and Supplementary Tables S2 and S3.

9This dataset was split into five partitions using GraphPart, with a maximum global sequence identity of 30% between sequences in different partitions, as measured by Needleman–Wunsch alignments. Four of these partitions were used for 4-fold cross-validation, while the last partition including 4933 sequences was used solely for testing (see Table 1). Supplementary Table S4 shows a detailed description of the label distribution of membrane protein types between all partitions.

10The DeepLoc 2.1 methodology for predicting the type of membrane protein has three stages that builds upon the pipeline developed for DeepLoc 2.0. The first stage involves creating per-position representations, extracted from a pre-trained transformer-based protein language model for each amino acid in a query protein. During the development of DeepLoc 2.1, four protein language models were assessed and compared for predicting membrane protein types. These included three models of the Evolutionary Scale Modeling project from Meta, ESM-1B-650M (used for DeepLoc 2.0 and referred to as ESM-1B), ESM-2-650M and the largest model from this project, the ESM-2-15B. Furthermore, we evaluated the performance of the ProtT5-XL-Uniref50, referred to as ProtT5, from the ProtTrans project, also used for DeepLoc 2.0. However, only ESM-1B and ProtT5 models are included in the DeepLoc 2.1 web server because no distinct improvements were observed for the membrane protein type predictions with the newer ESM-2 protein language models compared to the previous ESM-1B model. The decision to only include these models was made to maintain compatibility with the existing DeepLoc 2.0 pipeline, which utilizes the ESM-1B and ProtT5 model so that only a single per-position representation for a query protein needs to be computed for the subcellular location and membrane protein type prediction tasks.

11The per-position embeddings of a sequence of length L undergo attention pooling to compute a single representation vector for the whole sequence. Attention pooling allows the model to focus on regions in the sequence that are important for localization prediction, such as hydrophobic transmembrane stretches or lipid anchor sites. The prediction of attention weights is regularized using a Discrete Cosine Transform (DCT) prior to yield more interpretable attention patterns.

12Finally, the attention-pooled representation vector is passed to a linear layer that outputs a probability for each label. The loss function used for training the models is a weighted focal loss function. Additionally, a probability threshold value is computed for each label independently by optimizing Matthews’ Correlation Coefficient (MCC) on the training data. As opposed to other accuracy metrics, the threshold optimization using the MCC-score better accommodates label imbalance.

13Details of the DeepLoc 2.1 implementation and training are given in Supplementary Section 2 and Supplementary Table S5, and the model architecture is illustrated in Supplementary Figure S1.

14The two models developed for DeepLoc 2.1, employing the ESM-1B (high-throughput) and the ProtT5 (high-quality) protein language models demonstrate nearly identical performance across the assessed metrics seen in Table 2 when evaluated on the test set (Table 1). The larger model, ProtT5, appears to have a marginal edge over the ESM-1B across the majority of the performance metrics. Considering the MCC for the different labels it can be seen that ProtT5 is slightly less prone to inferring wrong predictions for the Peripheral and Soluble labels. These two classes exhibit a significant biochemical overlap, posing a challenge for the models to effectively differentiate between them, and for this purpose the larger model seems to possess a marginal advantage.

15To establish a baseline, the models were evaluated against an alignment-based baseline, utilizing MMSeqs2 for sequence alignment. Positive labels for each specific sample were deduced based on the true labels of the sequence with the highest alignment score in any other partition. A comparison between the findings in Table 2 and the baseline results, described in Supplementary Section 3 and Supplementary Table S6, reveals that the models significantly outperform these baseline predictions, indicating that simple homology-based models lack the contextual depth necessary for accurate predictions.

16To evaluate the performance of DeepLoc 2.1 we have compared our model to the Mem-ADSVM, MemPype and MemType-2L methods, which are all established servers for the task of predicting the membrane association of a protein. As mentioned, all of these servers are homology-based to a certain extent, which has led us to construct different sub-sets of our test set in order to achieve a fair comparison, with as little overlap as possible between the test set and the database used by the various models. More details on this can be found in Supplementary Section 4 and Supplementary Tables S7, S8 and S9. Additionally, some of the models that have been included for comparison have different output characteristics. Both MemPype and Memtype-2L are single-label multi-class predictors, which implies that we have excluded all multi-label samples for comparison with these models. To retrieve single-label predictions for our models, we applied the softmax activation function to the raw outputs of the models that were trained for multi-label prediction.

17As seen in Table 3, all performance metrics indicate a considerable advantage in favor of the protein language models employed by DeepLoc 2.1. The only metric where Mem-ADSVM demonstrates a performance approaching that of DeepLoc 2.1 is in terms of subset accuracy. The reason for this lies in the decision making mechanism of the Mem-ADSVM server, where if a protein has been predicted as soluble the model will not proceed to make any further predictions. Due to the biochemical overlap between the soluble and peripheral class coupled with the significant sample imbalance, where soluble proteins are eight times more abundant than peripheral proteins in the dataset, a decision mechanism, such as that employed by Mem-ADSVM, could potentially exhibit a bias towards the majority class. However, this characteristic of Mem-ADSVM contradicts its assertion of being a multi-label predictor, particularly in light of the fact that a majority of the multi-label samples emerge from the soluble and peripheral classes (refer to Supplementary Section 1 for further details on the dataset and label distribution).

18Comparing our models to the MemPype server also shows a considerable advantage for DeepLoc 2.1 across all reported metrics (see Table 4). As the MemPype server is a single-label predictor and additionally does not distinguish between soluble and peripheral membrane proteins, we decided to construct a merged class for samples that had true positive labels for the peripheral and/or soluble class and recorded a correct prediction of the merged class if one of the two classes was correctly predicted.

19A comparison of the DeepLoc 2.1 models against the MemType-2L model shows superior performance of the models based on the transformer protein language models for every metric recorded (see Table 5). Relatively low MCC scores are observed for the single-label case of peripheral membrane proteins, and here the performance of the MemType-2L model approaches that of the DeepLoc 2.1 models. This can be attributed to the fact that the DeepLoc 2.1 models appear to be biased towards the soluble class when constrained to make only single-label predictions. This assertion is supported by the data presented in Table 2, where the sensitivity score for the peripheral class of proteins is notably low. According to the definition of sensitivity, this suggests a significant number of false negative labels are predicted for the peripheral class. Conversely, Table 2 indicates a rare occurrence of false positives for the peripheral class, as evidenced by the specificity for this class being comparable to that of the soluble class for the DeepLoc 2.1 models.

20In addition to the already existing DeepLoc 2.0, used for predicting the subcellular localization of eukaryotic proteins, we include state-of-the-art prediction of the membrane association of a query protein. The models developed for prediction of membrane-association build upon the same methodology as DeepLoc 2.0, utilizing transformer protein language models, and we show that these purely sequence-based models outperform every other available tool for membrane protein type predictions.