DDMut: predicting effects of mutations on protein stability using deep learning

1Proteins are versatile and dynamic tools tailored by nature over the course of evolution to coordinate a range of biochemical processes central to life. They are involved in many biological processes, including cell signalling, proliferation, metabolism, and cell death. It is, therefore, essential to understand how changes in the protein sequence might impact its structure, function and interactions, giving rise to different phenotypes.

2Characterising the molecular consequences of mutations can provide key insights into their biological outcomes. Over the past few years, missense mutations have been extensively studied due to the accumulation of data and their subtle effect on proteins. A single amino acid change at the protein sequence level can lead to local atomic changes in the 3D structure, thereby affecting the kinetics of protein folding, stability, flexibility and dynamics. Despite large advances in protein structural modelling tools, these changes, however, are currently poorly captured by protein structure prediction tools.

3Significant efforts have been invested into understanding and predicting the molecular consequences of mutations in protein coding regions, however most approaches have been limited in their throughput (preventing genome wide and saturation mutagenesis implementation), restricted to predicting consequences of single point missense mutations, and are poorly predictive of stabilising mutations, essential for biotechnological applications, due to inherent biases in the data.

4To fill this gap, here we report DDMut, a user-friendly web server that implements our well validated concept of graph-based signatures within a novel deep learning framework (Figure 1), enabling us to rapidly screen both single and multiple point mutations, with comparable performance on both stabilising and destabilising mutations. DDMut was made available as an easy-to-use web server and API, for seamless integration with analytical pipelines at https://biosig.lab.uq.edu.au/ddmut/.

5DDMut training set for predicting the effects of single point mutations was curated from S2648 (originally from ProTherm) and FireProtDB. Redundant entries (at mutation level) in the blind test sets were removed from the training set if they have the same Uniprot ID and the same mutation. This is followed by removing the duplicates (at mutation level) in each dataset, where the chosen ΔΔG among the duplicates was measured under physiological conditions closest to the pH of 7 and the temperature of 25°C. To balance the ΔΔG distribution, the hypothetical reverse mutations were introduced into each dataset under the following scheme, where is defined as the unfolding free energy:

6This led to our final DDMut training set S9028 (9028 mutations, across 153 proteins). The ΔΔG distribution of S9028 is shown in Supplementary Figure S1.

7For predicting the effects of multiple mutations, we removed duplicates and redundant entries (at multiple-mutation level) from DynaMut2 training set, and included reverse mutations. This led to our training set SM1242 (98 structures across 94 proteins). We also implemented a protein-level non-redundancy split, which led to the training set SM1218 (67 structures across 60 proteins).

8For single point mutations, the five universal non-redundant blind test sets at protein or mutation level for most available protein stability predictors were used for the purpose of benchmark comparisons. S276 and S669 include proteins which have low sequence identity with the original ProTherm dataset and are non-redundant at protein level, whereas S1342 is a blind test non-redundant at mutation level, which means mutations in this dataset may occur on the same protein with mutations in the training set, but at different positions or the same position with different mutant residues. Deep mutational scanning (DMS) datasets from the CAGI5 challenge (including variants for PTEN and TPMT) and Gerasimavicius et al., which includes functional scores of 161,441 variants across 45 independent assays, were also evaluated. After removing redundant data within the same test set (at mutation level) and including the hypothetical reverse mutations, our final blind test sets comprised of 552 (37 structures on 37 proteins), 1,304 (94 structures on 87 proteins), 2,024 (129 structures on 120 proteins) mutations. The overlaps between these three datasets at both mutation and protein level are shown in Supplementary Figure S2. Since the CAGI5 challenge data infer protein stability changes from the abundance of EGFP fused to the mutant protein, PTEN (3,736 mutations) and TPNT (3,627 mutations), hypothetical reverse mutations were not included for these datasets. Similarly, reverse mutations were not included for the DMS datasets from Gerasimavicius et al. as they indicate the functional impacts of mutations and do not reflect the thermodynamic cycle of protein stability per se.

9The blind test set for predicting multiple mutations was originally reported in DynaMut2. Removing the duplicates and including the reverse mutations led to our multiple point mutation blind test set SM420 (61 structures in 63 proteins, 420 double and triple mutations). Under the protein-level non-redundancy split, the blind test set SM444 has 44 structures across 44 proteins.

10The wild-type structures for all the datasets were downloaded from Protein Data Bank, and the mutant structures were generated from the corresponding wild-type using MODELLER with its default minimisation pipeline. Both wild-type and mutant structures were utilised for generating the features for both forward and reverse mutations.

11Two sets of features were generated, graph-based signatures and complementary features:

12The graph-based signatures were generated using mCSM, a Cut-off Scanning Algorithm operated within a graph-based representation of local residue environment, to capture the distance patterns between pairs of atoms labelled with eight different pharmacophores (Hydrophobics, Positives, Negatives, Hydrogen Acceptors, Hydrogen Donors, Aromatics, Sulphurs and Neutrals)

13The complementary features include both sequence- and structure-based features: (a) sequence-based features were calculated using substitution matrices such as AAindex to capture changes in physico­chemical and biochemical properties, BLOSUM and PAM which are based on sequence alignment; (b) structure-based features include solvent accessibility, residue depth, secondary structure, atomic interactions between the residue of interest and its neighbouring residues calculated by Arpeggio, as well as the changes in interactions upon mutations. The tools used to calculate each set of complementary features are detailed in Supplementary Table S1.

14The generated features were then normalised by their mean values and standard deviations.

15For both predictive tasks, single and multiple point mutations, DDMut models were trained using siamese networks (Supplementary Figure S3), where two sub-networks with the same architecture and weights were applied on the features calculated from forward and reverse mutations separately. In each sub-network, graph-based signatures were processed with convolutional layers followed by a transformer encoder, whereas complementary features were processed with two dense layers. These two feature components were then concatenated along with residual connections, and then followed by a dense layer. In the end, a contrastive loss function adapted and modified from was calculated to not only consider the errors between the predicted and actual ΔΔG, but also take into account the antisymmetry (errors between forward and the corresponding reverse mutations):

16where and are predictions for forward and the corresponding reverse mutations, is the experimental ΔΔG for the forward mutation. For a perfectly anti-symmetric and accurate model, the following rules will be satisfied:

17which results in a loss of 0.

18During the training phase, the hyperparameters of the architecture were fine-tuned based on the cross-validation performance on the training set. This process was carried out independently for single and multiple point mutation models. The evaluation metrics used include Pearson's (r), Kendall's (k) and Spearman's (s) correlations, root mean square error (RMSE), mean absolute error (MAE) and mean signed error (MSE).

19We deployed DDMut as a freely available and user-friendly web server at https://biosig.lab.uq.edu.au/ddmut/. The frontend is built using MaterializeCSS (version 1.0.0), and the backend uses the Flask module (2.0.3) from Python. The web server is hosted on a Linux machine running Nginx.

20DDMut can be used to predict ΔΔGs for both single point mutations and multiple mutations under two different options. Users are required to provide a protein of interest by either uploading a PDB file or input a valid PDB accession code. Mutation details for the ‘Single Mutation’ option (Supplementary Figure S4) can be provided manually as a text string (in the format of wild-type residue one-letter code followed by residue position and mutant residue one-letter code) and Chain identifier, or by uploading a text file with a list of mutations. Alternatively, users can run automated alanine scanning. For the ‘Multiple Mutations’ option (Supplementary Figure S5), mutations should be separated by a semi-colon for each entry (‘A F7A;A V13M’ as an example, where two mutations F7A and V13M are on the same chain A). Here, we are considering double and triple mutations only. Although the web server does accept submissions for more than three simultaneous mutations, it is important to note that the model has only been validated on up to triple point mutations. Users should therefore exercise caution when submitting more than three simultaneous mutations. In both options, users may choose to include predictions for hypothetical reverse mutations, and provide an email address which will be used to send notification once the job's results are ready.

21To assist users with job submission, a help page is available at https://biosig.lab.uq.edu.au/ddmut/help.

22For ‘Single Mutation’, predicted ΔΔG values are displayed alongside information of the wild-type residue environment and a 3D interactive viewer built using NGL viewer, highlighting inter-residue interactions (Supplementary Figure S6). For ‘Mutation List’, results are summarised as a downloadable table with buttons redirecting to the detail page, which is similar to the ‘Single Mutation’ outputs of each variation (Supplementary Figure S7). For ‘Alanine Scanning’, predictions are mapped onto the protein sequence and 3D structure displayed using NGL viewer, and results are downloadable either in the format of a table, or a 3D protein structure with the predictions annotated on the b-factor column (Supplementary Figure S8).

23For ‘Multiple Mutations’, results are summarised as a downloadable table, and users can select specific entries from the table to be highlighted in the interactive viewer with residue contacts (Supplementary Figure S9).

24DDMut provides an API (Application Programming Interface) to facilitate convenient integration into different research pipelines. A unique ID will be assigned to each single submitted job, and can be used to query the job status or access the website interface. Our API requires the same inputs as our website. More detailed explanations and examples using curl and Python can be found at https://biosig.lab.uq.edu.au/ddmut/api.

25The processing time of DDMut was compared with DynaMut, DynaMut2, MAESTRO, FoldX, DDGun and BoostDDG (Supplementary Table S2) using proteins with different sequence lengths. DDMut demonstrated competitive efficiency especially for larger proteins (Supplementary Figure S10).

26DDMut was able to accurately and robustly predict the effects of both single and multiple point mutations. The performance on the training sets (under 10-fold cross-validation) and blind test sets are shown in Table 1.

27We evaluated the performance of DDMut on our training set comprising 9,028 single point mutations under 10-fold cross-validations. Two different fold-splitting strategies were implemented, low redundancy at the amino acid level and low redundancy at the protein level. Our method achieved a Pearson's correlation of 0.77 (RMSE: 1.25 kcal/mol) under the amino acid low redundancy scheme, and 0.70 (RMSE: 1.37 kcal/mol) for the protein low redundancy split (Figure 2A). The comparable performance between the two splits provided confidence in the robustness of the models. DDMut also achieved consistent performance across both forward and reverse mutations (RMSE: 1.36 and 1.38 kcal/mol respectively), with a Pearson's correlation of –0.93 between the forward and corresponding reverse mutations, indicating high model anti-symmetry (Supplementary Figure S11). To build a more robust model capable of predicting mutation effects on stability changes of a broader group of proteins, the hyperparameters in the neural network were tuned based on the cross-validation performance under the protein low redundancy split. The final model was trained on the entire training set, and then evaluated on blind test sets.

28To fairly compare our model with other available methods, we tested DDMut on non-redundant blind test sets comprising 276, 1,304 and 2,024 mutations, and a DMS dataset from the CAGI5 challenge including variants for PTEN and TPMT. Our model achieved the top performance for three out of the four blind test sets (Figure 2B), with consistent performance on both forward and the hypothetical reverse mutations, and on both stabilising (ΔΔG ≥ 0 kcal/mol) and destabilising (ΔΔG < 0 kcal/mol) mutations (Figure 2B, Supplementary Tables S3–S9). This provided confidence in the generalisability of the DDMut model. We then further evaluated the capability of DDMut on predicting the functional scores of 161,441 variants across 45 independent DMS assays, ranging from protein abundance, protein binding, activity assays, growth experiments and viral replication. DDMut demonstrated competitive performance when compared to nine other protein stability predictors, while the performance is highly heterogeneous across different assay types (Supplementary Figure S12).

29The contribution of each architecture component was evaluated using ablation studies. By disabling sub-blocks in the network architecture, we found all the components were essential for the final predictions. While the two dense layers contributed more to the final performance, the transformer encoder and convolutional layers contributed less (Supplementary Table S10). This can also be caused by the sets of features they processed respectively. To further understand what makes a mutation to be stabilising (ΔΔG ≥ 0 kcal/mol) or destabilising (ΔΔG < 0 kcal/mol), we then evaluated the importance of each feature to the neural network by a model-agnostic approach, i.e. feature permutation importance. We randomly shuffled the feature values across all the mutations in the balanced blind test set S1304, while maintaining the mean and variance of the feature. Notably, the top two important features (Δ hydrophobic contacts; Δ hydrophobic atoms) are both related to the changes in hydrophobicity upon mutations, shuffling each of them alone dropped the Pearson's correlation on forward mutations by around 0.05 (Supplementary Table S11).

30We then evaluated the performance of DDMut on predicting the effects of double/triple point mutations. To keep consistency with data used by other methods, DDMut was trained and optimised on a dataset consisting of 1242 entries (SM1242), and performance was assessed using a blind test non-redundant at multiple-mutation level comprising 420 entries (SM420). On the training set SM1242, DDMut achieved Pearson's correlations of 0.69 (RMSE: 1.83 kcal/mol) under 10-fold cross validation using the amino acid low redundancy splitting scheme. On the blind test SM420, DDMut achieved a Pearson's correlation of 0.70 (RMSE: 1.84 kcal/mol) (Figure 2C), outperforming previous methods on the forward mutations with consistent performance on both stabilising and destabilising mutations (Supplementary Table S12), and also on both double and triple point mutations (Supplementary Table S13).

31Under the protein-level non-redundancy scheme, we trained and optimised DDMut on SM1218, and tested on SM444. On the training set SM1218, DDMut achieved a Pearson's correlation of 0.45 (RMSE: 2.17 kcal/mol) under 10-fold cross validation. On the blind test SM444 non-redundant at protein level, DDMut achieved a Pearson's correlation of 0.49 (RMSE: 2.45 kcal/mol) (Figure 2D), comparable to performance across training. The performance on stabilising and destabilising mutations, and on double and triple point mutations are shown in Supplementary Tables S14 and S15, demonstrating that despite a drop in performance, DDMut outperforms previous methods including MAESTRO, FoldX, and DDGun. Due to the limited sample size for multiple mutations and a demand for scalability to unseen proteins, the final model deployed on our web server was built on the combined training and blind test sets.

32Here we present DDMut, a fast and accurate tool to investigate the effects of single and multiple missense mutations on protein stability. DDMut is a siamese network utilising both forward and the hypothetical reverse mutations to account for model anti-symmetry, and integrating our well-established graph-based signatures with convolutional layers and transformer encoder to better capture short- and long-range atomic interactions step-wisely within a localised 3D residue environment. DDMut achieved consistent performance on both stabilising and destabilising mutations, and outperformed other tools on different blind-test sets in terms of both accuracy and efficiency. By allowing users to perform alanine scanning, DDMut could also be potentially used for probing important residue side chains for protein folding and stability. We believe DDMut will be an invaluable tool for various applications such as detecting functional residues, inferring disease-associated missense mutations, and engineering more stable proteins. DDMut is freely available at https://biosig.lab.uq.edu.au/ddmut/.

33DDMut web-server and all the datasets used in this study are freely available at https://biosig.lab.uq.edu.au/ddmut/.