State-of-the-Art Estimation of Protein Model Accuracy Using AlphaFold

1Knowledge of 3D protein structures is critical for designing drugs, characterizing diseases, and creating a mechanistic understanding of cellular biology. Experimental approaches to protein structure determination can be costly and time consuming, so the ability to computationally predict protein structures from amino acid sequences is extremely useful. Recently, alphafold demonstrated breakthrough performance on protein structure prediction, with predictions often nearing experimental accuracy. Approaches like alphafold have advanced the state-of-the-art in protein structure prediction by using deep learning methods to analyze coevolutionary information. To predict the structure of a target amino acid sequence, these methods first search a database of protein sequences to compile a multiple sequence alignment (MSA), which is essentially a collection of sequences that are evolutionarily related to the target sequence. MSAs are known to provide extremely useful information for predicting protein structures. Intuitively, if two residues are in contact in a folded protein structure, mutations in the first position may induce a selective pressure for the second position to mutate. Such mutational covariance can be detected in MSAs, and this signal has been critical to the success of recent protein structure prediction models, including alphafold. However, the requirement of MSAs for protein structure prediction is sometimes problematic, since some proteins have few known homologs.

2In theory, it should often be possible to predict protein structures without using MSAs, since protein structures are fully determined by their amino acid sequences. More specifically, Anfinsen’s dogma states that protein structures fold to minimize free energy, which is a function of the protein’s 3D configuration and its amino acid sequence. Therefore, if one could model this energy function with sufficient accuracy, then one could predict protein structures by optimizing this function over the space of 3D configurations. Classical protein structure prediction methods like rosetta take this approach, and sample structures from a hand-designed energy function. The challenge with this approach is twofold. First, it is difficult to accurately model the biophysical energy function that governs protein folding at a level of abstraction that is computationally tractable. Second, even with perfect knowledge of the energy function, there are an astronomically large number of possible protein geometries, so searching for the optimum is a difficult global optimization task.

3Given the theoretical possibility of predicting protein structures without MSAs, it is interesting to speculate why alphafold remains dependent on MSAs for its accuracy. One intriguing possibility is that alphafold has learned an accurate energy function for scoring the accuracy of candidate protein structures, but the coevolutionary information in the MSA is necessary to locate an approximate global minimum in this energy function and circumvent the challenging optimization problem. After finding the neighborhood of the global minimum using the MSA, the later stages of the alphafold model may act as an “unrolled optimizer” and locally descend the learned energy surface to produce a refined structure prediction. alphafold also outputs various confidence scores related to the predicted accuracy of its structures, and these confidence scores may be determined by the value of its internal energy function. This hypothetical prediction mechanism is illustrated in Fig. 1. Note that we use the term “energy” to describe a function that has an optimum around the native structure and generally correlates with the probability that a protein sequence will adopt a given conformation, rather than a literal thermodynamic free energy. This notion of an energy function is reminiscent of energy-based machine learning models, which learn an unnormalized Boltzmann distribution to represent a target data density.

4Our hypothesized prediction mechanism lends itself to experimental testing. Candidate structures can be supplied to alphafold as templates, which are used to incorporate known structural information from proteins that are related to the target sequence. In this Letter we show that, when a candidate structure is introduced as a template, alphafold’s confidence metrics are closely correlated with the actual accuracy of the candidate structure, even when no coevolutionary information is supplied. This suggests that alphafold has learned an accurate energy function for scoring protein structures that does not rely on coevolutionary information.

5Computational biologists have historically predicted protein structures based on related sequences with solved structures. alphafold incorporates this approach by allowing the structures of up to four related proteins to be supplied to the model as templates. For each template, alphafold receives the template’s one-hot-encoded amino acid sequence, Cβ distance matrix, and backbone and side chain torsion angles as inputs. In addition, alphafold is given a mask indicating which atoms are unresolved in the template structure, and ignores torsion angles involving those atoms. Recent papers have demonstrated that alphafold’s template mechanism can be used to refine experimentally and computationally derived structural hypotheses.

6We investigated whether alphafold has learned a coevolution-independent energy function for scoring protein structures by supplying alphafold with (i) a target amino acid sequence to be predicted and (ii) a “decoy structure” that is passed to the model as a template. The goal of this procedure is to score the plausibility of the target amino acid sequence adopting the geometry given by the decoy structure. It is motivated by the hypothesis that alphafold’s output structure will resemble the decoy introduced as a template and therefore, if alphafold has learned an accurate energy function that does not require coevolution information, the output confidence metrics will closely track the quality of the decoy. Note that no coevolutionary information is supplied to the model during this procedure.

7We used a sequence of all “gap” tokens (which represent missing amino acids) to fill in the one-hot-encoded amino acid sequence associated with the decoy. We used the gap sequence due to an initial observation that high sequence identity between the decoy sequence and the target sequence caused alphafold to be overconfident in the decoy’s accuracy (Supplemental Material, Fig. S1). To keep the structural information supplied to alphafold from leaking the true decoy sequence, we masked out all side chain atoms aside from Cβ, and added a Cβ atom to all glycine residues (we decided to retain the Cβ atoms because alphafold uses a Cβ distance matrix to encode the template structure).

8After processing its inputs, alphafold produces an output structure and two confidence metrics: the predicted template modeling score (pTM) score and the predicted Local Distance Difference Test (pLDDT) score. To determine whether alphafold has learned a MSA-free energy function for assessing protein structure accuracy, we investigated whether we could accurately rank the decoy structures based on alphafold’s outputs. For each decoy, we computed a “composite confidence score” by multiplying the output pLDDT, the output pTM, and the TMscore between the decoy structure and the alphafold output structure. The last term adjusts for the fact that alphaf’s confidence metrics ultimately reflect the accuracy of the output structure (which can differ from the decoy structure), while we were interested in scoring the decoy structures for the sake of direct comparison with other decoy-ranking methods.

9Using the procedure outlined above, we aimed to determine whether AlphaFold’s outputs could be used to assess the accuracy of decoy structures introduced as templates. For our initial evaluation we used the rosetta decoy dataset, which contains 133 native protein structures (targets) with thousands of decoys for each native structure. We compared alphafold’s ability to assess the quality of decoy structures with the rosetta energy function, as well as DeepAccNet, which is a state-of-the-art machine learning model for estimating the accuracy of protein structure models. All reported results are from alphafold model 1 with one recycling iteration.

10We found the correlation between the composite confidence score and decoy quality to be robust and consistent. The average Spearman rank correlation between the composite confidence score and the quality of the decoy (as measured by TM Score to the native structure) was 0.925, compared to average correlations of 0.831 and 0.760 for DeepAccNet and the rosetta energy function. Another practical indicator of decoy-ranking performance is the quality of the top-ranked decoy for each target. on the rosetta decoy dataset, the top-ranked decoys selected via the composite alphafold confidence score had an average TM Score of 0.933 compared to 0.917 for DeepAccNet and 0.901 for rosetta. More details on the rosetta dataset are given in Fig. 2.

11Overall, these evaluations indicate that alphafold can assess the quality of candidate protein structures with state-of-the-art accuracy, even when no coevolution information is provided. It should be noted that alphafold’s structure predictions were of low quality when no templates were provided (average TM score of 0.408). Yet despite being unable to predict the structures of these proteins without a MSA, alphafold achieved excellent performance assessing the quality of decoys without any MSA inputs. This provides evidence for the hypothesis that alphafold has learned an energy function that is largely independent of coevolution information, but needs coevolution information to search for global optima in this energy landscape.

12To assess the decoy-ranking ability of alphafold on a novel sample of proteins, we performed an additional evaluation on the estimation of model accuracy (EMA) task from casp14. To set up the casp14 EMA experiment, the CASP organizers created a set of decoy structures by taking the 150 most accurate server submissions for each structure prediction target in casp14. Note that the decoy set does not include predictions from alphafold, since alphafold was entered in casp14 as a human group rather than a server. We replicated this evaluation using alphafold (with the gap sequence) to assess the decoy structures, and compared the results with ranking methods entered in casp14.

13The CASP assessors evaluated EMA methods based on their top-1 GDT_TS loss, which is the difference in GDT_TS scores between the best decoy and the top-ranked decoy by a given EMA method. EMA methods were ranked based on their average GDT_TS loss over targets where at least one decoy had GDT_TS over 0.4, as well as the average Z-score of their GDT_TS loss over these targets. For both metrics, the alphafold composite confidence score significantly outperformed all other EMA methods entered in casp14. Results from the casp14 evaluation are presented in Fig. 3.

14These results indicate that alphafold can reliably assess the accuracy of candidate protein structures without the use of coevolution information. However, coevolution data (or a method that can generate decoys close to the correct structure) are still necessary for accurate structure prediction, since alphafold generally fails to predict accurate structures for the casp14 targets without a MSA (Fig. S3).

15Our finding that alphafold can assess the accuracy of candidate protein structures without the need for coevolution data opens up several exciting applications. One such application is the prediction of protein structures without MSAs. In theory, it should be possible to accurately predict protein structures by searching over the space of possible decoy structures and finding those that are highest ranked by alphafold. However, given the vast number of possible candidate structures, an exhaustive search is intractable.

16One way of mitigating this intractability is to search over the output space of a generative model of realistic protein structures. Instead of training a new generative model of candidate structures, we designed a generator-discriminator pipeline that links two instances of alphafold [Fig. 4(b)]. The first instance of alphafold (the generator) takes an arbitrary amino acid sequence as input, and produces a candidate protein structure as output. This candidate structure is then supplied to the discriminator as a template (with a sequence of gap tokens). Finally, the discriminator tries to predict the structure of the target sequence using the template, and produces confidence outputs in the process. As demonstrated by our previous experiments, these confidence metrics are strongly correlated with the accuracy of the candidate structure. By perturbing the input sequence to the generator, we can explore the space of candidate structures while using the discriminator’s confidence metrics as an indicator of accuracy. We performed this exploration by backpropagating the discriminator’s confidence signal to the input sequence and updating it via gradient ascent, thereby molding the input sequence to produce a high-quality candidate structure from the generator.

17Using this approach we were able to improve upon alphafold’s structure predictions when no MSAs were available. Although we did not use recycling in the generator and discriminator models, we compared the quality of our optimized predictions to a baseline created by running alphafold with a single sequence input and three recycling iterations. On the rosetta decoy set, we could significantly improve prediction quality (ΔTM score > 0.1) on 50 examples out of 123 compared to running the three-recycle baseline [Fig. 4(d)]. We hypothesize that this procedure is able to improve alphafold’s predictions because it performs a more wide-ranging search than the “unrolled optimizer” implemented by alphafold itself. Though our results demonstrate the potential of searching over candidate structures using alphafold’s learned energy function, the current optimization protocol sometimes gets stuck in local minima with low accuracies and low confidence scores (Figs. S11–13).

18Since alphafold’s learned energy function can determine the level of compatibility between a protein sequence and structure without the need for coevolution data, it is potentially applicable to protein design (i.e., the problem of finding a protein sequence that folds into a target backbone geometry). Our Letter suggests a straightforward approach to protein design using alphafold: supply the desired backbone structure to alphafold as a template (with a sequence of gap tokens and side chains masked), and optimize the composite confidence score with respect to the input sequence. To facilitate gradient-based optimization, we used the categorical cross entropy between alphafold’s predicted distance matrix and the template distance matrix as a surrogate loss for the composite confidence score. Using this loss circumvents the need to differentiate the TM score, which involves an iterative alignment procedure with potentially unstable gradients. The cross entropy loss can replace the entire composite score (including the pTM and pLDDT components), because when alphafold’s confidence in its output structure is low its predicted distance distributions become wider, thereby increasing the cross entropy. Pairing our target backbones with sequences designed by cross entropy optimization resulted in higher composite confidence scores than pairing them with their native sequences, indicating that optimizing the cross entropy loss effectively optimizes the composite confidence as well (Fig. S9).

19Fixed-backbone protein design methods are often benchmarked based on the average fraction of residues that match between the designed sequence and the true native sequence for the target backbone. Our design procedure achieved an average sequence recovery of 29.1% on the rosetta decoy dataset, which is comparable to energy-based design methods like rosetta.

20Repeating the same procedure without a template input resulted in significantly lower sequence recovery [Fig. 4(e)]. This is likely because, without a template or MSA, alphafold often fails to predict the correct structure for the input sequence. This leads to “false negatives” while optimizing the input sequence to match the target backbone (i.e., input sequences that would actually fold into the target backbone are mispredicted by alphafold and incorrectly assigned high loss). The template input eliminates false negatives by providing a good starting point for alphafold’s structural optimization, allowing alphafold to confidently and accurately predict when an input sequence will fold into the target structure. The effectiveness of template inputs at increasing sequence recovery supports our hypothesis that alphafold has learned an energy function that can assess sequence-structure agreement, but needs coevolution data or templates to help search for optimal structures.

21In this Letter we have provided evidence that alphafold has learned a protein structure energy function that does not need coevolution information to achieve high accuracy, although alphafold still needs coevolution data to search for global minima in this function. This finding has significance for the interpretation of protein structure prediction models, as well as practical applications. These applications include the prediction of protein structures when MSAs are not available and the improvement of protein design methods.

22The code used to run the evaluations in the Letter, as well as the raw data, is available at.