Fine-tuning protein language models boosts predictions across diverse tasks

1How to speak protein? Transformer-based language models (LMs) have revolutionized Natural Language Processing (NLP). Large language models (LLMs) now perform at or above average human level. The newest generative models (GPT4 or PaLM2) expand upon NLP through impressive capabilities in coding, math, and even common sense reasoning. The success of LLMs has led to their widespread application from computer vision over time series forecasting to biologic language models–. For instance, protein language models (pLMs) are trained on many protein sequences–. PLMs learn from large data sets without any experimental annotation other than the sequence. The information extracted by the pLM, more precisely the value describing the last hidden layers, dubbed the embeddings, can be readily transferred to any protein-related prediction task. This generality makes pLMs suitable to a wide variety of prediction tasks spanning from secondary structure over membrane regions, intrinsic disorder, protein structure,, and protein-protein interaction to predictions of stability, or solubility. Successful applications to more function-related predictions include the identification of paratopes, epitopes, and signal peptides, as well as, other tasks, e.g., related to the effect of sequence variation–. Embedding-based predictions seem particularly advantageous when experimental data are very limited. Effectively, the embeddings from pLMs condense the understanding of the language of life,. Over the last 30 years, the de facto standard in protein prediction has been the use of evolutionary information, i.e., information from Multiple Sequence Alignments (MSAs) as input to machine learning. Now, pLM-based predictions have reached and often even superseded the MSA-based state-of-the-art (SOTA) expert devices for many prediction tasks. Embeddings can be input to artificial feed-forward (ANN) or convolutional neural networks (CNN). More complex architectures have also been explored. Continued unsupervised training can focus models on specific protein families or enrich embeddings with structural information essentially creating a bi-lingual pLM. Training specialist models from scratch on smaller, specific proteins, e.g., antibodies,, seems an alternative to continued training (train pLM on large generic data and refine on specific proteins).

2In contrast to the training recipes described above, which extract static representations from the pLM’s last hidden layer without changing its weights, it was shown for NLP that finetuning the parameters of the LLM is beneficial for downstream prediction performance–. This aligns well with the fact that the last layer’s representation is not always optimal, and finetuning gives access to information stored in any layer of the model. However, for pLMs, fine-tuning remains less studied than for NLP, although some prediction tasks have been reported to profit from supervised, task-specific fine-tuning,–. Here, we evaluated the impact of task-specific supervised fine-tuning by adding a simple ANN as a prediction head on top of the pLM encoder and applied supervised training to both, the pLM encoder and the prediction head. We compared the results to predictions using static, pre-trained embeddings (no finetuning of the pLM). For the larger models (ProtT5, ProstT5, both Ankh models, and ESM2 3B), we utilized Low Rank Adaptation (LoRA), a particular version of a more general approach known as PEFT (Parameter Efficient Fine-Tuning). Freezing most of the model and updating only a small fraction of the weights (Table 1) accelerates training and prevents catastrophic forgetting,. This will become especially relevant, as pLMs follow the NLP trend where bigger models usually translate to better downstream performance. We assessed diverse prediction tasks from eight previously established benchmarks through the lens of three pLMs from which we derived some rules of thumb (and scripts) that simplify navigating the vastly growing space of PEFT methods applied to pLMs.

3We trained 615 individual prediction methods (295 for fine-tuning, 320 using frozen embeddings from pre-trained pLMs - protein Language Models) comprising eight models (Table 1), each trained on eight different data sets (Table 2). We trained each model-task combination multiple times with different random seeds and all results constituted averages over those runs. The corresponding validation set selected the best training stop. For each prediction task (PT), we compared the performance between fine-tuning and pre-training (1). For ProtT5-XL-U50 (labeled ProtT5) and all five tested ESM2 versions (differing in parameter size between 8 M (8*10^6) and 3B (3*10^9)), not all improvements were statistically significant within the 95% confidence interval (CI: Methods). Nevertheless, supervised fine-tuning numerically increased performance for almost all combinations (Fig. 1, detailed results in supplementary online material (SOM) Tables S1–S6). The exceptions were ESM2-150M applied to Stability prediction, and both Ankh models. Ankh gained significantly by fine-tuning only for the mutational landscape data (GFP, AAV, and GB1: blue in Fig. 1).

4For these data, performance relied less on transfer from model pretraining (Fig. S7) and mainly depended on the underlying transformer architecture. This might explain why Ankh performed similarly to ProtT5 and the ESM2. For the diverse data sets, this was not the case. Two major factors differentiate Ankh from the other pLMs. Firstly, the T5 masked span pre-training differs from that of BERT-like objective used for the other models. Secondly, the training procedure and architecture of Ankh was optimized using data (GFP, GB1, subcellular location, and secondary structure) also utilized in this work. This might have reduced the ability to fine-tune these models.

5For five of the 64 pLM/task combinations (tiles in Fig. 1), fine-tuning performed worse. The observation ESM2-150M on stability (Fig. 1 red tile) originated from instability in training picking a suboptimal model (Fig. S5). The other four originated from the Ankh pLM family on disorder and secondary structure. We were not able to track down a root cause here but suspect that the different nature of the pre-training plays a role.

6For ProtT5 and sub-cellular location prediction, we compared three parameter-efficient fine-tuning methods to LoRA. Not having sufficient resources to do this analysis for all prediction tasks/pLMs, we chose this problem due to its limit in size and because of the success of fine-tuning on this problem (configuration in Method and Fig. 2). The fraction of trained model parameters were 0.25% for LoRA, 0.28% for DoRA, 0.12% for IA3 and 0.5% for Prefix tuning. Despite these differences, runtimes for training and testing (inference) were within ±10% between methods, except for DoRA which was about 30% slower than the other three. In terms of prediction performance, LoRA and DoRA outperformed IA3 and Prefix-tuning (Fig. 2). Overall, all fine-tuning methods improved, on average, over pre-trained embeddings (61.3% from Table S5). As no method improved significantly over the well-established LoRA, we used it throughout our experiments. Of course, these results for a single model and dataset must not hold true in general. We encourage to explore parameter efficient fine-tuning of pLMs, utilizing new combinations of high-quality datasets, state-of-the-art models, and PEFT methods in future work and hope the notebooks made available by us help to pursue this research more easily.

7For per-residue, three-class secondary structure prediction (helix, strand, other), fine-tuning improved only slightly (Fig. 2a; up to 1.2 percentage points for CASP12 and NEW364). We confirmed this for the general-purpose ProtT5 and the bilingual, structure-tuned ProstT5. Two effects might have hindered substantial improvement. Firstly, secondary structure might already have been captured in unsupervised pre-training. In fact, embeddings already capture some aspects of inter-residue contact formation,. Secondly, performance may have reached an upper limit. One limitation of the benchmark is highlighted by the two data sets (CASP12 and NEW364). Both were introduced to gauge the performance for unknown proteins. Other than that CASP12 is much smaller (12 proteins vs. 364) implying higher statistical errors, there seems no a priori reason for choosing one over the other, and no method compared here is expected to have any systematic bias toward any of the two. Thus, the difference between both should estimate the statistical error. In other words, bootstrapping error estimates should be similar to the difference between the two sets. This was not the case at all (Fig. 3a: differences between CASP12 and NEW364 exceeded standard errors marked by distributions). Arguably, secondary structure prediction assessment is the best-solved task in protein structure prediction since decades,. Even for this relatively trivial problem, such a simple dichotomy seems not easily resolvable. In fact, the standard mantra: larger data sets without redundancy appears not to solve this dichotomy. These results underscore how difficult it is to just plug in standard data sets to assess the performance of prediction methods without updating data and adapting it to advancing methods.

8PLM-based SETH reached the level of MSA-based SOTA methods, such as ODiNPred in the prediction of per-residue protein disorder as described by CheZOD scores. SETH inputs ProtT5 embeddings into a two-layer CNN.

9Keeping those hyper-parameters and adding LoRA fine-tuning (dubbed SETH-LoRA), improved performance by 2.2 percentage points (from Spearman 0.72 to 0.736, Fig. 3b). Fine-tuning the much smaller 150 M parameter ESM2 model (Spearman: 0.742) improved overall solutions compared (Fig. 3b), including its larger counterparts (ESM2 with 650M/3B parameters, Table S4). Compared to SETH-LoRA where only 2.5 million out of its 1.2 billion parameters are trained, for ESM2-150M all parameters were fine-tuned. Both approaches (2.5 m for ProtT5 vs 150 m for ESM2) performed similarly (Fig. 3b).

10Most predictions of subcellular location input signals averaged over entire proteins (e.g., amino acid composition). Embedding-based solutions do this through pooling, i.e., through embeddings derived from averaging over all intrinsic residue-level embeddings. Light Attention (LA) substantially improves over such coarse-grained averaging by learning the optimal per-residue signal and combining this with the average. LoRA fine-tuning combined the advantage of a small model (fewer free parameters) with the learned, weighted averaging of LA. Thereby, LoRA fine-tuning numerically surpassed LA, although the difference was statistically significant only at an 88% confidence interval (CI and not at the more common CI = 95% Table S9).

11For predicting mutation landscapes (Fig. 1 leftmost three columns) fine-tuning any pLM succeeded substantially. As differences between fine-tuned models were small (Fig. S3), we averaged performance across all fine-tuned pLMs (Fig. 4, for individual values refer to Table S2), and compared to homology-based inference (HBI, using MMseqs2 search) and to reference-free analysis (RFA). RFA fits a decent first-order model for the fitness landscape reflecting some mutations for GB1 (protein G domain B1; all possible variants for four residues, i.e., at four positions). For AAV2 (adeno-associated virus 2) for which a much larger 28-residue window was mutated, RFA performed less well. For GFP (green fluorescent protein) the RFA analyses failed because some specific substitutions XnY (amino acid X at position n mutated to Y) occurred only in the test set. The fact that smaller and larger models performed alike on these tasks raised the prospect of using small, fine-tuned pLMs as computationally affordable, high-quality solutions for protein engineering.

12The main drivers for the amount of computational resources required for model training were the parameter sizes of pLMs along with quadratic scaling of the attention mechanism (more resources for longer proteins). More recent GPUs used for LLM training (anything beyond 40GB of memory) will have sufficient memory to allow usage of all pLMs tested here. For less powerful hardware (Fig. 5b), mixed precision training nearly halved the required GPU memory without performance loss (both Ankh models were exceptional, as they do not support mixed precision training). Where GPU memory still was a bottleneck, we applied gradient accumulation to reduce the actual on-device batch size as far as needed. When even an on-device batch size of 1 was insufficient, we used DeepSpeed to offload the optimizer and potential parameters to CPU-reduced GPU memory requirements further. As a trade-off, both gradient accumulation and CPU offloading slowed down training. Hence, both should be used cautiously. Implementing all these measures, we could fine-tune most pLMs tested here even on older GPUs with as little as 8GB memory (Fig. 5b). Unintuitively, both full model fine-tuning and parameter-efficient LoRA fine-tuning required the same amount of GPU memory and only differed in training speed (Fig. 5a) when CPU offloading was utilized. Embedding creation required much less GPU memory rendering it feasible even for datasets with very long sequences (Fig. S1).

13To ease the simplicity of fine-tuning pLMs for your data set, we added the following recommendations. Before starting model training, dataset splits to measure model generalization and prevent over-estimating performance, are essential. First off: you need at least three data sets: training (optimizing weights), cross-training/validation (optimization of hyper-parameters, e.g., to decide between CNN and ANN), and testing (only touched to estimate performance). Typically, all entities in the test set (e.g. proteins) should adhere to the same split required between training/validation and testing. In particular, proteins have to be made non-redundant. This requires clustering by sequence identity using standard alignment methods such as MMseqs2 (simpler solutions tend to lead more likely to information leakage). For structure-related tasks, redundancy is best removed through 3D clustering as realized by Foldseek. To optimize the prediction of mutational landscapes for a single protein, it might be best to train on k-mers with k = 1 (single amino acid variants) and test on k-mers with k > 1, (although this approach might focus more on avoiding over-fitting than on generating the best optimal model).

14To predict landscapes of mutational effects for specific proteins, a challenge encountered in protein engineering, we recommend to first fine-tune a smaller pLM (pre-trained embeddings were limited: Fig. S3). Optimize hyperparameters and head architectures on this smaller model. If done, you could explore additional improvements from larger pLMs. For the fine-tuning on diverse tasks, larger mostly outperformed smaller models (Figs. S3 & S4). Therefore, starting with raw embedding-based solutions to identify the best model and to then investigate different prediction heads appeared better than optimizing the fine-tuning directly. Applying parameter-efficient LoRA fine-tuning and optimizing hyperparameters for the selected model afterward, will probably lead to an even better solution.

15For our tasks, over-fitting mostly originated from data set characteristics. On the one hand, given a data set prone to over-fitting (e.g. too small, uninformative, or complex), neither hyperparameter nor model optimization could fully avoid the trap. On the other hand, for data sets not prone to over-fitting the training of fine-tuning was stable regardless of other factors. These factors affected raw embedding-based and fine-tuned models alike. Avoiding imbalanced datasets, providing sufficient high-quality training data, and choosing smaller models for limited data sets could mitigate over-fitting (SOM Sections 9 and 10).

16On the computational side, we recommend mixed precision training. Gradient accumulation and DeepSpeed’s CPU-offloading should only be reserved to avoid memory constraints. With all these measures in place, a single 16 GB GPU enables fine-tuning in many cases (Fig. 5). Comparing different common PEFT methods (Fig. 2) did not suggest a clear winner. The established LoRA method was among the best solutions and was stable across our experiments. The codebase provided by us simplifies experimenting with different PEFT approaches as it utilizes the Hugging Face PEFT framework. We encourage you to compare different PEFT methods for your specific use cases. PEFT is memory efficient, but CPU-offloading could achieve the same. However, PEFT is also most compute-efficient for larger pLMs (Fig. 5a); it stabilizes training (SOM Section 5) and it renders saving model checkpoints orders of magnitude more memory efficient, as only the trained parameters need to be stored. Thus, we recommend LoRA fine-tuning for all models larger than ESM2 150 M (Fig. S2). We see little reason not to fully fine-tune smaller models.

17In our hands, different random initialization seeds significantly altered results. These random variations reached the magnitude effect of hyperparameters or even model selection.

18We applied fine-tuning, to a diversity of prediction tasks clearly showing improvements, on average. The extent of this improvement varied by task and pLM/model and was impacted by the amount of training data (Fig. S10), dataset balance (Table S8), models size (Fig. S3), and initial representation quality (Fig. S4).

19Overall, our results revealed the gains initially observed in NLP from supervised task-specific fine-tuning of LLMs, to also apply to large protein LMs (pLMs). Supervised fine-tuning unlocks additional degrees of freedom in the predictor models. The last hidden layer has been optimized for the unsupervised pre-training objective (learning to reproduce masked sequences). This optimization might be suboptimal for any downstream task,. PEFT (or finetuning in general) enables information from middle layers to flow to the last layer, making it accessible to downstream tasks. Additionally, for per-protein predictions, the LoRA optimization may have learned weighted pooling of the last hidden layer, and that improved significantly over average pooling. Lastly, the transformer models might extract additional information directly from the task-specific training. Randomly initialized smaller ESM2 models supported this view (Fig. S7, Table S17).

20Therefore, we suggest to add supervised fine-tuning whenever applying transfer-learning, i.e., when inputting pLM embeddings into subsequent supervised prediction tasks. Our results suggested that you will most often benefit from this. To ease this additional step, we provided all our resources and added step-by-step recommendations.

21The pLMs used differed in size and architecture (Table 1), ranging from 8 million (ESM2 8 M) to 3 billion parameters (ESM2 3B). ESM2 pLMs are RoBERTa based encoder models trained using an unsupervised masked language modeling objective. The other three pLMs are built on T5, an encoder-decoder LLM pre-trained applying span masking,. We initialized our models using the pre-trained checkpoints available on Huggingface.

22The data sets differentiated two aspects: prediction task level and sequence diversity (Table 2). The prediction task level collected cases of per-residue (e.g. secondary structure) and per-protein (e.g. sub-cellular location) prediction. Sequence diversity is distinguished between data sets with many sequence-diverse proteins and those from mutational studies analyzing single proteins through deep mutational scanning (DMS also known as MAVE) experiments.

23Mutational landscapes: these data described fitness landscapes for three individual proteins: the green fluorescent protein (GFP), the adeno-associated virus 2 capsid protein VP-1 (AAV2), and the first binding domain of the protein G (GB1). All three constitute regression tasks that measure prediction performance by ranking the correlation between the predicted and the experimentally measured property for each set. For GFP the property/fitness was measured through fluorescence intensity (experimental data, data split). Training and validation set sequences were all within Hamming distance 3 (i.e. all variants up to three changes from wild type). For the AAV task, fitness was measured as the viability for packaging DNA payloads by mutating 28-amino acid window. We used the 2-vs-rest data split from the FLIP benchmark (variants with ≤2 in training and validation sets, those with more in the test set). The GB1 fitness score measures stability and binding affinity. The original experiment mutated four positions, while we took the three-vs-rest data split from FLIP for easier comparability to existing benchmarks.

24Per-protein prediction tasks included three prediction tasks. The first two sets focused on stability prediction formulated as regression tasks in analogy to the fitness landscapes. Stability predictions were assessed on measurements of protease susceptibility to digestion by de novo-designed mini-proteins. We reused the TAPE data split in which training and validation sets contain sequences from four design cycles while the test set holds neighborhoods at 1-Hamming distance to 17 promising candidates. Meltome utilizes data from measuring thermostability for proteins from 13 species. We used the mixed split from the FLIP benchmark, which clusters proteins at >20% pairwise sequence identity (PIDE) through MMseqs2. Excluding any pair within the same cluster from test and train/validation set, is the minimal means to reducing potential information leakage. The third data set included DeepLoc data (incl. training/validation/testing splits) along with a novel test data set (setHARD). The task is to predict the sub-cellular location in one of ten classes. MMseqs2 removed all sequence pairs at >20 PIDE between training, validation, and test sets.

25Per-residue prediction tasks included disorder and secondary structure. Disorder predictions used CheZOD data from nuclear magnetic resonance (NMR) spectroscopy. We used previously published data splits clustering at <20 PIDE with MMseqs2. The task is to predict CheZOD scores which quantify the level of intrinsic disorder for each residue in a protein (each amino acid position) through a continuous scale. We bench-marked secondary structure predictions through data sets provided by NetSurfP-2.0 distinguishing three classes (H: helix, E: strand, and C: other/non-regular). In splitting these data, we used a recent split with more stringent redundancy-reduction, along with another test set (NEW364).

26Performance measures were copied from the data set developers (Table 2). All regression tasks were evaluated using Spearman rank correlations, i.e., all three mutational landscapes (GFP, AAV, GB1), both stability-related data sets (Stability and Meltome), as well as, the per-residue regression of Disorder. For the classification tasks, accuracy was defined as a 10-class per-protein accuracy (Q10) for sub-cellular location, and as a 3-class per-residue accuracy (Q3) for secondary structure (for a detailed per-class analysis see SOM Section 2).

27Top-level comparisons contrasted pre-trained to fine-tuned models as follows. For the pre-trained results, we generated embeddings for all data sets. For per-protein tasks, we averaged over the sequence length; for each protein, this yielded a vector of dimension 1 x embedding_size. For per-residue tasks, we used all residue embeddings and their labels; this resulted in vectors of the same dimension as for the per-protein task (1 x embedding_size), albeit this time for each residue (i.e., L*embedding_size for a protein of length L). Next, we trained a single fully connected layer of size 32, inputting exclusively these embeddings and outputting either a single value (regression) or going into another output layer with one neuron for each possible output class, followed by a softmax layer to get a probability distribution. Reaching a plateau in training loss terminated training. We repeated each training five times with different random seeds. For fine-tuning, we added the same fully connected layer (size 32) to the pLM encoder as a prediction head. For ProtT5 and Ankh, we used average pooling of the last hidden states over the sequence length dimension during training on per-protein tasks. Following the author’s advice for ESM2, we connected the prediction head only to the very first token (special token <CLS>). The training was repeated three times with different random seeds. Training terminated when training and validation loss flattened out. To increase training efficiency and save time, we applied Parameter Efficient Fine-Tuning (PEFT) to all models. In particular, we applied the Low-Rank Adaptation (LoRA) implementation of PEFT. For the smaller ESM2 models (up to the 650M version), we could investigate full model fine-tuning (SOM Section 3 and Fig. S1).

28For embedding and fine-tuning we reported the performance of the checkpoint with the lowest validation loss for each run. Finally, we simply computed the percentage differences between the fine-tuned and pre-trained models (1):

29We also performed a limited hyperparameter optimization at the beginning (Table S11). Once selected, the hyperparameters were frozen for most comparisons (Tables S10 and S12). We used the Adam optimizer with default parameters. For LoRA fine-tuning we reused a previously suggested configuration, namely rank 4, alpha 1, applied to query, key, value, and the output of the attention layers. A minimal LoRA rank of four has also been suggested previously for pLMs. To realize the batch size (Table S12) for fine-tuning, we applied gradient accumulation as needed given our hardware. Initially, we fine-tuned models on full precision but switched to mixed precision for larger models. Embedding generation used half precision. The Ankh models only support full precision for both. All training ran on a single NVIDIA A10G GPU with 24GB. We used Torch version 1.13.1 with transformers version 4.26.1.

30Training times for fine-tuning depended crucially on the available GPU. Nevertheless, the following basic trends were notable. Training times were mostly driven by model and dataset size, as well as, the quadratic scaling with sequence length. As all these factors accumulate, smaller models and shorter sequences led to much faster training (Fig. 5a). For instance, for the small Disorder data (average sequence length: 118 residues), fine-tuning the full ESM2-8M took 12 minutes, while the much larger ProtT5 (LoRA fine-tuning) took 190 minutes (16-fold increase). On the other end, for the AAV data (average length: 736 residues with nearly 30k proteins), ESM2-8M training took 130 minutes while a single ProtT5 training ran over 4 K minutes (>30-fold increase). The time to train an embedding-based prediction method was mostly determined by the time needed to compute embeddings because the relatively small predictor models that we used required negligible runtime. The creation of embeddings took approximately as long as fine-tuning the same model for a single epoch (typically we needed 5-50 epochs, Table S12).

31Per-residue secondary structure: We fine-tuned five models for ProtT5 and another five for ProstT5, initializing with different random seeds. For ProstT5, we added the prefix token <AA2fold> to each sequence to code the input type (amino acid rather than structure as ProstT5 is a bilingual pLM). We trained for five epochs and calculated the validation loss at the end. For both models (ProtT5 and ProstT5), utilizing the same two-layer CNN as applied previously, for pre-trained embeddings to simplify comparisons. Of the five, we selected the model with the lowest validation loss and measured performance on common data sets (CASP12 and NEW364). Bootstrapping established confidence intervals (Fig. 2a).

32Per-residue disorder fine-tuning stacked up two model variants to compare to other methods. SETH-LoRA used ProtT5 with the same two-layer CNN as the original SETH. ESM2 150 M reused the ESM2 setup from the top-level evaluation (last hidden states of <CLS> token with single dense layer prediction head). For both variants, we trained five models with different random seeds for ten epochs, and calculated the validation loss twice per epoch, selecting the model with the lowest validation loss out of 100 checkpoints (5 random seeds, 10 epochs, 2 points per epoch). Bootstrapping provided confidence intervals (Fig. 3b).

33Per-protein subcellular location: For the 10-class classification, we reused the single-layer dense network from our top-level evaluation. We trained five models with different random seeds for five epochs, calculating validation loss twice per epoch. We selected the model with the lowest validation loss in each run, and then calculated Q10 accuracy and standard errors from all five models on setHARD and reported the averages (Table S9).

34Fitness landscapes contrasted fine-tuned pLMs to two baselines, namely, homology-based inference (HBI) and reference-free analysis (RFA). We averaged test performance overall fine-tuned pLMs (checkpoints with lowest loss on validation set) for the three mutational landscape data sets (GFP, AAV, GB1).

35HBI: MMSeqs2 searched each query (Q) in the test set against all proteins training set proteins. The fitness value of the top training hit (closest homolog) predicted that for Q.

36RFA: We applied the R implementation of RFA (version 1.0.0) without modification to the GB1 data. For AAV, we removed sequences not containing 735 residues (the algorithm failed on insertions and deletions). This reduced the training data by about 1%. RFA failed for GFP because some substitutions in the test set were missing from the training set. For AAV and GB1, we fitted a first- and second-order model and reported results for the better of the two.

37Replication of computational results is made possible by defining random states (seed). These random seeds are given in the detailed results (Table S1 and S2). The code and data provided by us will allow complete reproduction of our results, in addition, results for each individual model training run are provided with the source data.

38During our research, we excluded a single training run (Finetuning ESM2 150 M on the AAV dataset, seed 98) which was a clear outlier, for details check Table S2.

39Since random data splits lead to a large overestimation of predictor performance, we used dataset splits from previous work (Table 2) specifically designed to test for generalization.

40Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.