fastp 1.0: An ultra‐fast all‐round tool for FASTQ data quality control and preprocessing

1Sequencing technology is the core technology for exploring life sciences and conducting molecular diagnosis. Sequencing data, due to its complexity and proneness to errors, usually needs to undergo quality control and cleaning before it can enter downstream analysis. This preprocessing step usually includes performing various quality statistics on the original sequencing data, removing adapters, filtering low‐quality sequences or bases, and making some possible corrections to some sequencing errors. Many studies have shown the importance of data preprocessing in obtaining reliable analysis results. For instance, Wang et al. demonstrated that data preprocessing can significantly improve the accuracy of RNA expression analysis, and Bonfiglio et al. also strongly suggested FASTQ data filtering and trimming in their best practices for germline variant and DNA methylation analysis. For these reasons, preprocessing of FASTQ files becomes the first and very important step in sequencing data analysis.

2Due to the importance of data quality control and preprocessing, many tools have been developed to solve this problem. Among them, Trimmomatic, Cutadapt, and fastp are the most widely used. Cutadapt was first published in 2011, when high‐throughput sequencing began to explode. This software has been continuously iterated and has been widely used in areas such as removing adapters and primers. However, Cutadapt is weak in terms of quality control and data filtering, and lacks some modern features such as adapter detection, splitting, merging, and UMI processing. Trimmomatic was first published in 2014. Its initial function was mainly to remove Illumina adapter sequences, and some other trimming and quality filtering functions were later added. However, Trimmomatic has not been updated in the last 5 years, since it also lacks some key features like handling UMI or performing deduplication. Furthermore, Trimmomatic and Cutadapt cannot simultaneously perform comprehensive statistical quality analysis of the data while performing adapter removal and data filtering. This often requires users to run FastQC or MultiQC before and after using these tools to analyze the data quality improvement. This usually results in a significant waste of computing resources and increases the cost of data analysis.

3As a latecomer, fastp was first released in 2018 and has been undergoing continuous development and iteration. When fastp was first designed, it was considered how to reduce costs as much as possible, including learning costs and computing costs. Therefore, fastp will simultaneously perform a comprehensive quality analysis of the data before and after preprocessing while completing filtering and trimming. At the same time, fastp's algorithm is highly optimized, so even though it provides richer functions, it's much faster than Trimmomatic and Cutadapt. In addition, fastp was designed with the use scenario in the cloud in mind, so it requires limited memory to reduce cloud server cost and provides rich HTML reports for cloud‐based web viewing.

4FASTQ preprocessing has a wide range of applications in different scenarios, so software is continuously being developed. For example, BigSeqKit as a parallel FASTQ preprocessing toolkit for large FASTQ data, UMIc as a preprocessing method for UMI deduplication and reads correction, FastqPuri as a high‐performance preprocessing software of RNA‐seq data, and FastqCleaner as an interactive Bioconductor FASTQ filtering and trimming. However, this software has not yet made enough impact, possibly due to its imperfection in some aspects such as simplicity, functionality, and performance. There are also some software for miscellaneous processing of FASTQ files. For instance, SeqKit and SeqKit2 are great tools for FASTA/Q file manipulation, but are not specialized to preprocess data for the goal of trimming unwanted sequences and filtering unwanted reads. In this article, I mainly compare fastp with Trimmomatic and Cutadapt, as all three are the most popular FASTQ preprocessors.

5Fastp is a multithreaded multifunctional preprocessor for FASTQ streams. Its design follows the principles of simplicity, efficiency, versatility and reproducibility. Regardless of whether the input file is single‐end or paired‐end, or what parameters are used, its multi‐threaded queue mechanism makes all results stable and reproducible.

6Simplicity does not mean fewer features and weaker functions, but rather providing an interface that is easy to understand while hiding its implementation details. Simplicity means anticipating the needs of the vast majority of users and making the best choice for them by default. Fastp has gone through nearly 50 minor version iterations, but has always adhered to the simplicity rule. Fastp's simplicity is reflected in many aspects. For example, its default parameters are extremely simple, requiring only input and output files to be specified, and the default parameter mode is powerful enough to meet most common scenarios. For example, in most cases, users do not need to enter the adapter sequence because fastp can detect it automatically. In fact, the algorithms for automatically detecting adapters can be very complex, but users do not need to know this.

7Another example of the simplicity that fastp provides is its HTML reports. Compared to providing reports in plain text or CSV format, HTML is obviously more user‐friendly and very suitable in cloud computing environments. In fastp 1.0, the biggest change in its HTML report is that it provides a side‐by‐side comparison of pre‐processing and post‐processing data (Figure 1).

8In fastp 1.0, I take simplicity a step further by providing a script to batch process many FASTQ files in parallel. This script preprocesses all FASTQ files within a folder, and it can automatically couple the paired‐end FASTQ files. It generates an overall HTML report to present an aggregate summary for all processed FASTQ files. Figure 2 shows the overall HTML report of 10 FASTQ data.

9Efficiency usually has two meanings: high speed and low computational cost, both of which are important for bioinformatics analysis. High‐throughput sequencing files are typically very large and require a long time to complete, making computational speed crucial. Furthermore, most bioinformatics analysis processes are currently run in the cloud, requiring software to minimize CPU and memory requirements to reduce costs.

10A lot of efforts have been made to improve fastp's computing speed and reduce its computing resource requirements. For example, fastp applies a simultaneous mode of filtering and performing quality analysis at the same time. It only needs to read the data once to complete the trimming and filtering of the data and obtain the quality statistics results before and after preprocessing.

11Gap‐tolerant comparisons between sequences are very common to detect the presence of adapters or primers. For very short sequences, typically only one gap is allowed to guarantee the precision of comparisons. In such cases, applying sequence alignment algorithms or calculating edit distances would be extremely slow, resulting in significant computational overhead. In fastp 1.0, I designed a one‐gap‐matching algorithm to accelerate such operation. Simply put, for two sequences S 1 and S 2 of the same length n, while S 2 is supposed to contain one gap to match with S 1, I calculate the cumulative mismatch number array from left to right array as CUML2R, and from right to left as CUMR2L, respectively. Then we can quickly calculate the mismatch of S 1 and S 2 with one gap in S 2 using the algorithm described below:

12The algorithm is very simple but effective in reducing the computational complexity from O(n 2 ) to O(n). The processing of boundaries is not shown, since it's generally handled by callers.

13Due to the wide application of sequencing technology, the requirements for data preprocessing are also extremely diverse, which requires data preprocessors to provide a variety of functions. Table 1 shows the functional comparison among Trimmomatic, Cutadapt, and fastp. The results of this comparison referred to the documentation of this software. fastp provides most of the necessary functions (Table 1). Of course, Trimmomatic and Cutadapt are both excellent tools, and they can surpass fastp in some cases. Among them, there is a demultiplexing function that Cutadapt has, but fastp does not provide it, because I suggest that this should be the job of the demultiplexer. To maintain simplicity, some features that are less relevant were not provided.

14Reproducibility is an important principle that many bioinformatics software developers tend to overlook. Bioinformatics software, due to its complexity and diversity, is beyond the reach of most users to understand its in‐depth details and therefore relies on the software's stable output to produce reproducible results. Fastp 1.0 provides reproducible output regardless of the input and output, the modules enabled, or the parameters used. Reproducibility is often broken due to the use of threading or random functions. Fastp implements multi‐threading, but it aligns the input and output block queues to ensure that output results are not affected by out‐of‐order execution of threads, as demonstrated.

15Fastp's high efficiency is one of the main reasons for its widespread adoption by the bioinformatics community. A performance evaluation experiment was conducted to compare the time taken by Trimmomatic, cutadapt, and fastp to process the same data (Table 2). To simulate common low‐cost bioinformatics cloud analysis scenarios, this experiment was conducted on a low‐profile cloud server (Tencent Cloud, 8 cores, 16 G, ubuntu 24.04), with a network cloud disk as storage. Trimmomatic was installed via Anaconda with the latest available version (0.39). Cutadapt couldn't be installed via Anaconda due to Python version conflicts, so it was installed using APT with the latest available version (4.4). All three software are configured with 4 threads, input gzipped FASTQ files, and output gzipped FASTQ files as well. In the experiment, Cutadapt removed the adapter and polyG as suggested for data of two‐color chemistry; Trimmomatic removed the adapter and performed quality trimming according to its recommended parameters. For fastp, only input/output and the threading parameters are given. However, fastp actually performed adapter trimming, quality trimming, quality filtering, length filtering, duplicate analysis, insert‐size evaluation, etc. Even though fastp performed much more analysis operations, it's still much faster than Trimmomatic and Cutadapt (Table 2).

16I also employed FastQC to evaluate the data filtered by Trimmomatic, Cutadapt, and fastp, respectively. The script used in this evaluation and the FastQC results can be found at https://github.com/sfchen/fastq_preprocessor_test. From the result, we can learn that fastp‐filtered data is of very high quality, generally the best among the three. Trimmomatic‐filtered data is also of good quality, but it drops a lot of data. Cutadapt‐filtered data is of lower quality (especially for the R2 read), since it performed little besides removing adapters in this experiment.

17In this paper, I introduce the new features of fastp and the design philosophy behind it. I believe that simplicity, efficiency, versatility, and reproducibility are very important for bioinformatics, and it is recommended that bioinformatics developers follow these principles. In fact, continuing to improve simplicity, efficiency, and versatility is also the direction that fastp will continue to deepen in the future. For example, in terms of simplicity, more data types and multiple file inputs will be supported in the future. In terms of improving efficiency, fastp will introduce more SIMD technologies in the future to further improve performance and reduce computing costs. In addition, in terms of versatility, fastp will continue to iterate, while retaining ease of use, and continue to add the most requested features. The changelog of fastp from v0.1 to v1.0 can be found in Table S1. For end‐users and bioinformatics community contributors, it's suggested to keep watch on fastp's repository (https://github.com/OpenGene/fastp) to follow up on minor version updates.

18Shifu Chen: Conceptualization; investigation; funding acquisition; methodology; validation; visualization; writing—original draft; writing—review and editing; software; formal analysis; project administration; data curation; supervision; resources.

19The author declares no conflicts of interest.

20No animals or humans were involved in this study.