No retraction, correction or concern found when last checked on Oct 15, 2026.

Waste Not, Want Not: Why Rarefying Microbiome Data Is Inadmissible

Paul J. McMurdie, Susan Holmes

PLoS Computational Biology · 2014

doi:10.1371/journal.pcbi.1003531

Abstract

Current practice in the normalization of microbiome count data is inefficient in the statistical sense. For apparently historical reasons, the common approach is either to use simple proportions (which does not address heteroscedasticity) or to use rarefying of counts, even though both of these approaches are inappropriate for detection of differentially abundant species. Well-established statistical theory is available that simultaneously accounts for library size differences and biological variability using an appropriate mixture model. Moreover, specific implementations for DNA sequencing read count data (based on a Negative Binomial model for instance) are already available in RNA-Seq focused R packages such as edgeR and DESeq. Here we summarize the supporting statistical theory and use simulations and empirical data to demonstrate substantial improvements provided by a relevant mixture model framework over simple proportions or rarefying. We show how both proportions and rarefied counts result in a high rate of false positives in tests for species that are differentially abundant across sample classes. Regarding microbiome sample-wise clustering, we also show that the rarefying procedure often discards samples that can be accurately clustered by alternative methods. We further compare different Negative Binomial methods with a recently-described zero-inflated Gaussian mixture, implemented in a package called metagenomeSeq. We find that metagenomeSeq performs well when there is an adequate number of biological replicates, but it nevertheless tends toward a higher false positive rate. Based on these results and well-established statistical theory, we advocate that investigators avoid rarefying altogether. We have provided microbiome-specific extensions to these tools in the R package, phyloseq.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 75

1.
Shendure J, Lieberman Aiden E (2012 ) The expanding scope of DNA sequencing. Nature Biotechnology 30: 1084–1094.
Not in our library10.1038/nbt.2421
2.
Shendure J, Ji H (2008 ) Next-generation DNA sequencing. Nature Biotechnology 26: 1135–1145.
Not in our library10.1038/nbt1486
3.
Mortazavi A, Williams BA, McCue K, Schaeffer L, Wold B (2008 ) Mapping and quantifying mammalian transcriptomes by RNA-Seq. Nature Methods 5: 621–628.
Not in our library10.1038/nmeth.1226
4.
Pace NR (1997 ) A molecular view of microbial diversity and the biosphere. Science 276: 734–740.
Not in our library10.1126/science.276.5313.734
5.
Wilson KH, Wilson WJ, Radosevich JL, DeSantis TZ, Viswanathan VS, et al (2002 ) High-Density Microarray of Small-Subunit Ribosomal DNA Probes. Appl Environ Microbiol 68: 2535–2541.
Not in our library10.1128/aem.68.5.2535-2541.2002
6.
Exploring Microbial Diversity and Taxonomy Using SSU rRNA Hypervariable Tag Sequencing · 2008
Corrected
7.
Riesenfeld CS, Schloss PD, Handelsman J (2004 ) Metagenomics: genomic analysis of microbial communities. Annual Review of Genetics 38: 525–552.
Not in our library10.1146/annurev.genet.38.072902.091216
8.
Allison DB, Cui X, Page GP, Sabripour M (2006 ) Microarray Data Analysis: from Disarray to Consolidation and Consensus. Nature Reviews Genetics 7: 55–65.
Not in our library10.1038/nrg1749
9.
Marioni JC, Mason CE, Mane SM, Stephens M, Gilad Y (2008 ) RNA-Seq: an assessment of technical reproducibility and comparison with gene expression arrays. Genome Research 18: 1509–1517.
Not in our library10.1101/gr.079558.108
10.
Lu J, Tomfohr JK, Kepler TB (2005 ) Identifying differential expression in multiple SAGE libraries: an overdispersed log-linear model approach. BMC Bioinformatics 6: 165.
Not in our library10.1186/1471-2105-6-165
11.
Robinson MD, Smyth GK (2007 ) Small-sample estimation of negative binomial dispersion, with applications to SAGE data. Biostatistics (Oxford, England) 9: 321–332.
Not in our library10.1093/biostatistics/kxm030
12.
Cameron AC, Trivedi P (2013) Regression analysis of count data, volume 53. Cambridge University Press.
Not in our library
13.
Differential expression analysis for sequence count data · 2010
No notice
14.
Yu D, Huber W, Vitek O (2013 ) Shrinkage estimation of dispersion in Negative Binomial models for RNASeq experiments with small sample size. Bioinformatics (Oxford, England) 29: 1275–1282.
Not in our library10.1093/bioinformatics/btt143
15.
Di Bella JM, Bao Y, Gloor GB, Burton JP, Reid G (2013 ) High throughput sequencing methods and analysis for microbiome research. Journal of Microbiological Methods 95: 401–414.
Not in our library10.1016/j.mimet.2013.08.011
16.
Computational meta'omics for microbial community studies · 2013
No notice
17.
Navas-Molina JA, Peralta-Sánchez JM, González A, McMurdie PJ, Vázquez-Baeza Y, et al (2013 ) Advancing Our Understanding of the Human Microbiome Using QIIME. Methods in Enzymology 531: 371–444.
Not in our library10.1016/b978-0-12-407863-5.00019-8
18.
Hughes JB, Hellmann JJ (2005 ) The application of rarefaction techniques to molecular inventories of microbial diversity. Methods in Enzymology 397: 292–308.
Not in our library10.1016/s0076-6879(05)97017-1
19.
A Guide to Enterotypes across the Human Body: Meta-Analysis of Microbial Community Structures in Human Microbiome Datasets · 2013
No notice
20.
Sanders HL (1968 ) Marine benthic diversity: A comparative study. The American Naturalist 102: 243–282.
Not in our library
21.
Gotelli NJ, Colwell RK (2001 ) Quantifying biodiversity: procedures and pitfalls in the measurement and comparison of species richness. Ecology Letters 4: 379–391.
Not in our library
22.
Mao CX, Colwell RK (2005 ) Estimation of Species Richness: Mixture Models, the Role of Rare Species, and Inferential Challenges. Ecology 86: 1143–1153.
Not in our library
23.
Lozupone C, Knight R (2005 ) UniFrac: a new phylogenetic method for comparing microbial communities. Applied and Environmental Microbiology 71: 8228–8235.
Not in our library10.1128/aem.71.12.8228-8235.2005
24.
Lozupone C, Lladser ME, Knights D, Stombaugh J, Knight R (2011 ) UniFrac: an effective distance metric for microbial community comparison. The ISME Journal 5: 169–172.
Not in our library10.1038/ismej.2010.133
25.
Hamady M, Walker JJ, Harris JK, Gold NJ, Knight R (2008 ) Error-correcting barcoded primers for pyrosequencing hundreds of samples in multiplex. Nature Methods 5: 235–237.
Not in our library10.1038/nmeth.1184
26.
Liu Z, DeSantis TZ, Andersen GL, Knight R (2008 ) Accurate taxonomy assignments from 16S rRNA sequences produced by highly parallel pyrosequencers. Nucleic Acids Research 36: e120.
Not in our library10.1093/nar/gkn491
27.
Hamady M, Lozupone C, Knight R (2010 ) Fast UniFrac: facilitating high-throughput phylogenetic analyses of microbial communities including analysis of pyrosequencing and PhyloChip data. The ISME Journal 4: 17–27.
Not in our library10.1038/ismej.2009.97
28.
Yatsunenko T, Rey FE, Manary MJ, Trehan I, Dominguez-Bello MG, et al (2012 ) Human gut microbiome viewed across age and geography. Nature 486: 222–227.
Not in our library10.1038/nature11053
29.
Caporaso J, Kuczynski J, Stombaugh J, Bittinger K, Bushman F, et al (2010 ) QIIME allows analysis of high-throughput community sequencing data. Nature Methods 7: 335–336.
Not in our library10.1038/nmeth.f.303
30.
Schloss PD, Westcott SL, Ryabin T, Hall JR, Hartmann M, et al (2009 ) Introducing mothur: Open-Source, Platform-Independent, Community-Supported Software for Describing and Comparing Microbial Communities. Applied and Environmental Microbiology 75: 7537–7541.
Not in our library10.1128/aem.01541-09
31.
Gilbert JA, Field D, Swift P, Newbold L, Oliver A, et al (2009 ) The seasonal structure of microbial communities in the Western English Channel. Environmental Microbiology 11: 3132–3139.
Not in our library10.1111/j.1462-2920.2009.02017.x
32.
phyloseq: An R Package for Reproducible Interactive Analysis and Graphics of Microbiome Census Data · 2013
No notice
33.
Disordered Microbial Communities in the Upper Respiratory Tract of Cigarette Smokers · 2010
No notice
34.
The Effects of Circumcision on the Penis Microbiome · 2010
No notice
35.
Kembel SW, Jones E, Kline J, Northcutt D, Stenson J, et al (2012 ) Architectural design influences the diversity and structure of the built environment microbiome. The ISME Journal 6: 1469–1479.
Not in our library10.1038/ismej.2011.211
36.
Flores GE, Bates ST, Caporaso JG, Lauber CL, Leff JW, et al (2013 ) Diversity, distribution and sources of bacteria in residential kitchens. Environmental Microbiology 15: 588–596.
Not in our library10.1111/1462-2920.12036
37.
Reduced Incidence of Prevotella and Other Fermenters in Intestinal Microflora of Autistic Children · 2013
No notice
38.
Composition of the adult digestive tract bacterial microbiome based on seven mouth surfaces, tonsils, throat and stool samples · 2012
No notice
39.
Statistical Methods for Detecting Differentially Abundant Features in Clinical Metagenomic Samples · 2009
No notice
40.
Paulson JN, Stine OC, Bravo HC, Pop M (2013 ) Differential abundance analysis for microbial marker-gene surveys. Nature Methods 10: 1200–1202.
Not in our library10.1038/nmeth.2658
41.
Robinson MD, McCarthy DJ, Smyth GK (2009 ) edgeR: a Bioconductor package for differential expression analysis of digital gene expression data. Bioinformatics (Oxford, England) 26: 139–140.
Not in our library10.1093/bioinformatics/btp616
42.
Gower JC (1966 ) Some distance properties of latent root and vector methods used in multivariate analysis. Biometrika 53: 325–338.
Not in our library
43.
Oksanen J, Blanchet FG, Kindt R, Legendre P, O'Hara RB, et al.. (2011) vegan: Community Ecology Package. R package version 1.17-10.
Not in our library
44.
Anderson M (2001 ) A new method for non-parametric multivariate analysis of variance. Austral Ecology 26: 32–46.
Not in our library
45.
Bray JR, Curtis JT (1957 ) An Ordination of the Upland Forest Communities of Southern Wisconsin. Ecological Monographs 27: 325.
Not in our library
46.
Witten DM (2011 ) Classification and clustering of sequencing data using a Poisson model. The Annals of Applied Statistics 5: 2493–2518.
Not in our library
47.
Lozupone CA, Hamady M, Kelley ST, Knight R (2007 ) Quantitative and qualitative beta diversity measures lead to different insights into factors that structure microbial communities. Applied and Environmental Microbiology 73: 1576–1585.
Not in our library10.1128/aem.01996-06
48.
Caporaso JG, Lauber CL, Walters WA, Berg-Lyons D, Lozupone CA, et al (2011 ) Global patterns of 16S rRNA diversity at a depth of millions of sequences per sample. Proceedings of the National Academy of Sciences 108: 4516–4522.
Not in our library10.1073/pnas.1000080107
49.
Kaufman L, Rousseeuw PJ (1990) Finding Groups in Data: An Introduction to Cluster Analysis, JohnWiley & Sons, chapter 2.
Not in our library
50.
Reynolds A, Richards G, Iglesia B, Rayward-Smith V (2006 ) Clustering rules: A comparison of partitioning and hierarchical clustering algorithms. Journal of Mathematical Modelling and Algorithms 5: 475–504.
Not in our library
51.
Pollard KS, Gilbert HN, Ge Y, Taylor S, Dudoit S (2010) multtest: Resampling-based multiple hypothesis testing. R package version 2.4.0.
Not in our library
52.
Benjamini Y, Hochberg Y (1995 ) Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society Series B (Methodological) 57: 289–300.
Not in our library
53.
Allaire J, Horner J, Marti V, Porte N (2014) markdown: Markdown rendering for R. R package version 0.6.4.
Not in our library
54.
Maechler M, Rousseeuw P, Struyf A, Hubert M, Hornik K (2013) cluster: Cluster Analysis Basics and Extensions.
Not in our library
55.
Revolution Analytics (2011) foreach: Foreach looping construct for R. R package version 1.3.2.
Not in our library
56.
Wickham H (2009) ggplot2: elegant graphics for data analysis. Springer New York.
Not in our library
57.
Wickham H (2011 ) The split-apply-combine strategy for data analysis. Journal of Statistical Software 40: 1–29.
Not in our library
58.
Wickham H (2007 ) Reshaping data with the reshape package. Journal of Statistical Software 21: 1–20.
Not in our library
59.
Sing T, Sander O, Beerenwinkel N, Lengauer T (2005 ) ROCR: visualizing classifier performance in R. Bioinformatics (Oxford, England) 21: 3940–3941.
Not in our library10.1093/bioinformatics/bti623
60.
R Development Core Team (2011) R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0.
Not in our library
61.
Hastie TJ, Pregibon D (1992) Generalized linear models. In: Chambers JM, Hastie TJ, editors, Statistical Models in S, Chapman & Hall/CRC, chapter 6..
Not in our library
62.
Nookaew I, Papini M, Pornputtapong N, Scalcinati G, Fagerberg L, et al (2012 ) A comprehensive comparison of RNA-Seq-based transcriptome analysis from reads to differential gene expression and crosscomparison with microarrays: a case study in Saccharomyces cerevisiae. Nucleic Acids Research 40: 10084–10097.
Not in our library10.1093/nar/gks804
63.
Evaluation of statistical methods for normalization and differential expression in mRNA-Seq experiments · 2010
No notice
64.
TCC: an R package for comparing tag count data with robust normalization strategies · 2013
No notice
65.
A comparison of methods for differential expression analysis of RNA-seq data · 2013
No notice
66.
baySeq: Empirical Bayesian methods for identifying differential expression in sequence count data · 2010
No notice
67.
Ozer HG, Parvin JD, Huang K (2012 ) DFI: gene feature discovery in RNA-Seq experiments from multiple sources. BMC Genomics 13 Suppl 8: S11.
Not in our library10.1186/1471-2164-13-s8-s11
68.
Bourgon R, Gentleman R, Huber W (2010 ) Independent filtering increases detection power for highthroughput experiments. Proceedings of the National Academy of Sciences 107: 9546–9551.
Not in our library10.1073/pnas.0914005107
69.
Chao A, Chazdon RL, Colwell RK, Shen TJ (2005 ) A new statistical approach for assessing similarity of species composition with incidence and abundance data. Ecology Letters 8: 148–159.
Not in our library
70.
Schloss PD (2008 ) Evaluating different approaches that test whether microbial communities have the same structure. The ISME Journal 2: 265–275.
Not in our library10.1038/ismej.2008.5
71.
Gentleman R, Temple Lang D (2004 ) Statistical analyses and reproducible research. Bioconductor Project Working Papers 1: 1–38.
Not in our library
72.
Peng RD (2011 ) Reproducible research in computational science. Science 334: 1226–1227.
Not in our library10.1126/science.1213847
73.
Donoho DL (2010 ) An invitation to reproducible computational research. Biostatistics (Oxford, England) 11: 385–388.
Not in our library10.1093/biostatistics/kxq028
74.
Bioconductor: open software development for computational biology and bioinformatics · 2004
No notice
75.
Wu GD, Chen J, Hoffmann C, Bittinger K, Chen YY, et al (2011 ) Linking long-term dietary patterns with gut microbial enterotypes. Science 334: 105–108.
Not in our library10.1126/science.1208344

Cited by papers we hold 111

Networks as tools for defining emergent properties of microbiomes and their stability · 2024
No notice
Rarefaction is currently the best approach to control for uneven sequencing effort in amplicon sequence analyses · 2024
No notice
Influence of nutrient supply on plankton microbiome biodiversity and distribution in a coastal upwelling region · 2022
Not checked
Tree Size Drives Diversity and Community Structure of Microbial Communities on the Bark of Beech (Fagus sylvatica) · 2022
Not checked
Grapevine treatment with bagasse vermicompost changes the microbiome of Albariño must and wine and improves wine quality · 2022
Not checked
Investigating differential abundance methods in microbiome data: A benchmark study. · 2022
Not checked
A comprehensive evaluation of microbial differential abundance analysis methods: current status and potential solutions · 2022
No notice
Microbiome differential abundance methods produce different results across 38 datasets · 2022
Corrected
Association of Broiler Litter Microbiome Composition and Campylobacter Isolation · 2021
Not checked
Clinical health issues, reproductive hormones, and metabolic hormones associated with gut microbiome structure in African and Asian elephants · 2021
Not checked
Maternal consumption of artificially sweetened beverages during pregnancy is associated with infant gut microbiota and metabolic modifications and increased infant body mass index · 2021
Not checked
You are more than what you eat: potentially adaptive enrichment of microbiome functions across bat dietary niches · 2021
Not checked
Enhancing diversity analysis by repeatedly rarefying next generation sequencing data describing microbial communities · 2021
No notice
Multivariable association discovery in population-scale meta-omics studies · 2021
No notice
Seasonal shifts in the gut microbiome indicate plastic responses to diet in wild geladas · 2021
No notice
Conserved and reproducible bacterial communities associate with extraradical hyphae of arbuscular mycorrhizal fungi · 2021
No notice
Open challenges for microbial network construction and analysis · 2021
No notice
A quantitative sequencing framework for absolute abundance measurements of mucosal and lumenal microbial communities · 2020
Corrected
Gut Microbiome Profiles Are Associated With Type 2 Diabetes in Urban Africans · 2020
No notice
Quorum Sensing Signals Alter in vitro Soil Virus Abundance and Bacterial Community Composition · 2020
Not checked

and 91 more

Reads

Read by people on this page

None yet.

Checked or read by AI through naulon

None yet.

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.

Are you an author?

Is this your paper?

Claim it to add your note and be paid when agents cite it. Free.