Error rates of human reviewers during abstract screening in systematic reviews

Abstract

BACKGROUND: Automated approaches to improve the efficiency of systematic reviews are greatly needed. When testing any of these approaches, the criterion standard of comparison (gold standard) is usually human reviewers. Yet, human reviewers make errors in inclusion and exclusion of references. OBJECTIVES: To determine citation false inclusion and false exclusion rates during abstract screening by pairs of independent reviewers. These rates can help in designing, testing and implementing automated approaches. METHODS: We identified all systematic reviews conducted between 2010 and 2017 by an evidence-based practice center in the United States. Eligible reviews had to follow standard systematic review procedures with dual independent screening of abstracts and full texts, in which citation inclusion by one reviewer prompted automatic inclusion through the next level of screening. Disagreements between reviewers during full text screening were reconciled via consensus or arbitration by a third reviewer. A false inclusion or exclusion was defined as a decision made by a single reviewer that was inconsistent with the final included list of studies. RESULTS: We analyzed a total of 139,467 citations that underwent 329,332 inclusion and exclusion decisions from 86 unique reviewers. The final systematic reviews included 5.48% of the potential references identified through bibliographic database search (95% confidence interval (CI): 2.38% to 8.58%). After abstract screening, the total error rate (false inclusion and false exclusion) was 10.76% (95% CI: 7.43% to 14.09%). CONCLUSIONS: This study suggests important false inclusion and exclusion rates by human reviewers. When deciding the validity of a future automated study selection algorithm, it is important to keep in mind that the gold standard is not perfect and that achieving error rates similar to humans may be adequate and can save resources and time.

References 24

1. 1.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

2. 2.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

3. 3.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

4. 4.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

5. 5.
   Expediting systematic reviews: methods and implications of rapid reviews · 2010

   No notice
6. 6.
   Methodology in conducting a systematic review of systematic reviews of healthcare interventions · 2011

   No notice
7. 7.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

8. 8.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

9. 9.

   This reference’s wording can’t be shown under the paper’s licence.

   Not in our library

10. 10.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

11. 11.
    Evidence summaries: the evolution of a rapid review approach · 2012

    No notice
12. 12.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

13. 13.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

14. 14.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

15. 15.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

16. 16.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

17. 17.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

18. 18.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

19. 19.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

20. 20.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

21. 21.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

22. 22.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

23. 23.
    Using text mining for study identification in systematic reviews: a systematic review of current approaches · 2015

    Corrected
24. 24.

    This reference’s wording can’t be shown under the paper’s licence.

    Not in our library

Cited by papers we hold 2

• Artificial intelligence in systematic reviews: promising when appropriately used · 2023
  No notice
• PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews · 2021
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.