No retraction, correction or concern found when last checked on Oct 31, 2026.

Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study

Eddie Guo, Mehul Gupta, Jiawen Deng and 3 more

Journal of Medical Internet Research · 2024

Abstract

Background. The systematic review of clinical research papers is a labor-intensive and time-consuming process that often involves the screening of thousands of titles and abstracts. The accuracy and efficiency of this process are critical for the quality of the review and subsequent health care decisions. Traditional methods rely heavily on human reviewers, often requiring a significant investment of time and resources.

Objective. This study aims to assess the performance of the OpenAI generative pretrained transformer (GPT) and GPT-4 application programming interfaces (APIs) in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review data sets and comparing their performance against ground truth labeling by 2 independent human reviewers.

Methods. We introduce a novel workflow using the Chat GPT and GPT-4 APIs for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the API with the screening criteria in natural language and a corpus of title and abstract data sets filtered by a minimum of 2 human reviewers. We compared the performance of our model against human-reviewed papers across 6 review papers, screening over 24,000 titles and abstracts.

Results. Our results show an accuracy of 0.91, a macro F1-score of 0.60, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. The interrater variability between 2 independent human screeners was κ=0.46, and the prevalence and bias-adjusted κ between our proposed methods and the consensus-based human decisions was κ=0.96. On a randomly selected subset of papers, the GPT models demonstrated the ability to provide reasoning for their decisions and corrected their initial decisions upon being asked to explain their reasoning for incorrect classifications.

Conclusions. Large language models have the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, models such as GPT-4 can enhance efficiency and lead to more accurate and reliable conclusions in medical research.

References 39

1.
Sargeant JM, O'Connor AM. Scoping reviews, systematic reviews, and meta-analysis: applications in veterinary medicine. Front Vet Sci. 2020;7:11.
Not in our library10.3389/fvets.2020.00011
2.
Garritty C, Stevens A, Hamel C, Golfam M, Hutton B, Wolfe D. Knowledge synthesis in evidence-based medicine. Semin Nucl Med. 2019;49(2):136-144.
Not in our library10.1053/j.semnuclmed.2018.11.006
3.
Luchini C, Veronese N, Nottegar A, Shin JI, Gentile G, Granziol U, Soysal P, Alexinschi O, Smith L, Solmi M. Assessing the quality of studies in meta-research: review/guidelines on the most important quality assessment tools. Pharm Stat. 2021;20(1):185-195.
Not in our library10.1002/pst.2068
4.
Gartlehner G, Affengruber L, Titscher V, Noel-Storr A, Dooley G, Ballarini N, König F. Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial. J Clin Epidemiol. 2020;121:20-28.
Not in our library10.1016/j.jclinepi.2020.01.005
5.
Fletcher J. What is heterogeneity and is it important? BMJ. 2007;334(7584):94-96.
Not in our library10.1136/bmj.39057.406644.68
6.
The Cochrane Collaboration, Higgins JPT, Thomas J. Cochrane Handbook for Systematic Reviews of Interventions. 2021.
Not in our library
7.
Chai KEK, Lines RLJ, Gucciardi DF, Ng L. Research screener: a machine learning tool to semi-automate abstract screening for systematic reviews. Syst Rev. 2021;10(1):93.
Not in our library10.1186/s13643-021-01635-3
8.
Clark J, McFarlane C, Cleo G, Ramos CI, Marshall S. The impact of systematic review automation tools on methodological quality and time taken to complete systematic review tasks: case study. JMIR Med Educ. 2021;7(2):e24418.
Not in our library10.2196/24418
9.
Tuijn S, Janssens F, Robben P, van den Bergh H. Reducing interrater variability and improving health care: a meta-analytical review. J Eval Clin Pract. 2012;18(4):887-895.
Not in our library10.1111/j.1365-2753.2011.01705.x
10.
Better duplicate detection for systematic reviewers: evaluation of Systematic Review Assistant-Deduplication Module · 2015
No notice
11.
Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses. Res Synth Methods. 2019;10(3):330-342.
Not in our library10.1002/jrsm.1354
12.
Toward systematic review automation: a practical guide to using machine learning tools in research synthesis · 2019
No notice
13.
Marshall IJ, Trikalinos TA, Soboczenski F, Yun HS, Kell G, Marshall R, Wallace BC. In a pilot study, automated real-time systematic review updates were feasible, accurate, and work-saving. J Clin Epidemiol. 2023;153:26-33.
Not in our library10.1016/j.jclinepi.2022.08.013
14.
Blaizot A, Veettil SK, Saidoung P, Moreno-Garcia CF, Wiratunga N, Aceves-Martins M, Lai NM, Chaiyakunapruk N. Using artificial intelligence methods for systematic review in health sciences: a systematic review. Res Synth Methods. 2022;13(3):353-362.
Not in our library10.1002/jrsm.1553
15.
Marshall I, Kuiper J, Banner E, Wallace BC. Automating biomedical evidence synthesis: RobotReviewer. 2017;7-12.
Not in our library10.18653/v1/p17-4002
16.
Marshall IJ, Nye B, Kuiper J, Noel-Storr A, Marshall R, Maclean R, Soboczenski F, Nenkova A, Thomas J, Wallace BC. Trialstreamer: a living, automatically updated database of clinical trial reports. J Am Med Inform Assoc. 2020;27(12):1903-1912.
Not in our library10.1093/jamia/ocaa163
17.
Hamel C, Kelly SE, Thavorn K, Rice DB, Wells GA, Hutton B. An evaluation of DistillerSR's machine learning-based prioritization tool for title/abstract screening—impact on reviewer-relevant outcomes. BMC Med Res Methodol. 2020;20(1):256.
Not in our library10.1186/s12874-020-01129-1
18.
Gates A, Johnson C, Hartling L. Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool. Syst Rev. 2018;7(1):45.
Not in our library10.1186/s13643-018-0707-8
19.
Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. J Am Med Inform Assoc. 2016;23(1):193-201.
Not in our library10.1093/jamia/ocv044
20.
Nye BE, Nenkova A, Marshall IJ, Wallace BC. Trialstreamer: mapping and browsing medical evidence in real-time. Proc Conf. 2020;2020:63-69.
Not in our library10.18653/v1/2020.acl-demos.9
21.
Moreno-Garcia CF, Jayne C, Elyan E, Aceves-Martins M. A novel application of machine learning and zero-shot classification methods for automated abstract screening in systematic reviews. Decis Anal J. 2023;6:100162.
Not in our library10.1016/j.dajour.2023.100162
22.
Semi-automated screening of biomedical citations for systematic reviews · 2010
No notice
23.
Rayyan—a web and mobile app for systematic reviews · 2016
No notice
24.
Olofsson H, Brolund A, Hellberg C, Silverstein R, Stenström K, Österberg M, Dagerhamn J. Can abstract screening workload be reduced using text mining? User experiences of the tool Rayyan. Res Synth Methods. 2017;8(3):275-280.
Not in our library10.1002/jrsm.1237
25.
Shree P. The journey of Open AI GPT models. Medium. 2020.
Not in our library
26.
Using text mining for study identification in systematic reviews: a systematic review of current approaches · 2015
Corrected
27.
Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study · 2024
No notice
28.
Assessing the research landscape and utility of LLMs in the clinical setting: protocol for a scoping review. OSF Registries.
Not in our library
29.
Deng J, Zhou F, Ali S, Heybati K, Hou W, Huang E, Wong C Y. Efficacy and safety of ivermectin for the treatment of COVID-19: a systematic review and meta-analysis. QJM. 2021;114(10):721-732.
Not in our library10.1093/qjmed/hcab247
30.
Deng J, Rayner D, Ramaraju HB, Abbas U, Garcia C, Heybati K, Zhou F, Huang E, Park YJ, Moskalyk M. Efficacy and safety of selective serotonin reuptake inhibitors in COVID-19 management: a systematic review and meta-analysis. Clin Microbiol Infect. 2023;29(5):578-586.
Not in our library10.1016/j.cmi.2023.01.010
31.
Deng J, Zhou F, Hou W, Heybati K, Ali S, Chang O, Silver Z, Dhivagaran T, Ramaraju HB, Wong CY, Zuo QK, Lapshina E, Mellett M. Efficacy of lopinavir-ritonavir combination therapy for the treatment of hospitalized COVID-19 patients: a meta-analysis. Future Virol. 2021.
Not in our library10.2217/fvl-2021-0066
32.
Zhou F, Huang E, Zheng E, Deng J. The use of acupuncture in patients with Raynaud's syndrome: a systematic review and meta-analysis of randomized controlled trials. Acupunct Med. 2023;41(2):63-72.
Not in our library10.1177/09645284221076504
33.
Heybati K, Zhou F, Lynn MJ, Deng J, Ali S, Hou W, Heybati S, Tzanis K, Krever M, Mughal R, Ramakrishna H. Comparative efficacy of adjuvant nonopioid analgesia in adult cardiac surgical patients: a network meta-analysis. J Cardiothorac Vasc Anesth. 2023;37(7):1169-1178.
Not in our library10.1053/j.jvca.2023.03.018
34.
Zhang Y, Liang S, Feng Y, Wang Q, Sun F, Chen S, Yang Y, He X, Zhu H, Pan H. Automation of literature screening using machine learning in medical evidence synthesis: a diagnostic test accuracy systematic review protocol. Syst Rev. 2022;11(1):11.
Not in our library10.1186/s13643-021-01881-5
35.
van de Schoot R, de Bruin J, Schram R, Zahedi P, de Boer J, Weijdema F, Kramer B, Huijts M, Hoogerwerf M, Ferdinands G, Harkema A, Willemsen J, Ma Y, Fang Q, Hindriks S, Tummers L, Oberski DL. An open source machine learning framework for efficient and transparent systematic reviews. Nat Mach Intell. 2021;3(2):125-133.
Not in our library10.1038/s42256-020-00287-7
36.
Przybyła P, Brockmeier AJ, Kontonatsios G, Le Pogam M, McNaught J, von Elm E, Nolan K, Ananiadou S. Prioritising references for systematic reviews with RobotAnalyst: a user study. Res Synth Methods. 2018;9(3):470-488.
Not in our library10.1002/jrsm.1311
37.
Gates A, Guitard S, Pillay J, Elliott SA, Dyson MP, Newton AS, Hartling L. Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools. AHRQ Methods for Effective Health Care. 2019.
Not in our library
38.
Schade M. Fine-tuning a classifier to improve truthfulness. OpenAI.
Not in our library
39.
Joshua J. What is the difference between the GPT-4 models? OpenAI.
Not in our library

Reads

Read by people on this page

None yet.

Checked or read by AI through naulon

None yet.

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.

Are you an author?

Is this your paper?

Claim it to add your note and be paid when agents cite it. Free.