A study of generative large language model for medical research and healthcare

Abstract

There are enormous enthusiasm and concerns in applying large language models (LLMs) to healthcare. Yet current assumptions are based on general-purpose LLMs such as ChatGPT, which are not developed for medical use. This study develops a generative clinical LLM, GatorTronGPT, using 277 billion words of text including (1) 82 billion words of clinical text from 126 clinical departments and approximately 2 million patients at the University of Florida Health and (2) 195 billion words of diverse general English text. We train GatorTronGPT using a GPT-3 architecture with up to 20 billion parameters and evaluate its utility for biomedical natural language processing (NLP) and healthcare text generation. GatorTronGPT improves biomedical natural language processing. We apply GatorTronGPT to generate 20 billion words of synthetic text. Synthetic NLP models trained using synthetic text generated by GatorTronGPT outperform models trained using real-world clinical text. Physicians’ Turing test using 1 (worst) to 9 (best) scale shows that there are no significant differences in linguistic readability ( p = 0.22; 6.57 of GatorTronGPT compared with 6.93 of human) and clinical relevance ( p = 0.91; 7.0 of GatorTronGPT compared with 6.97 of human) and that physicians cannot differentiate them ( p < 0.001). This study provides insights into the opportunities and challenges of LLMs for medical research and healthcare.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 56

1. 1.

   Introducing ChatGPT. https://openai.com/blog/chatgpt.

   Not in our library

2. 2.

   Lee P, Bubeck S, Petro J. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. N. Engl. J. Med. 2023;388:1233-1239.

   Not in our library10.1056/nejmsr2214184

3. 3.

   Patel SB, Lam K. ChatGPT: the future of discharge summaries? Lancet Digit Health. 2023;5:e107-e108.

   Not in our library10.1016/s2589-7500(23)00021-3

4. 4.

   Ali SR, Dobbs TD, Hutchings HA, Whitaker IS. Using ChatGPT to write patient clinic letters. Lancet Digit Health. 2023;5:e179-e181.

   Not in our library10.1016/s2589-7500(23)00048-1

5. 5.
   Diagnostic Accuracy of Differential-Diagnosis Lists Generated by Generative Pretrained Transformer 3 Chatbot for Clinical Vignettes with Common Chief Complaints: A Pilot Study · 2023

   No notice
6. 6.

   Grünebaum, A., Chervenak, J., Pollet, S. L., Katz, A. & Chervenak, F. A. The Exciting Potential for ChatGPT in Obstetrics and Gynecology. Am. J. Obstet. Gynecol. 10.1016/j.ajog.2023.03.009 (2023).

   Not in our library10.1016/j.ajog.2023.03.009

7. 7.
   Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research Scenarios · 2023

   No notice
8. 8.
   Large language models and the perils of their hallucinations · 2023

   No notice
9. 9.

   Straw I, Callison-Burch C. Artificial Intelligence in mental health and the biases of language based models. PLoS One. 2020;15:e0240376.

   Not in our library10.1371/journal.pone.0240376

10. 10.

    Li, H. et al. Ethics of large language models in medicine and medical research. Lancet Digital Health10.1016/S2589-7500(23)00083-3 (2023).

    Not in our library10.1016/s2589-7500(23)00083-3

11. 11.

    Kojima, T., Gu, S. S., Reid, M., Matsuo, Y. & Iwasawa, Y. Large Language Models are Zero-Shot Reasoners. Adv. Neural Inf. Process. Syst. 35, 22199–213 (2022).

    Not in our library

12. 12.

    Bommasani, R. et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021).

    Not in our library

13. 13.

    Brown T, Mann B, Ryder N. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 2020;33:1877-1901.

    Not in our library

14. 14.

    Liu P, et al. Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 2023;55:1-35.

    Not in our library

15. 15.
    A large language model for electronic health records · 2022

    No notice
16. 16.

    Gao, L. et al. The Pile: an 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027 (2020).

    Not in our library

17. 17.

    Floridi L, Chiriatti M. GPT-3: its nature, scope, limits, and consequences. Minds Mach. 2020;30:681-694.

    Not in our library10.1007/s11023-020-09548-1

18. 18.

    Luo, R. et al. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Brief. Bioinform. 23, bbac409 (2022).

    Not in our library10.1093/bib/bbac409

19. 19.

    Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) 4171–4186 (Association for Computational Linguistics, 2019). 10.18653/v1/N19-1423.

    Not in our library

20. 20.

    Mohammed, M., Khan, M. B. & Bashier, E. B. M. Machine Learning (CRC Press, 2016). 10.1201/9781315371658.

    Not in our library

21. 21.
    The shaky foundations of large language models and foundation models for electronic health records · 2023

    No notice
22. 22.
    MIMIC-III, a freely accessible critical care database · 2016

    No notice
23. 23.

    Searle T, Ibrahim Z, Teo J, Dobson R. Estimating redundancy in clinical text. J. Biomed. Inform. 2021;124:103938.

    Not in our library10.1016/j.jbi.2021.103938

24. 24.

    Li J, et al. Are synthetic clinical notes useful for real natural language processing tasks: a case study on clinical entity recognition. J. Am. Med. Inform. Assoc. 2021;28:2193-2201.

    Not in our library10.1093/jamia/ocab112

25. 25.

    Huguet Cabot, P.-L. & Navigli, R. REBEL: relation extraction by end-to-end language generation. in Findings of the Association for Computational Linguistics: EMNLP 2021 2370–2381 (Association for Computational Linguistics, 2021). 10.18653/v1/2021.findings-emnlp.204.

    Not in our library

26. 26.

    Peng, C. et al. Clinical concept and relation extraction using prompt-based machine reading comprehension. J. Am. Med. Inform. Assoc. 10.1093/jamia/ocad107 (2023).

    Not in our library10.1093/jamia/ocad107

27. 27.

    Gaffney A, et al. Medical documentation burden among US office-based physicians in 2019: a national study. JAMA Intern. Med. 2022;182:564-566.

    Not in our library10.1001/jamainternmed.2022.0372

28. 28.

    Downing NL, Bates DW, Longhurst CA. Physician burnout in the electronic health record era: are we ignoring the real cause? Ann. Intern. Med. 2018;169:50.

    Not in our library10.7326/m18-0139

29. 29.
    Association of Electronic Health Record Design and Use Factors With Clinician Stress and Burnout · 2019

    No notice
30. 30.

    Diaz, N. Epic to use Microsoft’s GPT-4 in EHRs. https://www.beckershospitalreview.com/ehrs/epic-to-use-microsofts-open-ai-in-ehrs.html.

    Not in our library

31. 31.

    Trang, B. We’re getting much more aggressive’: Microsoft’s Nuance adds GPT-4 AI to its medical note-taking tool. https://www.statnews.com/2023/03/20/microsoft-nuance-gpt4-dax-chatgpt/.

    Not in our library

32. 32.
    Health system-scale language models are all-purpose prediction engines · 2023

    No notice
33. 33.

    Kleesiek, J., Wu, Y., Stiglic, G., Egger, J. & Bian, J. An opinion on ChatGPT in health care-written by humans only. J. Nucl. Med. 10.2967/jnumed.123.265687 (2023).

    Not in our library10.2967/jnumed.123.265687

34. 34.

    Ouyang, L. et al. Training language models to follow instructions with human feedback. arXiv [cs.CL] (2022).

    Not in our library

35. 35.

    Ray, S. Samsung bans ChatGPT among employees after sensitive code leak. Forbes Magazine (2023).

    Not in our library

36. 36.

    Caliskan A, Bryson JJ, Narayanan A. Semantics derived automatically from language corpora contain human-like biases. Science. 2017;356:183-186.

    Not in our library10.1126/science.aal4230

37. 37.

    Center for Devices & Radiological Health. Artificial Intelligence and Machine Learning in Software as a Medical Device. U.S. Food and Drug Administrationhttps://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device.

    Not in our library

38. 38.

    Yang X, et al. A study of deep learning methods for de-identification of clinical notes in cross-institute settings. BMC Med. Inform. Decis. Mak. 2019;19:232.

    Not in our library10.1186/s12911-019-0935-4

39. 39.

    Levine, Y., Wies, N., Sharir, O., Bata, H. & Shashua, A. The depth-to-width interplay in self-attention. arXiv [cs.LG] (2020).

    Not in our library

40. 40.

    Shoeybi, M. et al. Megatron-LM: training multi-billion parameter language models using model parallelism. arXiv [cs.CL] (2019).

    Not in our library

Cited by papers we hold 2

• The Breakthrough of Large Language Models Release for Medical Applications: 1-Year Timeline and Perspectives · 2024
  No notice
• Optimization of hepatological clinical guidelines interpretation by large language models: a retrieval augmented generation-based framework · 2024
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.