Applications and Concerns of ChatGPT and Other Conversational Large Language Models in Health Care: Systematic Review

1Since ChatGPT (OpenAI) was released on November 30, 2022, extensive attention has been drawn to generative artificial intelligence (AI) and large language models (LLMs). ChatGPT is a representative conversational LLM that generates text based on its training on an extremely large amount of data from mostly the public domain. Modern LLMs (such as GPT-4) incorporate in-text learning, which enables them to interpret and generalize user inputs in the form of natural language prompts that require little to no fine-tuning. These LLMs have surpassed their non–transformer-based counterparts and are now capable of performing various complex natural language processing tasks, including translation and question-answering. In comparison with traditional chatbots, the current array of conversational LLMs can generate seemingly human-like coherent texts. Moreover, because these models are trained on publications from digital libraries, such as Common Crawl and Wikipedia, they can generate seemingly scientific and competent answers.

2Due to the high quality of their responses and the broad training database of modern LLMs, a growing body of studies has emerged regarding the applications of chatbots, particularly ChatGPT, in the domain of health and medicine. However, most LLMs are not specially designed for health care, and therefore, certain practical pitfalls may exist when they are put into practice in that setting. Thus, there is a need to compile the latest achievements in this domain so that potential issues and guidance for new research directions can be laid out. Several reviews have been published to discuss the appropriateness of a particular application of LLMs in a specific aspect but none of them summarized the overall problems systematically. For example, Huang et al and Giannakopoulos et al summarized the application of ChatGPT only in dentistry without considering the broader landscape of other subfields in health care. Wang et al discussed the ethical considerations of using ChatGPT in health care; they did not consider other LLMs for analysis, account for other common challenges, such as reliability, or mention detailed applications of the models. Moreover, their work focused on LLMs’ educational and research applications rather than their clinical use. Although Sallam conducted a systematic review, the articles considered in the review were mostly editorials, letters to the editors, opinions, commentaries, news articles, and preprints, as opposed to research articles. In addition, Sallam focused on educational and research applications of ChatGPT only. Puladi et al narratively reviewed papers on the applications of LLMs in oral and maxillofacial surgery. Pool et al reviewed papers on the application of LLMs in telehealth. Park et al conducted a scoping review of papers on the medical applications of LLMs. These papers are limited in either focusing on a specific medical application area, including nonpeer-reviewed articles, or lacking a systematic examination of the concerns regarding conversational LLMs.

3This review focuses on peer-reviewed research articles on conversational LLMs that emerged after ChatGPT, which was initially based on GPT-3 (OpenAI), and their applications in health care. We aim to summarize the applications of conversational LLMs in the field of health care with concrete applications and identify potential concerns about the use of such LLMs in this field that need to be addressed in the future.

4We searched for articles that contained at least 1 word associated with LLMs (“ChatGPT,” “LLaMA,” “GPT-3,” “LaMDA,” “PalM,” “MT-NLG,” “GATO,” “BLOOM,” “Alpaca,” “Large Language Model”) and at least 1 word associated with health care (“health,” “diagnosis,” “intervention,” “patient”) published before September 1, 2023, on PubMed, ACM Digital Library, and IEEE Xplore. This systematic review applied the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines (Multimedia Appendix 1) to steer the literature search. Relevant publications were gathered and downloaded on September 3, 2023. For simplicity, all the LLMs mentioned henceforth refer to conversational LLMs.

5Textbox 1 summarizes the inclusion and exclusion criteria for articles. Specifically, the inclusion criteria for a paper were as follows: (1) it was published as a peer-reviewed scientific research article between November 1, 2022, and September 1, 2023, and (2) it focuses on applications of LLMs in addressing a health care–related problem, which includes, but is not limited to, promotion of personal or public health and well-being or the potential to alleviate the workload of health care providers. We excluded a paper if it was (1) not a peer-reviewed research article, (2) not related to health care applications (eg, LLMs applied to preparing manuscripts for peer review), (3) not accessible, (4) a duplicate of an existing paper, or (5) about LLMs released before GPT-3, such as bidirectional encoder representations from transformers (BERT). We excluded BERT-related papers because this LLM, which was built upon the encoder of a transformer, is mainly applied in fine-tuning downstream machine-learning tasks. While the implementation of a chatbot based on BERT is feasible, it waned in popularity as an LLM after the introduction of ChatGPT, which was built upon the decoder of a transformer. The complete set of papers meeting the criteria were downloaded from the 3 digital libraries for further screening. Specifically, 5 of the authors of this review (LW, ZW, CN, QS, and YL) participated in paper screening and summarization under the supervision of the corresponding author, ZY. A screening protocol was created collectively after the team jointly reviewed 50 randomly selected papers. Each unreviewed paper was then screened by not fewer than 2 authors based on the protocol. All the papers in the final collection were summarized by the coauthors according to their LLM applications in health care and the concerns raised.

6Inclusion criteria

7Article type: peer-reviewed scientific research article

8Written language: English

9Time of publications: published between November 1, 2022, and September 1, 2023

10Accessibility: accessible

11Duplication: is not a duplicate of an existing article

12Models: conservational large language models (LLMs) after GPT-3 was launched

13Topic: any topics related to health care, which includes, but is not limited to, promotion of personal or public health and well-being or the potential to alleviate the workload of health care providers

14Exclusion criteria

15Article type: any other types of publications

16Written language: any non-English language

17Time of publications: published before November 1, 2022, or after September 1, 2023

18Accessibility: not accessible

19Duplication: is a duplicate of an existing article

20Models: LLMs before GPT-3 was launched or not used for conversations

21Topic: any other topics that are not related to health care applications (eg, preparing manuscripts for peer review)

22Figure 1 demonstrates the paper selection process. The initial keyword search identified 820 articles, with 736 (89.8%) articles from PubMed, 49 (6%) papers from ACM Digital Library, and 35 (4.3%) papers from IEEE Xplore. The evaluation of the 820 articles was distributed among the authors for screening the titles and abstracts. The interrater reliability was assessed by computing a κ score, yielding a value of 0.72. After screening, we excluded 599 (81.4%) of the 736 articles from PubMed, 46 (94%) of the 49 articles from ACM Digital Library, and 33 (94%) of the 35 papers from IEEE Xplore because they were either not relevant to the research topic or were not research articles. No duplicates were found after the screening. Next, we extracted the full papers for the remaining 142 (17.3%) of 820 research articles and manually examined them for the 5 exclusion criteria (refer to the Methods section). This led to a final set of 65 (7.9%) of 820 articles for full-paper review and summarization—63 (97%), 2 (3%), and 0 from PubMed, ACM Digital Library, and IEEE Xplore, respectively. Among these selected articles, 60 (92%) were related to ChatGPT, 1 (2%) was related to LLaMA (Meta), 1 (2%) was related to Bard based on Language Model for Dialogue Applications (Google LLC), and 6 (9%) were related to other LLMs (Table 1 lists the specific LLM or LLMs mentioned in each selected paper). Note that 2 selected papers were related to >1 LLM, respectively.

23Five of the authors (LW, ZW, CN, QS, and YL) compiled the topics related to applications and concerns independently during the paper screening and summarization process. Furthermore, through extensive discussions, all the authors refined and categorized these topics into main applications and concerns with corresponding subcategories. Figure 2 illustrates the main topics of applications and concerns mentioned by the reviewed papers on applying LLMs in health care settings. The multifaceted applications of LLMs can be divided into 4 primary categories: summarization, medical knowledge inquiry, prediction, and administration: summarization (25/65, 38% papers)—LLMs are potential tools for summarizing complex information or documentation in clinical domains. Medical knowledge inquiry (30/65, 46% papers)—LLMs demonstrate proficiency in answering a diverse array of medical questions and examinations, which enhance public access to medical knowledge. Prediction (22/65, 34% papers)—LLMs demonstrate high diagnostic accuracy in multiple medical scenarios (15/65, 23% papers), offer assistance in diverse treatments (12/65, 18% papers), and excel in predicting drug interactions and synergies (1/65, 2% paper). Administration (9/65, 14% papers)—LLMs streamline various tasks, including documentation (5/65, 8% papers) and information collection (5/65, 8% papers) to monitor the trend of public health.

24The concerns surrounding the application of LLMs in health care were varied, each with nuanced considerations: Reliability (55/65, 85% papers)—This includes accuracy (45/65, 69% papers), or the correctness of the responses from LLMs; consistency (13/65, 20% papers), whether LLMs produce the same response to the same questions with different prompts; interpretability (5/65, 8% papers), whether LLMs can explain their responses well, and the data quality of the training dataset (16/65, 25% papers). Bias (16/65, 25% papers)—The applications of LLMs may result in biased responses, which will exacerbate disparity and inequality in health care, particularly in terms of financial costs (1/65, 2% paper), readability (5/65, 8% papers), and accessibility (3/65, 5% papers). Privacy (6/65, 9% papers)—Training LLMs in health care settings requires a large number of health data which, however, is sensitive and may bring privacy issues.

25Public acceptance (4/65, 6% papers): Building trust in LLMs from the public is pivotal for widespread acceptance and use of LLM-based health care applications.

26All reviewed research papers demonstrated the usability or tested the capability of LLMs for health care applications in clinical or research domains, which can be further classified into the following 4 categories: summarization, medical knowledge inquiry, prediction, and administration.

27ChatGPT has been shown to be effective in summarizing medical documents for a diverse set of applications, including tasks such as adapting clinical guidelines for diagnosis, treatment, and disease management, summarizing medical notes, assisting in writing medical case reports, and generating and translating radiological reports. Notably, efforts have been made to integrate ChatGPT-4 with the Link Reader plugin for automating medical text synthesis, which boosted model performance in providing answers according to clinical guidelines. Another study by Zhou explored ChatGPT’s role in supporting health care professionals in creating medical reports from real patient laboratory results to offer treatment recommendations based on patients’ health conditions.

28ChatGPT proved beneficial for summarizing research papers as well. Notably, it demonstrated impressive performance in summarizing conference panels and recommendations, generating research questions, extracting data from literature abstracts, drafting medical papers based on given datasets, and generating references from medical articles. ChatGPT was also used to evaluate the quality and readability of web-based medical text regarding shock-wave therapy for erectile dysfunction. These applications highlighted the potential of LLMs to condense complex and extensive research materials, allowing for more accessible comprehension and use of information in health care.

29ChatGPT can be applied to answer questions about health care, as evidenced by its excellent performance in various studies. For instance, ChatGPT has shown remarkable accuracy in reasoning questions and medical exams, even successfully passing the Chinese Medical Licensing Examination and the United States Medical Licensing Examination. It also performed well in addressing radiation oncology physics examination questions. Likewise, “ChatGPT would have been at the 87th percentile of Bunting’s 2013 international cohort for the Cardiff Fertility Knowledge Scale and at the 95th percentile on the basis of Kudesia’s 2017 cohort for the Fertility and Infertility Treatment Knowledge Score”. In addition, ChatGPT showed promising results in a simulated Ophthalmic Knowledge Assessment Program exam. However, the average score of ChatGPT was 60.17% in the Membership of the Royal College of General Practitioners Applied Knowledge Test, which is <70.4%, the mean passing threshold in the last 2 years.

30Furthermore, LLMs have been shown to be effective at making medical knowledge accessible to the public. In particular, a fine-tuned chatbot based on LLaMA demonstrated enhanced performance in identifying patients’ needs and providing informed suggestions. In the realm of medical advice, ChatGPT-generated educational documents, answered questions about allergy and immunology, and countered vaccine conspiracy theories. It can also answer the most frequently asked questions about the COVID-19 pandemic. Its overall responses to queries related to cognitive decline were equivalent to and, at times, more reliable than Google’s. According to Bulck and Moons, in comparison with Google search, 40% (8 experts) of the 20 experts (19 nurses and 1 dietitian) considered answers from ChatGPT of greater value, 45% (9 experts) regarded them as equal value, and 15% (3 experts) deemed them less valuable. Therefore, many experts predicted that patients will gradually rely more on LLMs (particularly ChatGPT) and less on Google searches due to the high quality and accessibility of the answers from LLMs. Regarding cancer myths and misconceptions, 97% (63/65) of expert reviews deemed answers from ChatGPT to be accurate. In addition, Bird and Lotfi optimized a chatbot that could answer mental health–related questions with an accuracy of 88.7% (26,595/30,000 tokens). Overall, LLMs, particularly ChatGPT, demonstrate an impressive performance in public education in health.

31LLMs have been shown to have predictive capabilities in diagnosis, treatment recommendations, and drug interactions and synergies.

32ChatGPT has exhibited the potential to achieve high accuracy in diagnosing specific diseases, providing diagnostic suggestions in simulated situations or using given laboratory reports for diagnosis. ChatGPT has been evaluated in dental, allergy, and mental disorders diagnoses. Particularly, GPT-3 can be used to differentiate patients with Alzheimer disease from healthy controls using speech data. Beyond ChatGPT, other generative AI frameworks, such as DR.BENCH, were used for clinical diagnostic reasoning tasks. Moreover, various pretrained LLMs can extract microbe-disease relationships from biomedical texts in zero-shot or few-shot contexts with high accuracy, with an average F1-score, precision, and recall >0.8. In addition, ChatGPT was the best LLM when predicting high acuity cases than predicting low acuity cases according to the emergency severity index, with a sensitivity of 0.762 and a specificity of 0.931, compared with the overall sensitivity of 0.571 and a specificity of 0.345.

33For example, Hirosawa et al obtained ChatGPT’s diagnostic response by describing a clinical scenario. The prompt began with “Tell me the top 10 suspected illnesses for the following symptoms;” Then, patients’ personal information (eg, age and family history) was provided in this prompt along with other clinical data (eg, symptoms, medication, and physical examination). According to the study, the top 10 suspected diseases generated by ChatGPT achieved a rate of 93% (28/30) in overall correctness. While such a level of performance is impressive, physicians still made a better prediction than ChatGPT. With respect to the top 5 diagnoses, physicians achieved an accuracy of 98% (59/60) while ChatGPT only achieved 83% (25/30). As for the top suspected disease, ChatGPT only had a correct rate of 53% (16/30), versus 93% (56/60) achieved by physicians.

34LLMs can offer treatment recommendations while listing the side effects of these treatments. They have been involved in the treatment of various diseases, such as allergy and immunology. ChatGPT can identify guideline-based treatments for advanced solid tumors, such as breast tumor treatment. LLMs can also assist with treatment planning and brain glioma adjuvant therapy decision-making. Similarly, NYUTron, an LLM trained on unstructured clinical notes, has been applied for clinical predictive tasks in treatments. ChatGPT can effectively recommend breast tumor management strategies based on clinical information from 10 patients, enhance clinical workflow, and assist in responsible decision-making in pediatrics. In addition, ChatGPT can recommend cancer screening given the radiology reports, with an accuracy of 88% (22/25). Overall, ChatGPT performs well in certain scenarios of disease prevention and screening recommendations.

35LLMs also demonstrate high utility when characterizing drug effects. Notably, ChatGPT was used to predict and explain drug-drug interactions. In this study, the LLMs were asked about pairing or interaction between drugs, and their responses are evaluated in terms of correctness and conclusiveness. Among the 40 pairs of drug-drug interactions, 39 (98%) responses were correct for the first question, and among these 39 correct answers, 19 (49%) were conclusive while 20 (51%) were inconclusive. For the second question, 39 (97%) were correct among 40 pairs, with 17 (44%) answers conclusive and 22 (56%) answers inconclusive.

36LLMs can serve a multifaceted role in the realm of health care and administrative tasks. Specifically, ChatGPT proves instrumental in streamlining administrative processes by generating texts, thereby alleviating the associated workload. Moreover, it can be used to track patients’ health status, particularly those with chronic diseases. Through the analysis of social media slang, GPT-3 aided in developing a drug abuse lexicon that was aimed at enhancing the monitoring of drug abuse trends. Notably, an LLM-based chatbot, called CLOVA CareCall, built by Naver, was applied as a health data–collecting tool in South Korea. Designed for individuals who need emotional support and are socially isolated, CareCall conducted periodic conversations, generating health reports with metrics such as meals, sleep, and emergencies. Implemented in 20 cities by May 2022, it targeted solitary adults, notably those with lower incomes and was proven effective in reducing loneliness. Social workers used the generated reports and call recordings to monitor users’ health, resulting in positive feedback and a streamlined workload for public health workers.

37Most of the reviewed research papers pointed out technical and ethical concerns that people harbor with respect to the application of LLMs in health care from several perspectives. This can generally be categorized into four groups: (1) reliability, (2) bias, (3) privacy, and (4) public acceptance.

38The reliability of LLMs is essential to their application in health care. It can be related to the accuracy, consistency, and interpretability of LLM responses and the quality of the training dataset. Specifically, in 100% (22/22) of prediction-related studies, 72% (18/25) of summarization-related studies, and 93% (28/30) of studies related to medical knowledge inquiries, the authors pointed out their concerns toward LLM reliability (Table 1).

39Several studies highlighted that ChatGPT exhibited inaccuracies when asked to respond to certain questions. For instance, ChatGPT could respond with incomplete information or exhibit an inability to distinguish between truth and falsehood. The generative nature of the LLM algorithms will likely fabricate a fake reference to substantiate false claims, a process that has been referred to as “hallucinations”. In addition, such hallucinations can be communicated via persuasive prose, making it more likely to mislead patients. For example, Jo et al mentioned that LLMs (specifically CLOVA CareCall built by NAVER in this paper) may make ambitious or impractical promises to patients, which may add extra burden to therapists or cause a trust crisis.

40The unreliability of LLMs may be attributed to limitations in data collection sources. There are concerns about the model’s limitation in medical knowledge because the general-purpose nature of ChatGPT may affect its reliability in self-diagnosis. Recent state-of-the-art LLMs are typically constructed on texts from the internet rather than verified resources about health and medicine.

41Of greater concern is the data availability. Health care institutions have shared no identifiable health information with widely accessible LLMs such as ChatGPT due to privacy concerns and legal compliances, and it is arduous to collect new data for LLM training. ChatGPT, for example, was not trained on patients’ clinical data. While a description of a clinical scenario without sensitive patient information can be fed into ChatGPT through prompts, it may lead to inaccurate responses.

42Another contributing factor to inaccuracy is the outdated knowledge base used to train LLMs. ChatGPT based on GPT3.5 was pretrained using data collected until 2021 and does not support internet connection, making it unable to perform appropriately on questions regarding events that happened after 2021.

43Many authors expressed concerns about the inconsistency of the responses from LLMs, where different answers result from various prompts of the same question. In addition, the output of ChatGPT to the same prompt may vary from user to user. This is because LLMs generate responses in a probabilistic manner. Therefore, nuance in the prompts to the LLM may lead to a completely different answer.

44Interpretability is another aspect regarding the reliability of the response. A study by Cadamuro et al highlights 2 key issues with an LLM (particularly ChatGPT) in health care. First, the interpretation of some normal results regarding suspected underlying diseases was not completely correct. Second, ChatGPT struggled to interpret all the coherent laboratory tests, generating superficial and incorrect responses. Indeed, ChatGPT could generate overly general answers without citing the original reference.

45It has been noted that ChatGPT has issues with disparity and bias among different populations. In other words, because certain groups of people have financial, readability, and accessibility barriers using LLMs, their outcomes of using LLMs will be divergent from others. For example, ChatGPT may exert some financial disparity on the users: unlike previous versions such as GPT-3.5, access to GPT-4 involves a monthly fee. These constraints potentially pose financial barriers, limiting widespread adoption and use of the newer, more advanced models in health care applications.

46Moreover, the readability of an LLM’s response may further accentuate health disparity. LLMs such as ChatGPT include texts from scientific websites (eg, Wikipedia) as their training data, which makes their responses sound professional and sophisticated. However, LLMs may produce biased results, making regulations to prevent bias necessary.

47Furthermore, the training data can also be biased. Since recent LLMs are trained based on human-generated texts from the internet, they also tend to provide biased answers. Besides, algorithms may reinforce current health disparities and inequities. Indeed, outputs from ChatGPT have been shown to be biased in terms of gender, race, and religion.

48Privacy issues are important when training or using LLMs in health care settings. All AI systems, including LLMs in health settings, should comply with privacy regulations, including compliance with the Health Insurance Portability and Accountability Act, and implement robust safeguards to ensure the protection of sensitive patient information. Specifically, LLMs have 3 privacy problems. First, the responses from LLMs may embed training examples directly, which breaches privacy if the training examples are identifiable. Second, LLMs may be susceptible to inferential disclosure. For example, a patient’s membership in a dataset or sensitive attributes may be inferred from LLMs’ responses. Third, it may not be clear whether text data are sufficiently deidentified for the anticipated recipients (which may be anyone in the world) when training LLMs. For instance, we may be able to deidentify text in a manner that sufficiently thwarts people who are not incentivized to attack the system, but we may not be addressing recipients who run machine-assisted attacks.

49Public acceptance, the trust of the public in the application of LLMs in health care, has been mentioned in a study by Shahsavar and Choudhary. A cross-sectional, survey-based study shows that 77.9% (371/476) participants claim that they trust ChatGPT’s diagnosis, most of whom possess a bachelor’s or even master’s degree. People are inclined to trust this new technique when using ChatGPT, partially due to the convenience of obtaining information and the patients’ inclination to search for information.

50This systematic review shows that LLMs have been applied to summarization, medical knowledge inquiry, prediction, and administration. At the same time, there are 4 major themes of concern when using these models in practice, including reliability, bias, privacy, and public acceptance. Specifically, the most popular application (30/65, 46% papers) for LLMs was for medical knowledge inquiries, with the second most popular (25/65, 38%) being summarization, followed by prediction (22/65, 34%), and then administration (9/65, 14%). At the same time, 55 (85%) papers expressed concerns about reliability, 16 (25%) about bias, 6 (9%) about privacy, and 4 (6%) about public acceptance.

51According to our systematic review, LLMs were heavily applied in summarization and medical knowledge inquiry tasks. The former is probably due to the training method of LLMs, which focuses on their capability to summarize documents and paraphrase paragraphs. The latter is due to the inclusion of general medical knowledge in the training data. Specifically, in the category of summarization, summarizing medical notes is the type of task in which LLMs were applied the most. This is probably due to the simplicity of the task and the existence of redundancy in those notes. By contrast, in the genre of medical knowledge inquiry, taking standard medical exams is the type of task in which LLMs were applied the most. This is probably due to the existence of medical questions and answers on the internet that have been included in the training data of some LLMs, such as ChatGPT.

52LLMs were applied in prediction tasks as well. Specifically, in the category of prediction, diagnosis is the type of task in which LLMs were applied but with the most reliability concerns. This is probably because diagnosis is a complex process in comparison with summarization and the current popular LLMs (eg, ChatGPT) used insufficient publicly available health datasets for model training. It might also be due to poorly constructed prompts without enough accurate information. Thus, LLMs are still not likely to be suitable for generating reliable answers to uncommon questions. In the category of administration, LLMs were applied equally heavily in various tasks, such as appointment scheduling, information collection, and documentation.

53For those applications of LLMs in health care, the 2 greatest concerns are reliability and bias (including disparity and inequality). These concerns might eventually drive this application away from practical implementation.

54Notably, about 85% (55/65) of the reviewed studies emphasized concerns about the reliability of LLMs’ responses given that they may impact a patient’s health-related behavior. The concerns about reliability arose mainly from 2 aspects: the quality of the training data in terms of data source and data timeliness, and the models themselves in terms of their performance (eg, accuracy). For example, GPT-3.5 was pretrained using data collected by September 2021, and it also does not have access to private health records. Furthermore, most data that are used to train LLMs are crawled from the internet rather than professionally validated sources. In addition, the generative nature of LLM may result in seeming professional writing but fabricating responses. However, according to Shahsavar and Choudhury, people are inclined to trust this new technique, due partially to the convenience of obtaining information and the patients’ inclination to search for information. By contrast, LLMs exhibit mixed predictive performance across different applications. In critical scenarios where incorrect predictions could lead to fatalities, even a 15% difference in accuracy from the gold standard (eg, 59/60, 98% vs 25/30, 83%) could significantly hinder their use in real-world applications.

55The issue of bias (or disparity) is mentioned in about 25% (16/65) of our included references. LLM biases come from the training stage (eg, biased training data and biased algorithms) and the application stage (eg, biased user base and biased outcomes). These papers discussed biases mainly from 3 different aspects: financial costs, readability, and accessibility. For example, Hirosawa et al pointed out that the bias encoded in human-generated texts will make LLMs generate biased output; Lee et al concerned that health disparity may result from low readability made by the sophistication of LLM wording; and Johnson et al noted that LLM algorithms tend to reinforce the health disparity and to prevent LLM algorithms from exacerbating current disparity in health.

56Another concern that prevents the wide application of LLMs in health care is privacy. When using third-party LLMs, such as ChatGPT, health care organizations face several privacy issues. Although no privacy breach of LLMs regarding patient information has been reported, attacks for other types of private information targeting ChatGPT have been found. For example, a breach led to the exposure of users’ conversations to unauthorized parties. As ChatGPT interacts with patients directly, it may gather personal health information and may breach their privacy. Therefore, many medical centers do not allow researchers and health care providers to use raw patient data as inputs to ChatGPT and other LLMs or even ban their access to these services during work. Training or fine-tuning open-source LLMs requires a large amount of clinical data, which may lead to violations of patients’ privacy, perhaps inadvertently.

57The reviewed papers demonstrated 2 common limitations of their approaches. First, almost all the studies relied on human experts to rate LLMs’ responses. This is problematic because the score may be subjective and more likely unrepresentative. Correspondingly, future works can focus on designing a formal and fair process to evaluate LLMs’ responses from a broad range of stakeholders, including researchers, health care providers, patients, or any users with diverse medical and sociodemographic backgrounds. Second, some of the concerns mentioned in this review (eg, bias) are merely researchers’ speculations of the potential risks that were included to provide directions for further work. However, the mechanisms of how the training of LLMs leads to such concerns have not been comprehensively examined through experiments. It is suggested the audience should be wary of taking these concerns for granted or as proven facts.

58Among all the included papers, few of them propose solutions to improve the reliability of LLMs. First, future research work should focus more on how to improve the accuracy of LLMs’ responses in the health care domain. More specifically, domain-specific health data are demanded for training and fine-tuning of LLMs to improve the performance of LLMs in various tasks in the health care domain. Therefore, data harmonization and consortia established for LLM training are potential directions that can benefit the broad research community. Qualified medical professionals can contribute to the creation of the dataset for LLM training. This, however, will be expensive in terms of time and effort. Alternatively, using retrieval-augmented generation to augment LLM with external knowledge that is up-to-date might be a solution for scenarios where accurate, in-depth professorial knowledge is required. Second, to prevent the hallucination issue, LLMs should be limited to making responses based on validated references. Blockchain technology can be used in this process to provide validation and traceability. Moreover, a holistic system, or a keep-experts-in-the-loop framework that efficiently facilitates the expert validation process becomes important to improve the accuracy and safety of health LLMs. Third, clinical trials based on health outcomes, such as mortality rates, should be conducted to validate the utility of LLM applications formally.

59How conversational LLMs lead to bias or privacy issues in health care research was not thoughtfully examined with experiments in our reviewed papers. Future studies should first focus on investigating the mechanisms of how LLMs caused bias and privacy issues with stringent experiments and then developing practical solutions.

60Regarding bias issues, it is suggested that systematic monitoring is necessary to ensure the impartial functioning of LLMs. However, all these sources discuss bias only with mere sentences and superficial summaries without any experimental investigation. Hence, it is worth noting that further work should also focus more on conducting experiments to understand how bias impacts the responses of LLMs in information, diagnosis, recommendation, and surveillance. More specifically, all applications of LLMs in health care should be tested regarding the exhibitions of bias and the bias mitigation strategies, such as data augmentation and targeted recruitment (eg, the All of Us Research Program targets the collection of data from historically underrepresented populations).

61Regarding privacy issues, 2 technical approaches to mitigate the privacy risk while training LLMs are data anonymization and synthetic data generation. For deep learning models, model inversion attacks can potentially infer training data by giving model weights. Considering the exponentially increased open-sourced LLMs with published model weights, a sensitive patient dataset needs to be deidentified or replaced with a synthetic dataset before being used to train or fine-tune an LLM. Otherwise, the patients with whom the data are associated should be informed about their participation in the training or fine-tuning process. To solve the privacy issues, legal, social, and technical protection approaches need to be implemented together to ensure the privacy and security of the whole process of training and using LLMs for health care applications.

62To raise the public acceptance level of LLMs, explainable AI should be used to address the interpretability issues of LLMs by making the training data and model architecture transparent. More rigorous experimental studies using LLMs are encouraged in the “AI in medicine” research community to demonstrate or improve the reliability of LLM applications in health care. Moreover, stakeholders and decision-makers can propose new policies or regulations to manage the accountability and transparency of AI-generated content, including the responses from LLMs.

63There appears to be research that is beginning to address some of these raised issues. For example, Zack et al assessed the potential of GPT-4 to perpetuate racial and gender biases in health care. Hanna et al assessed the racial bias of ChatGPT in health care–related text generation tasks. However, more research studies in these directions are needed to validate these findings and conduct more comprehensive and transparent assessments.

64Furthermore, it is important to consider the interconnections among these categories of concerns. For instance, privacy protection methods may negatively affect the quality of training data and, consequently, the model’s reliability due to the tradeoff between data utility and privacy. In addition, tackling privacy issues can influence model fairness in different ways, depending on the approach used. Therefore, we recommend addressing these concerns holistically rather than in isolation.

65Moreover, almost all the research studies LLMs’ responses in 1 language. For example, 95% (62/65) of studies focus on English, 1 (2%) focuses on Korean, 1 (2%) focuses on Chinese, and 1 (2%) focuses on Japanese. Their findings cannot be extrapolated to other languages directly. Considering that many patients or people around the world or even in the United States do not speak English, it is necessary to guarantee that LLMs are usable universally or equitably and conduct more research to investigate the performance of LLMs in other languages.

66Despite notable findings, this review has several limitations. Firstly, the review used PubMed, ACM Digital Library, and IEEE Xplore as the primary sources for the papers. Other sources, such as Scopus, Web of Science, ScienceDirect, or non-English sources, may provide additional candidate papers regarding LLMs for health. However, because PubMed is the main digital library for medical publications, the research findings of this review should be valuable to health care researchers or policy makers. Second, although this review intended to study the application of state-of-the-art conversational LLMs in health care, most of the papers included are about ChatGPT. This is because ChatGPT is still the most powerful conversational LLM. However, its closed-source nature, which is against its company name—OpenAI—may be a hurdle to its wide application in health care, due primarily to the privacy concern when sharing sensitive patient information within prompts with OpenAI. Third, our search terms did not include any medicine-related keywords, which may have limited the number of papers included in this review. Finally, only peer-reviewed papers published before September 2023 are included in our review. Therefore, on one hand, the latest LLM application developments in this area are not included in this review. Specifically, papers focused on LLMs other than ChatGPT, such as LLaMA, were very limited in our initial keyword search results, and only a few of them are included in this review. This is a problem because, while monomodal conversational LLMs have been applied to many fields in health care, the multimodal LLMs that can process medical images, such as GPT-4, Large Language and Vision Assistant (LLaVA) based on LLaMA, and LLaVA-Med based on LLaVA, were just released before September 2023 and are still being examined by researchers regarding their capabilities in health care research. Therefore, no peer-reviewed research papers about applications of multimodal LLMs in health care have been published before September 2023. The main challenge of the application of multimodal LLMs in health care is that multimodal LLMs are still not perfect, either due to insufficient training data or due to insufficient model parameters. Specifically, with the development of computing power, reduced computing cost, and reduced data access cost, LLMs can be applied to multimedia-based diagnosis and analysis in radiology and other departments. By contrast, the latest studies addressing the concerns are not included in this review. Although there is research that is beginning to address some of the issues raised in the systematic review, there may not have been sufficient time for all recent papers to be deposited into the repositories upon which this investigation relied yet.

67This review summarized applications of the state-of-the-art conversational LLMs in health care and the concerns that need to be resolved in the future. According to the reviewed research articles, conversational LLMs perform well in summarizing health-related texts, answering general questions in health care, and collecting information from patients. However, their performance is relatively less satisfying in making diagnoses and offering recommendations based on patients’ symptoms and other information. Most authors were concerned about the accuracy and consistency of the LLM responses, which should be the primary issues that researchers need to address in the near future. Nevertheless, other concerns regarding bias and privacy issues also prevent conversational LLMs from being broadly applied in the health care domain. However, these concerns still receive insufficient attention: few studies examine the bias and privacy issues in LLMs’ health-related applications with rigorous scientific experiments. Future research should focus more on conducting such research to investigate the mechanisms of how the training and application of conversational LLMs leads to such concerns and to address these concerns that have been seen on any AI tools so that they can be safely applied in the health care domain.