Large Language Models for Chatbot Health Advice Studies

1Large language models (LLMs) have expanded the potential for integrating artificial intelligence (AI) in medicine. LLMs are AI systems that are pretrained through a variety of word prediction tasks across enormous volumes of text taken from datasets, books, articles, and internet sources., With additional fine-tuning involving varying amounts of human feedback, LLMs acquire natural language processing (NLP) capabilities and can generate appropriate text outputs using layperson terminology in response to free-text prompts. Many patients and physicians use the internet to access health information, and generative AI-driven chatbots present a convenient new way to access answers to medical questions. Early data suggest that generative AI-driven chatbots can even respond to patients with expressions of empathy.

2Patients have reported interest in seeking medical advice from chatbots, leading to a rapid expansion in literature assessing the ability of chatbots to offer sound health advice to physicians and patients: chatbot health advice studies (CHAS). This research has investigated the ability of chatbots to summarize evidence and provide health advice about screening, diagnosis, treatment, and disease prevention. However, chatbots are typically not designed for medical application and currently lack regulation for their use in medicine. Furthermore, generative AI-driven chatbots present unique risks of hallucinating (ie, confident but incorrect) false results and propagating misleading information and advice. Thus, it is unsurprising that CHAS report varying levels of chatbot performance, raising concerns about their limitations and risks.,,

3CHAS also present their own unique problems. Currently, these studies are inconsistent in their design, analysis, and reporting.,, With a lack of standardized reporting, the assessment of the quality of CHAS and the interpretation of their findings remain challenging. These studies have the potential to provide useful assessment of LLM performance and insight into the patient safety implications of chatbot advice, yet the apparent limitations of the studies themselves necessitate clarifying and standardizing current approaches. In this systematic review, we investigated chatbot selection, query approach, and performance evaluation. The information gathered from this systematic review may contribute to the development of a novel chatbot assessment reporting tool (CHART) to standardize reporting methods in chatbot assessment research.

4This systematic review adheres to the reporting standards of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting guideline. In this study, we refer to LLMs, generative AI-driven chatbots, and chatbots synonymously. Our protocol was prospectively registered on Open Sciences Framework.

5We set out to characterize the approach of studies that assess the performance of generative AI-driven chatbots for summarizing evidence and providing health advice. To capture all relevant studies, an academic health sciences librarian (S.S.) with expertise in systematic reviews aided the study team in designing a comprehensive search strategy. The detailed literature search syntax can be found in the eAppendix in Supplement 1. Records were retrieved from MEDLINE via Ovid, Embase via Elsevier, and Web of Science on October 27, 2023. The study team completed 2 rounds of screening, first by title and abstract and then by full text. In all cases, 2 independent researchers (W.T. and J.P.S.) conducted screening, with a third independent researcher casting a deciding vote to resolve cases of disagreement. We excluded irrelevant studies and nonprimary studies. The Box illustrates our eligibility criteria.

6Studies assessing the performance of chatbots for summarizing clinical evidence and providing health advice.

7Systematic reviews and literature/narrative reviews.

8Editorials, commentaries, opinions, studies published as correspondence articles, and studies of any type outlining earlier or potential future applications of chatbots.

9Cross-sectional studies assessing opinions of future work for chatbot research.

10Abstracts.

11Data extraction was conducted with Covidence and aimed to capture basic study details (clinical specialty, topic, intervention, and comparator) in addition to the following: (1) how LLMs are selected for use in CHAS, (2) clinical purposes for which LLMs have been evaluated, (3) methods applied, (4) approach to performance evaluation (ie, summarizing evidence alone or summarizing evidence to inform recommendation), (5) reporting practices used, and (6) the presence of ethical considerations of the clinical application of generative AI-driven chatbots.

12The study team performed data analysis using descriptive statistics. We summarized findings pertaining to the methodologic approach of CHAS using frequencies as counts and percentages including the types of LLMs evaluated, the query strategy, and their performance evaluation. A narrative synthesis with no meta-analysis was planned, due to anticipated heterogeneity in study subjects and designs. We describe the full details related to the literature search and data extraction variables in our study protocol. The quality of the evidence was not evaluated due to the lack of high-quality, validated risk-of-bias tools available for CHAS.

13The study team reviewed 7752 articles that yielded 137 eligible CHAS (Figure). Table 1 lists all evaluated articles,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,, including their relevant specialty and study aims.

14Overall, of the 137 articles reviewed, 51 (37.2%) studies queried LLMs for advice relevant for medical topics, 55 (40.1%) on surgical topics, 13 (9.5%) on primary care topics, 9 (6.6%) on radiology topics, and 7 (5.1%) on psychiatry topics (Table 1). Articles most commonly examined advice regarding treatment (91 [66.4%]), diagnosis (60 [43.8%]), and differential diagnosis (23 [16.8%]). Overall, 16 studies (11.7%) explained why the specific LLMs were selected to be studied. Almost all studies (135 [99.3%]) assessed the ability of the OpenAI Chat Generative Pretrained Transformer (ChatGPT) to provide health advice, and 136 of the LLMs studied (99.3%) were inaccessible (proprietary/closed-source) models. A single study (0.7%) fully described the version of the LLM under evaluation, and no studies described the characteristics of the LLM in comprehensive detail including temperature (a parameter to control the randomness of LLM output), token length, fine-tuning availability, penalties, add-on availability, language, and layers related to the LLM being studied. More than one-quarter (36 [26.3%]) of the studies described none of the characteristics of the LLM being examined (Table 2). Sources used to develop prompts included expert opinion (43 [31.4%]), professional society websites (28 [20.4%]), and clinical practice guidelines (24 [17.5%]).

15Table 2 presents details of our study performance evaluation. More than one-quarter (37 [27.0%]) of the studies did not explain the source of their prompts or describe at least one aspect of their query strategy (40 [29.2%]). Few studies noted the date (54 [39.4%]) and location (5 [3.6%]) of their query. Study performance was suboptimal in noting the number of chat windows (32 [23.4%]) and the number of queries (48 [35.0%]). Studies performed poorly (20 [14.6%]) in using standardized prompts to evaluate their LLMs. Overall, 136 articles (99.3%) did not describe a prompt testing/engineering phase, while 93 (67.9%) shared the full transcript of their prompts and 65 (47.4%) included the full transcript of chatbot responses.

16With respect to performance evaluation of chatbots, few studies described a standardized evaluation process (18 [13.1%]) or blinding (16 [11.7%]), and many studies did not describe a structured approach to performance evaluation (39 [28.5%]). Rather than using clinical practice guideline recommendations as in 21 studies (15.3%), most investigators evaluated LLMs based on their opinion without a reference standard (89 [65.0%]). Less than one-third of the studies addressed the ethical (45 [32.8%]) and patient safety (44 [32.1%]) implications of the clinical integration of LLMs, while 22 (16.1%) addressed the lack of regulation of LLMs (Table 2).

17This systematic review identified large heterogeneity in the methods of CHAS and a need for improved reporting standards. Many studies failed to adequately report the methods of chatbot performance evaluation. Moreover, most studies that described an approach to evaluation relied on anecdotal opinion rather than systematic summaries of available evidence or official guidelines. No study described the characteristics of the LLM under evaluation in sufficient detail to reproduce experiments, and many failed to describe any aspect of the LLM characteristics. Most reports did not include reasons for chatbot selection, and they seldom described a prompt engineering phase. Authors infrequently shared prompt and chatbot response transcripts. Less than half mentioned ethical issues, patient safety, or regulation of LLMs.

18Generative AI-driven chatbots constitute the intervention in these studies, yet there is a lack of critical information about their characteristics. As these publications are largely prepared by clinicians, study teams may not have access to expertise in machine learning. Without full knowledge of the intervention being applied, it is difficult to interpret LLM performance.,,

19LLMs are trained to learn associations between words in large training text datasets, transform algorithm inputs to outputs, and mimic human language using NLP. This process enables LLMs to predict the next word in a sequence, and preset model characteristics, such as temperature, token length, fine-tuning availability, penalties, add-on availability, language, and layers, further impact their functionality.,, Just as sufficient detail about medical or surgical interventions must be reported in traditional studies to reproduce experiments and guide critical appraisal, LLM characteristics are similarly vital. Physicians may not be AI experts, but they should understand the principles and limitations of their LLM interventions and may benefit from involving expert LLM researchers.

20Most included studies did not report a prompt engineering phase. However, the generated output depends greatly on the initial input. LLMs have in-context learning capabilities that allow them to adapt to the given prompt, so prompts can be intentionally designed to support performance on further tasks, such as patient communication, administrative tasks, risk assessments, or clinical decision-making., Prompt engineering improves chatbot performance by tailoring LLM output to be focused and helpful for a specific task., These crucial concepts underscore the importance of including experts in machine learning, computer science, and engineering. In the push for the clinical integration of LLMs in medicine, multidisciplinary stakeholder collaboration between physicians and expert LLM and NLP researchers is essential.

21Additionally, the existing literature base emphasizes a structured approach to performance evaluation. However, most studies did not provide transcripts of chatbot prompts and responses. Inclusion of obtained responses and the details of subsequent analysis is important to maximize transparency, internal validity, and the external appraisal of chatbot assessment study findings. Authors frequently apply subjective measures of chatbot performance, such as expert opinion, in CHAS, limiting the generalizability of their methods and conclusions. A shift toward objective measures, preferably basing accuracy and relevance with respect to high-quality evidence, such as clinical practice guidelines, is necessary. Many studies describe this as defining a ground truth or evaluation dataset, and it is an essential step in LLM performance evaluation., Currently, few studies use clinical practice guidelines to define successful performance, and several chatbots have experienced inconsistent accuracy by these standards., This is especially important when considering the implications of using these LLMs on patient safety.

22Artificial intelligence–linked medical devices are considered software as a medical device and are regulated under federal departments such as Health Canada and the US Food and Drug Administration. However, LLM applications can be excluded from this regulation if they were not designed for medical purposes or intended to replace clinical judgment.,, Although no devices that use generative AI or LLMs have been approved by the Food and Drug Administration, LLMs are being used in medical applications through this gap in regulation. Some LLMs have already been integrated into electronic medical record systems, such as the Nuance DAX Copilot, a generative AI-program embedded in an electronic health record software called Epic, which records patient interactions and automatically produces clinical documentation., To safely implement generative AI-driven chatbots in practice, there is a need for updated regulation that addresses the differences between emerging generative AI and previous forms of AI. While other regulated AI medical technologies, such as radiology image analysis or diagnosis, are trained on domain-specific medical data for targeted uses, LLMs have broader complexity, applicability, and an ability to adapt in real time, which complicates their regulation.,,

23Additionally, the specific issues of algorithmic bias, hallucinations, and risks of privacy breaches must be addressed by regulatory standards,, as these concepts have ethical implications in patient care. Several examples of bias have been reported among LLMs. These have included the propagation of biases based on race and ethnicity, gender, educational level, social class, sexuality, and age.,, Many of these biases stem from the data used to train LLMs. Additionally, LLMs have been known to produce hallucinations., Large language model performance is largely influenced by the content used for pretraining and fine-tuning, yet LLMs are generally not trained exclusively with high-quality medical literature or real-world clinical data. While clinical practice is informed by evidence from peer-reviewed research, LLMs replicate their input data and thus may propagate online health misinformation and prejudices,, compromising patient safety. Moreover, the lack of transparency surrounding the development and external validation of closed-source or proprietary LLMs presents additional challenges to their evaluation for health care purposes., Concerns have also been raised related to data privacy of patient health information among proprietary LLMs., There is a potential for misuse of patient data, medicolegal implications, and loss of physician-patient trust.,

24Taken together, these factors raise major issues as we move from the technological development of LLMs to their clinical validation for use in health care settings. The World Health Organization released guidance on ethics and governance of AI for health, which overlaps substantially with LLMs, which are unimodal models. This document outlines key messages for stakeholders involved in the health care integration of generative AI. For instance, the World Health Organization calls for developers to commit to ethical principles of inclusiveness, use higher-quality data during model development, adhere to data protection laws, be transparent about model training data, and involve key stakeholders in the design and development of LLMs, including health care professionals, patients, laypersons, and vulnerable persons. Moreover, developers may focus on the explainability of their LLMs, a concept that aims to elucidate how models arrive at outputs given a specific input. In addition, governments should develop clear regulatory policies and legislation to capture many of the concepts discussed herein, such as target product profiles, design and development standards, audits, disclosure requirements, data protection, and public infrastructure for the development of LLMs. Regulators may further look toward incentivizing researchers toward the development of standardized metrics for the clinical validation of health care applications of generative AI models such as LLMs.

25This systematic review has limitations. First, the reporting used in CHAS may change beyond the time of this review due to the dynamic nature of this field. By engineering a comprehensive search strategy, we aimed to capture an accurate representation of the literature. Second, much of the literature evaluated in this review used hypothetical patient cases. CHAS should focus on patient-centered, prospective research using LLMs in the clinical setting. Multidisciplinary collaboration between clinicians and LLM/NLP experts may help facilitate this, while also improving the understanding and reporting of LLM characteristics in these studies. In addition, there is a paucity of literature that comments in detail on the ethical and regulatory considerations of the clinical integration of LLMs that impact patient safety. Clinicians, patients, and regulators should be involved in collective efforts to agree and enforce appropriate deployment strategies for LLM technology in medicine.

26Standardized frameworks are under development to establish detailed reporting standards to help authors design and report these studies. The CHART reporting tool may guide transparent reporting of CHAS and promote the publication of CHAS with higher methodologic rigor. Future research should address the lack of quality appraisal tools available to evaluate CHAS. Moreover, standardized metrics to validate the health care application of generative AI models are needed to facilitate their evaluation for adoption in clinical pathways. Interdisciplinary work among hospital managers, policymakers, regulatory bodies, and physicians of varying specialty backgrounds should be conducted to respond to the rapid advancement in LLM technology and the push for clinical integration by producing regulatory frameworks to support the safe clinical integration of LLMs.

27In this systematic review of 137 CHAS, we noted that detailed and transparent reporting of key aspects often was not included in the studies, such as LLM characteristics, prompt engineering, query strategy, and performance evaluation. An emphasis should be placed on high-quality methods to justify the deployment of novel applications of LLMs and related technologies in clinical practice.