Current applications and challenges in large language models for patient care: a systematic review

1Public and academic interest in large language models (LLMs) and their potential applications has increased substantially, especially since the release of OpenAI’s ChatGPT (Chat Generative Pre-trained Transformers) in November 2022–. One of the main reasons for their popularity is the remarkable ability to mimic human writing, a result of extensive training on massive amounts of text and reinforcement learning from human feedback.

2Since most LLMs are designed as general-purpose chatbots, recent research has focused on developing specialized models for the medical domain, such as Meditron or BioMistral, by enriching the training data of LLMs with medical knowledge,. However, this approach to fine-tuning LLMs requires significant computational resources that are not available to everyone and is also not applicable to closed-source LLMs, which are often the most powerful. Therefore, another approach to improve LLMs for biomedicine is to use techniques such as Retrieval-Augmented Generation (RAG). RAG allows information to be dynamically retrieved from medical databases during the model generation process, enriching the output with medical knowledge without the need to train the model.

3LLMs hold great promise for improving the efficiency and accuracy of healthcare delivery, e.g., by extracting clinical information from electronic health records, summarizing, structuring, or explaining medical texts, streamlining administrative tasks in clinical practice, and enhancing medical research, quality control, and education–. In addition, LLMs have been shown to be versatile tools for supporting diagnosis or serving as prognostic models,.

4However, despite the growing body of research and the clear potential of LLMs in healthcare, there is a gap in terms of systematized information towards their use in patient care (i.e., the use of LLMs by patients or their caregivers for disease management and support). In contrast to applications primarily aimed at healthcare professionals, LLMs in patient care could be used for education and empowerment by providing answers to medical questions and translating complex medical information into more accessible language,. Thereby, LLMs may promote personalized medicine and broaden access to medical knowledge, empowering patients to actively participate in their healthcare decisions.

5To the best of our knowledge, there has been no evaluation of existing research to understand the scope of applications and identify limitations that may currently limit the successful integration of LLMs into clinical practice. This systematic review aims to analyze and synthesize the literature on LLMs in patient care, providing a systematic overview of 1) current applications and 2) challenges and limitations, with the purpose of establishing a foundational framework and taxonomy for the implementation and evaluation of LLMs in healthcare settings.

6This systematic review was pre-registered in the International Prospective Register of Systematic Reviews (PROSPERO) under the identifier CRD42024504542 before the start of the initial screening and was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyzes (PRISMA) guidelines (see checklist in the Supplementary Dataset file 1),.

7We searched 5 databases, including the Web of Science, PubMed, Embase/Embase Classic, American for Computing Machinery (ACM) Digital Library, and Institute of Electrical and Electronics Engineers (IEEE) Xplore as of January 25, 2024, to identify qualitative, quantitative, and mixed methods studies published between January 1, 2022, and December 31, 2023, that examined the use of LLMs for patient care. LLMs for patient care were defined as any artificial neural network that follows a transformer architecture and can be used to generate and translate text and other content or perform other natural language processing tasks for the purpose of disease management and support (i.e., prevention, preclinical management, diagnosis, treatment, or prognosis) that could be directly directed to or used by patients. Articles had to be available in English and contain sufficient data for thematic synthesis (e.g., conference abstracts that did not provide sufficient information on study results were excluded). Given the recent surge in publications on LLMs such as ChatGPT, we allowed for the inclusion of preprints if no corresponding peer-reviewed article was available. Duplicate reports of the same study, non-human studies, and articles limited to technology development/performance evaluation, pharmacy, human genetics, epidemiology, psychology, psychosocial support, or behavioral assessment were excluded.

8Initially, we conducted a preliminary search on PubMed and Google Scholar to define relevant search terms. The final search strategy included terms for LLMs, generative AI, and their applications in medicine, health services, clinical practices, medical treatments, and patient care (as detailed by database in the Supplementary Methods). After importing the bibliographic data into Rayyan and removing duplicates, LH and CR conducted an independent blind review of each article’s title and abstract. Any article flagged as potentially eligible by either reviewer proceeded to the full-text evaluation stage. For this stage, LH and CR used a custom data extraction form created in Google Forms (available online) to collect all relevant data independently from the studies that met the inclusion criteria. Quality assessment was also performed independently for each article within this data extraction form, using the Mixed Methods Appraisal Tool (MMAT) 2018. Disagreements at any stage of the review were resolved through discussion with the author FB. In cases of studies with incomplete data, we have tried to contact the corresponding authors for clarification or additional information.

9Due to the diversity of investigated outcomes and study designs we sought to include, a meta-analysis was not practical. Instead, a data-driven convergent synthesis approach was selected for thematic syntheses of LLM applications and limitations in patient care. Following Thomas and Harden, FB coded each study’s numerical and textual data in Dedoose using free line-by-line coding,. Initial codes were then systematically categorized into descriptive and subsequently into analytic themes, incorporating new codes for emerging concepts within a hierarchical tree structure. Upon completion of the codebook, FB and LH reviewed each study to ensure consistent application of codes. Discrepancies were resolved through discussion with the author KKB, and the final codebook and analytical themes were discussed and refined in consultation with all contributing authors.

10Of the 4349 reports identified, 2991 underwent initial screening, and 126 were deemed suitable for potential inclusion and underwent full-text screening. Two articles could not be retrieved because the authors or the corresponding title and abstract could not be identified online. Following full-text screening, 35 articles were excluded, and 89 articles were included in the final review. Most studies were excluded because they targeted the wrong discipline (n = 10/35, 28.6%) or population (n = 7/35, 20%) or were not original research (n = 8/35, 22.9%) (see Supplementary Dataset file 2). For example, we evaluated a study that focused on classifying physician notes to identify patients without active bleeding who were appropriate candidates for thromboembolism prophylaxis. Although the classification tasks may lead to patient treatment, the primary outcome was informing clinicians rather than directly forwarding this information to patients. We also reviewed a study assessing the accuracy and completeness of several LLMs when answering Methotrexate-related questions. This study was excluded because it focused solely on the pharmacological treatment of rheumatic disease. For a detailed breakdown of the inclusion and exclusion process at each stage, please refer to the PRISMA flowchart in Fig. 1.

11Supplementary Dataset file 3 summarizes the characteristics of the analyzed studies, including their setting, results, and conclusions. One study (n = 1/89, 1.1%) was published in 2022, 84 (n = 84/89, 94.4%) in 2023,–, and 4 (n = 4/89, 4.5%) in 2024– (all of which were peer-reviewed publications of preprints published in 2023). Most studies were quantitative non-randomized (n = 84/89, 94.4%),–,–,,,,,–, 4 (n = 4/89, 4.5%),,, had a qualitative study design, and one (n = 1/89, 1.1%) was quantitative randomized according to the MMAT 2018 criteria. However, the LLM outputs were often first analyzed quantitatively but followed by a qualitative analysis of certain responses. Therefore, if the primary outcome was quantitative, we considered the study design to be quantitative rather than mixed methods, resulting in the inclusion of zero mixed methods studies. The quality of the included studies was mixed (see Supplementary Dataset file 4). The authors were primarily affiliated with institutions in the United States (n = 47 of 122 different countries identified per publication, 38.5%), followed by Germany (n = 11/122, 9%), Turkey (n = 7/122, 5.7%), the United Kingdom (n = 6/122, 4.9%), China/Australia/Italy (n = 5/122, 4.1%, respectively), and 24 (n = 36/122, 29.5%) other countries. Most studies examined one or more applications based on the GPT-3.5 architecture (n = 66 of 124 different LLMs examined per study, 53.2%),–,–,–,–,–,–,,–,,,,,,,–,,,,,–,–,–,, followed by GPT-4 (n = 33/124, 26.6%),,,,,–,,,,,,,,,,–,,,–,,,,,,,,,,, Bard (n = 10/124, 8.1%; now known as Gemini),,,,,,,,,, Bing Chat (n = 7/124, 5.7%; now Microsoft Copilot),,,,,,, and other applications based on Bidirectional Encoder Representations from Transformers (BERT; n = 4/124, 3.2%),,, Large Language Model Meta-AI (LLaMA; n = 3/124, 2.4%), or Claude by Anthropic (n = 1/124, 0.8%). The majority of applications were primarily targeted at patients (n = 64 of 89 included studies, 73%),,,,–,–,–,–,–,,,,,–,–,–,–,,,,– or both patients and caregivers (n = 25/89, 27%),–,,,,,,–,,,,,,,–,,,. Information about conflicts of interest and funding was not explicitly stated in 23 (n = 23/89, 25.8%) studies, while 48 (n = 48/89, 53.9%) reported that there were no conflicts of interest or funding. A total of 18 (n = 18/89, 20.2%) studies reported the presence of conflicts of interest and funding,,,,,,,,–,,,,,,,. Most studies did not report information about the institutional review board (IRB) approval (n = 55/89, 61.8%) or deemed IRB approval unnecessary (n = 28/89, 31.5%). Six studies obtained IRB approval (n = 6/89, 6.7%),,–,.

12An overview of the presence of codes for each study is provided in the Supplementary Dataset file 3. The majority of articles investigated the use and feasibility of LLMs as medical chatbots (n = 84/89, 94.4%),–,–,,,–,–, while fewer reports additionally or exclusively focused on the generation of patient information (n = 18/89, 20.2%),,,,,,,,,,–,,,,, including clinical documentation such as informed consent forms (n = 5/89, 5.6%),,,, and discharge instructions (n = 1/89, 1.1%), or translation/summarization tasks of medical texts (n = 5/89, 5.6%),,,,, creation of patient education materials (n = 5/89, 5.6%),,,,, and simplification of radiology reports (n = 2/89, 2.3%),. Most reports evaluated LLMs in English (n = 88/89, 98.9%),–,–, followed by Arabic (n = 2/84, 2.3%),, Mandarin (n = 2/84, 2.3%),, and Korean or Spanish (n = 1/89, 1.1%, respectively). The top-five specialties studied were ophthalmology (n = 10/89, 11.2%),,,,,,,,,, gastroenterology (n = 9/89, 10.1%),,,,,,,,, head and neck surgery/otolaryngology (n = 8/89, 9%),,,,,,,, and radiology,,–, or plastic surgery,,,,, (n = 6/89, 6.7%, respectively). A schematic illustration of the identified concepts of LLM applications in patient care is shown in Fig. 2.

13The thematic synthesis of limitations resulted in two main concepts: one related to design limitations and one related to output. Figure 3 illustrates the hierarchical tree structure and quantity of the codes derived from the thematic synthesis of limitations. Supplementary Dataset file 5 provides an overview of the taxonomy of all identified limitation concepts, including their description and examples.

14In terms of design limitations, many authors noted the limitation that LLMs are not optimized for medical use (n = 46/89, 51.7%),,,,,–,,,,–,,,,,,,,–,–,,,–,–,, including implicit knowledge/lack of clinical context (n = 13/89, 14.6%),,,,,,,–,,, limitations in clinical reasoning (n = 7/89, 7.9%),,,–, limitations in medical image processing/production (n = 5/89, 5.6%),,,,, and misunderstanding of medical information and terms by the model (n = 7/89, 7.9%),,,,,,. In addition, data-related limitations were identified, including limited access to data on the internet (n = 22/89, 24.7%),,,,–,,,,,,–,,,,,,, the undisclosed origin of training data (n = 36/89, 40.5%),,,,,,,,,,,,,–,,,,,,,,,–,,,, limitations in providing, evaluating, and validating references (n = 20/89, 22.5%),,–,,,,,,,,,,,,,,, and storage/processing of sensitive health information (n = 8/89, 9%),,,,,,,. Further second-order concepts included black-box algorithms, i.e., non-explainable AI (n = 12/89, 13.5%),,,,,,,,,,,, limited engagement and dialog capabilities (n = 10/89, 11.2%),,,,,,,,,, and the inability of self-validation and correction (n = 4/89, 4.5%),,,.

15The evaluation of limitations in output data yielded 7 second-order codes concerning the non-reproducibility (n = 38/89, 42.7%),,,,,,,,,,–,,,–,,,,,,,,,,,,,–, non-comprehensiveness (n = 78/89, 87.6%),,,–,–,,–,,,–,–,,–,–, incorrectness (n = 78/89, 87.6%),–,,–,–,–,–,–,–,–, (un-)safeness (n = 39/89, 43.8%),,,,,,–,,,,–,,,,,,,,,–,,,,,,,–,,,, bias (n = 6/89, 6.7%),,,,,, and the dependence of the quality of output on the prompt-/input provided (n = 27/89, 30.3%)–,,,,,,,,,–,,,,,–,,,,, or the environment (n = 16/89, 18%),,,–,,,,,,,,,,.

16For non-reproducibility, key concepts included the non-deterministic nature of the output, e.g., due to inconsistent results across multiple iterations (n = 34/89, 38.2%),,,,,,,,–,,,,,,,,,, and the inability to provide reliable references (n = 20/89, 22.5%),,–,,,,,,,,,,,,,,.

17Non-comprehensiveness included nine concepts related to generic/non-personalized output (n = 34/89, 38.2%),,,,,,,,,,,,,,,,,,,–,,,,,–,, incompleteness of output (n = 68/89, 76.4%),,,–,,–,–,,–,–,,,–,–,,–,–,,–,–, provision of information that is not standard of care (n = 24/89, 27%),,,,,,,,,,,,,,,,,,,,,,, and/or outdated (n = 12/89, 13.5%),,,,,,,,,,,, and production of oversimplified (n = 10/89, 11.2%),,,,,,,,, superfluous (n = 16/89, 18%),,,,,,,,,,,,,,, overcautious (n = 7/89, 7.9%),,,,,,, overempathic (n = 1/89, 1.1%), or output with inappropriate complexity/reading level for patients (n = 22/89, 24.7%),,,,,,,,,,,,,,,,,,,,,.

18For incorrectness, we identified 6 key concepts. Some of the incorrect information could be attributed to what is commonly known as hallucination (n = 38/89, 42.7%),,,,–,–,–,–,,,,,,,,,,,–,,,,,, i.e., the creation of entirely fictitious or false information that has no basis in the input provided or in reality (e.g., “You may be asked to avoid eating or drinking for a few hours before the scan” for a bone scan). Other instances of misinformation were more appropriately classified under alternative concepts of the original psychiatric analogy, as described in detail by Currie et al.,,. These include illusion (n = 12/89, 13.5%),,,,,,,,,,,, which is characterized by the generation of deceptive perceptions or the distortion of information by conflating similar but separate concepts (e.g., suggesting that MRI-type sounds might be experienced during standard nuclear medicine imaging), delirium (n = 34/89, 38.2%),,,,,,,,,,,,–,,,–,–,,,,,,,,, which indicates significant gaps in vital information, resulting in a fragmented or confused understanding of a subject (e.g., omission of crucial information about caffeine cessation for stress myocardial perfusion scans), extrapolation (n = 11/89, 12.4%),,,,,,,,,, which involves applying general knowledge or patterns to specific situations where they are inapplicable (e.g., advice about injection-site discomfort that is more typical of CT contrast administration), delusion (n = 14/89, 15.7%),,,,,,,,,,,,,, a fixed, false belief despite contradictory evidence (e.g., inaccurate waiting times for the thyroid scan), and confabulation (n = 18/89, 20.2%),,–,,,,,,,–,,,, i.e., filling in memory or knowledge gaps with plausible but invented information (e.g., “You should drink plenty of fluids to help flush the radioactive material from your body” for a biliary system–excreted radiopharmaceutical).

19Many studies rated the generated output as unsafe, including misleading (n = 34/89, 38.2%),,,,,,,,–,,,,,,,,–,,,,,,–,,, or even harmful content (n = 26/89, 29.2%),,,,,,,,,–,,,,,,,,,,,–,.

20A minority of reports identified biases in the output, which were related to language (n = 2/89, 2.3%),, insurance status, underserved racial groups, or underrepresented procedures (n = 1/89, 1.1%, each).

21Many authors suggested that performance was related to the prompting/input provided or the environment, i.e., depending on the evidence (n = 7/89, 7.9%),,,,,,, complexity (n = 11/89, 12.4%),,,,,,,,,, specificity (n = 13/89, 14.6%),,,,,,,,,,,,, quantity (n = 3/89, 3.4%),, of the input, type of conversation (n = 3/89, 3.4%),,, or the appropriateness of the output related to the target group (n = 9/89, 10.1%),,,,,,,,, provider/organization (n = 4/89, 4.5%),,,, and local/national medical resources (n = 5/89, 5.6%),,,,.

22In this systematic review, we synthesized the current applications and limitations of LLMs in patient care, incorporating a broad analysis across 29 medical specialties and highlighting key limitations in LLM design and output, providing a comprehensive framework and taxonomy for describing and categorizing limitations that may arise when using LLMs in healthcare settings.

23Most articles examined the use of LLMs based on the GPT-3.5 or GPT-4 architecture for answering medical questions, followed by the generation of patient information, including medical text summarization or translation and clinical documentation. The conceptual synthesis of LLM limitations revealed two key concepts: the first related to design, including 6 second-order and 12 third-order codes, and the second related to output, including 9 second-order and 32 third-order codes. By systematically categorizing the limitations of LLMs in clinical settings, our taxonomy aims to provide healthcare professionals and developers with a framework for assessing potential risks associated with the use of LLMs in patient care. In addition, our work highlights key areas for improvement in the development of LLMs and aims to enable clinicians to make more informed decisions by understanding the limitations inherent in the design and output, thereby supporting the establishment of best practices for LLM use in clinical settings.

24Although many LLMs have been developed specifically for the biomedical domain in recent years, we found that ChatGPT has been a disruptor in the medical literature on LLMs, with GPT-3.5 and GPT-4 accounting for almost 80% of the LLMs examined in this systematic review. While it was not possible to conduct a meta-analysis of the performance on medical tasks, many authors provided a positive outlook towards the integration of LLMs into clinical practice. However, we have conceptualized several key limitations in the design and output of LLMs, some of the most prevalent in our systematic review are briefly discussed in the following paragraphs.

25The majority of studies (n = 55/89) reported limitations that were conceptualized as related to the underlying data of the LLMs studied. Especially the use of proprietary models such as ChatGPT in the biomedical field was a concern in many of the studies analyzed, mainly because of the lack of training data transparency (third-order code: undisclosed origin of training data). In practice, it is widely recognized that limited access to the underlying algorithms, training data, and data processing and storage mechanisms of LLMs is a significant barrier to their application in healthcare. This opacity makes it difficult for healthcare professionals to fully understand how these models function, assess their reliability, or ensure compliance with local medical standards and regulations. Consequently, the use of such models in healthcare settings can be problematic, and the need to recognize and correct potential limitations in the outputs of such models is paramount.

26Moreover, integrating proprietary models into clinical practice introduces a vulnerability to performance changes that occur with model updates. As these models are updated by their developers, functionalities that healthcare providers rely on may be altered or broken, potentially leading to harmful outcomes for patients, which was also conceptualized in our study under output limitations (second-order code: unsafe; third-order codes: misleading/harmful). This unpredictability is a serious concern in the biomedical field, where consistency and reliability are crucial. Notably, the unpredictability of LLMs was another concept of output limitations in our systematic review (second-order code: non-reproducible; third-order codes: non-deterministic/non-referenceable).

27As a result, open-source models such as BioMistral may offer a viable solution. Such open source models not only offer more transparency, as their algorithms and training data are accessible but can also be adapted locally. However, given the limited number of articles on open-source LLMs in our review, we strongly encourage future studies investigating the applicability of open-source LLMs in patient care.

28About half of the studies analyzed reported limitations related to LLMs not being optimized for the medical domain. One possible solution to this limitation may be to provide medical knowledge during inference using RAG. However, even when trained for general purposes, ChatGPT has previously been shown to pass the United States Medical Licensing Examination (USMLE), the German State Examination in Medicine, or even a radiology board-style examination without images–. Although outperformed on specific tasks by specialized medical LLMs, such as Google’s MedPaLM-2, this suggests that general-purpose LLMs can comprehend complex medical literature and case scenarios to a degree that meets professional standards. Furthermore, given the large amounts of data on which proprietary models such as ChatGPT are trained, it is not unlikely that they have been exposed to more medical data overall than smaller specialized models despite being generalist models. Notably, a recent study even suggested that fine-tuning LLMs on biomedical data does not improve performance compared to their general-purpose counterparts.

29It should also be noted that passing these exams does not equate to the practical competence required of a healthcare provider, which was also a limitation identified in our review (third-order codes: implicit knowledge/lack of clinical context; limited clinical reasoning; misunderstanding of medical information/terms; limited in processing/producing medical images). In addition, reliance on exam-based assessments carries a significant risk of bias. For example, if the exam questions or similar variants are publicly available and, thus, may be present in the training data, the LLM does not demonstrate any knowledge outside of training data memorization. In fact, these types of tests can be misleading in estimating the model’s true abilities in terms of comprehension or analytical skills.

30The non-reproducibility of LLM output, as conceptualized in 38 studies, highlights key challenges in ensuring consistency and determinism in LLM-generated results. One major issue is the inherent stochasticity in the models’ architecture, particularly in transformer-based models, which utilize probabilistic techniques during inference (e.g., beam search or temperature sampling). This non-determinism can lead to different outputs for the same input, making it difficult to replicate results exactly across different instances or even across models with identical training data. Further external factors contributing to non-reproducibility, such as variations in hardware, software versions, or context windows, complicate the assurance of reproducibility. As the reproducibility of results is a central principle in medical practice, our concepts highlight the need for more standardized protocols, improved documentation of model configurations, the examination of non-determinism for evaluation purposes, and further research on how robust results can be achieved before implementing LLMs in real-world clinical practice. Interestingly, Ouyang et al. reported that only a minority of studies take non-determinism into account in their experimental evaluation when using ChatGPT for code generation, suggesting that this limitation is also prevalent and overlooked in other domains of LLM use.

31The concept of non-comprehensiveness was prevalent in almost 90% of the studies analyzed (n = 78/89). For this concept, the majority of third-order codes were related to LLM outputs that were incomplete. This issue is particularly significant when considering the application of LLMs in medical tasks such as clinical decision support or diagnosis, where incomplete or partial results can have serious consequences. In clinical practice, missing key information could lead to suboptimal patient outcomes, incorrect diagnoses, or improper treatment recommendations. For instance, an incomplete therapy suggestion could render the entire treatment plan insufficient, potentially resulting in harm to the patient. Given the potential of using LLMs in medical decision-making, these limitations underscore the necessity for expert supervision and validation of LLM outputs depending on their application. While LLMs used as chatbots for general patient inquiries may not require consistent human oversight, using LLMs for treatment advice would require consistent validation to ensure that incomplete information does not lead to adverse outcomes. Depending on their application, the same problem arises when the LLM generates generic or non-personalized information, which was another third-order code identified. The generation of content with high complexity and an inappropriate reading level, which was above the American Medical Association (AMA) recommended 6th-grade reading level in almost all of the 22 studies that analyzed the complexity level of the output, may further limit its usefulness for patient information. Again, the best solution to the lack of comprehensiveness in clinical practice so far seems to be human oversight.

32Incorrectness, alongside non-comprehensiveness (as above), was the most common second-order code, identified in about 90% of studies (n = 78/89). In our conceptual synthesis of incorrect results, we followed the taxonomy of Currie et al. to classify incorrect outputs more precisely into illusions, delusions, delirium, confabulation, and extrapolation, thus proposing a framework for a more precise and structured error classification to improve the characterization of incorrect outputs and enabling more detailed performance comparisons with other research,,.

33Many studies currently refer to all non-factual LLM results as “hallucinations.” However, this generalization fails to capture the complexity of errors when considering the original psychiatric analogy. Simply classifying errors as hallucinations restricts their description to invented information, overlooking errors that, for example, omit critical information and leading to fragmented or confused understanding (third-order code: delirium). Notably, the third-order code “delirium” was observed in nearly as many studies as the third-order code “hallucination.” However, a non-detailed classification of incorrect results can affect not only the comparability of research findings but also has implications for clinical practice. While hallucinations (for example, fabricating instructions like “You may need to fast before a bone scan.”) may not always have serious consequences, errors classified as delirium—such as omitting crucial details like caffeine cessation before a stress myocardial perfusion scan—would always result in undesired outcomes (in the here presented example, most likely in repeating or postponing the examination). As a result, our review advocates for a more detailed classification of incorrect results in order to increase the qualitative comparability of incorrect LLM outputs and, ultimately, the relevance and implications of these results for clinical practice.

34The conceptualization of unsafeness in 39 of 89 studies presents a significant concern when considering the integration of LLMs into medical practice. In the field of medicine, any tool or intervention that could lead to misleading or harmful outcomes must be critically assessed, as the potential for patient harm is high,. Such tools are generally only accepted when the benefits clearly outweigh the risks, and even then, informed consent from the affected individual is essential. While informed consent might ensure that patients understand the risks involved and are able to make an educated decision about their care, which could be obtained, for example, in the form of a disclaimer before using the LLM, studies suggest that even when obtaining informed consent the patient understanding increases not significantly. In the case of disclaimers, there might also be the risk that these are accepted without proper reading or understanding. The practicality of informed consent once LLMs are deeply integrated into clinical workflows also remains an issue, as it is when patients no longer have the ability to opt out, such as in the case of serious illness. In any case, the finding that nearly half of the studies reported limitations related to LLM unsafeness suggests that LLMs are not yet reliable enough for autonomous medical use, and there is a critical need for safety measures and regulatory and human oversight to prevent adverse consequences in medical contexts.

35Further second-order concepts suggested that the output is influenced by the input or environment in which it is expressed. In fact, LLMs can be highly dependent on the quality and specificity of input, making their output prone to errors when faced with vague or incomplete information–. Again, this poses significant risks in patient care, where incorrect outputs can lead to adverse outcomes, such as inappropriate triage or treatment. For example, in our review, eleven studies reported a decrease in performance with increasing complexity of the input, which can have implications in clinical practice, such as failing to consider multifactorial medical issues like comorbidities, thus compromising the quality of care for those patients.

36We found that the environment also influences the appropriateness of LLM outputs in medical settings. Models may recommend treatments that are inappropriate for certain patient populations, such as offering adult care protocols for pediatric patients or suggesting therapies that are not available in certain regions. This also raises ethical concerns, particularly in resource-constrained settings, where making inappropriate or inaccessible recommendations may reinforce existing inequalities and lead to uncomfortable situations for both the healthcare provider and the patient. One solution may be to provide adequate training to LLM users, or in our scenario, to patients, on how to present input to the model to achieve the best results. Another solution is to train or fine-tune the model to the environment in which it will be used. For example, if the LLM is trained on the standard operating procedure for handling patients with major adverse cardiovascular events in a particular hospital, it is more likely to recommend the adequate procedure in this setting than when it is trained on worldwide data from an unknown time frame, where there is a chance that it will suggest non-standard care that may only be relevant in other countries where most of the training data is coming from, or even provides outdated information (which is another third-order code that was conceptualized under non-comprehensiveness) if it is trained on data that is not current.

37Ultimately, only six studies have identified biases in their results, for example, reflecting the unequal representation of certain content or the biases inherent in human-generated text in the training data. Here, we conceptualized the results of studies that identified bias in their analysis and not only mentioned bias as a theoretical limitation. Thus, these results may indicate that the implemented safeguards are effective. On the other hand, identifying bias was not the primary outcome of most studies, and not much is known about the technology and developer policies of proprietary LLMs. Moreover, previous work has shown that automated jailbreak generation is possible across various commercial LLM chatbots. In the end, LLMs are trained on large datasets that inevitably contain biases—such as gender, racial, or cultural biases—embedded in the text. These biases can be amplified or reflected by the models, leading to unfair or harmful outputs. Despite the use of various mitigation techniques, such as debiasing algorithms, curating balanced datasets, or incorporating fairness-focused training objectives, eliminating bias entirely is a persistent challenge–. This is because LLMs learn patterns from their training data, and human biases are inherently present in much of the data they consume. Moreover, the biases introduced or reinforced by LLM are not always obvious, making them more difficult to detect and correct, which may have contributed to the comparatively low number of studies that reported any bias in their results. Notably, subtle biases, such as those related to linguistic connotations, regional dialects, or implicit associations, can be especially insidious and difficult to eliminate through technical safeguards. Therefore, the results of our review may encourage future studies to more explicitly examine the biases inherent in LLMs when used for medical tasks and how such biases could be mitigated.

38Our findings raise a key question when applying LLMs to the medical domain: how can we entrust our patients to LLMs if they are neither reliable nor transparent? Given that models like ChatGPT are already publicly accessible and widely used, patients may already refer to them for medical questions in much the same way they use Google Search, making concerns about their early adoption somewhat academic.

39In addition to the advances in the development of LLMs and the focus on open source, adopting appropriate security measures to prevent the identified LLM limitations in clinical practice out-of-the-box will become increasingly important. For example, strategies to ensure LLM security and privacy can include continuous monitoring for new vulnerabilities, implementing input validation, conducting regular audits of training data, and using secure data pipelines. Additionally, data anonymization, encryption, access controls, and regular security updates are essential to prevent data leakage, model theft, and privacy breaches.

40Moreover, expert oversight of the final LLM output could mitigate any remaining risks in the last instance, ensuring that erroneous or inappropriate suggestions are identified and corrected before they can impact patient care. Recently, efforts have been made in this direction by adopting the widely recognized Physician Documentation Quality Instrument (PDQI-9) for the assessment of AI transcripts and clinical summaries. However, whether ongoing human oversight and validation of LLM-generated content is feasible and can reduce the likelihood of adverse outcomes remains the subject of further research at this early stage of LLM deployment in healthcare.

41Another important factor for the successful clinical implementation of LLMs in patient care could be patient acceptance, which was not assessed in any of the studies analyzed. The growing use of LLMs in healthcare might be perceived as a reduction in the interpersonal relationship between healthcare professionals and patients, potentially leading to a sense of dehumanization in medicine. Therefore, to promote a positive reception of AI tools among patients, incorporating their perspectives already during the AI development and implementation process could be key. Eventually, patient perspectives are already considered in AI regulatory frameworks, such as in the European Union AI Act, which came into force in August 2024. The associated challenges faced by generative AI and LLM, for example, in terms of training data transparency and validation of non-deterministic output, will show which approaches the companies will take to bring these models into compliance with the law. How the notified bodies interpret and enforce the law in practice will likely be decisive for the further development of LLMs in the biomedical sector.

42Our study has limitations. First, our review focused on LLM applications and limitations in patient care, thus excluding research directed at clinicians only. Future studies may extend our synthesis approach to LLM applications that explicitly focus on healthcare professionals. Second, while it was not possible to conduct a meta-analysis of LLM performance due to the different study designs and evaluation methods used, this will be an important area for future work as the field of LLM research in clinical settings continues to evolve. Third, there is a risk that potentially eligible studies were not included in our analysis if they were not present in the 5 databases reviewed or were not available in English. However, we screened nearly 3,000 articles in total and systematically analyzed 89 articles, providing a comprehensive overview of the current state of LLMs in patient care, even if some articles could have been missed. With our chosen cut-off date of January 2022, there is also a risk of missing relevant publications on predecessor LLM models, such as GPT-3, which was introduced in 2020. However, as our review focused on current LLM applications and limitations, it seemed most beneficial to include only recent publications from the last two years on the most advanced models, especially when considering that ChatGPT was first made available in November 2022. Finally, the rapid development and advancement of LLMs make it difficult to keep this systematic review up to date. For example, Gemini 1.5 Pro was published in February 2024, and corresponding articles are not included in this review, which synthesized articles from 2022 to 2023. This also has implications for our introduced taxonomy of LLM limitations, as new limitations may emerge as models evolve, and previous limitations may become less relevant or even obsolete. For example, our taxonomy identifies “limited access to internet data” as a limitation; however, with the introduction of web browsing capabilities for GPT-4 in May 2023, this particular limitation no longer applies to that model. Given these ongoing developments, we strongly encourage future studies to test, update, and extend our taxonomy to ensure that it remains a relevant tool for categorizing LLM limitations in clinical and other high-stakes applications.