Generative Artificial Intelligence to Transform Inpatient Discharge Summaries to Patient-Friendly Language and Format

1Actively involving patients in their own care improves their health-related outcomes., Implementation of the 21st Century Cures Act mandates patients’ access to their clinical notes and other information in their electronic health records. Increased patient access to their clinical notes through widespread availability of electronic patient portals has the potential to improve patient involvement in their own care, as well as confidence in their care from their care partners.,, However, clinical notes are typically filled with technical language and abbreviations that make notes difficult to read and understand for patients and their care partners. This issue can create unnecessary anxiety or potentially delay care recommendations or follow-up for patients and their families. Generative artificial intelligence (AI) has the potential to transform medical information into a patient-friendly language and format that supports high-quality care.

2Discharge from inpatient acute care medicine units is a complex process that has been a frequent focus of transition of care interventions. One important aspect of successful transition from inpatient to outpatient care includes educating patients about their own health. Studies have shown poor quality of written discharge information, with up to 88% of discharge instructions not readable to the population served.,,, Meanwhile, studies suggest that improving readability in discharge summaries might improve outcomes like readmissions. Some efforts to remedy this disconnect include developing patient-friendly discharge letters or using student volunteers to help patients understand their notes. Early studies have already shown that AI-generated plain language notes may be more usable, beneficial to the patient-clinician relationship, and empowering to patients.

3Generative AI has become a potential disruptor across many fields, including medicine. Research on generative AI products in a medical context has shown that they can pass standardized United States Medical Licensing Examinations and write clinical notes. Generative AI has also shown promise in creating medical materials meant for patient consumption. For example, in responses to medical questions, generative AI has given high-quality and empathic responses., Studies have also shown that generative AI may be useful for creating lay summaries of research articles.

4However, despite its promise, implementing generative AI technologies in patient care has raised important patient safety and physician liability concerns. Inaccuracies in generative AI content are a major source of this concern. These inaccurate albeit convincing outputs are termed hallucinations. For instance, Lee et al highlighted an AI chatbot that generated a medical summary from a dialogue between a patient and clinician. The summary indicated that the patient’s body mass index was 14.8 when in fact the necessary information to calculate it was not present. It is notable that when the chatbot was asked to “reread” the output, it was able to identify the hallucination.

5No published work to our knowledge has investigated the use of current-generation generative AI platforms to transform real medical notes into patient-friendly language and format. In our study, we hypothesized that with appropriate programming, a large language model (LLM) could create safe and accurate patient-friendly discharge summaries. To test this hypothesis, we measured the readability and understandability of these patient-friendly discharge summaries and compared them with the readability and understandability of the original discharge summaries. As balancing metrics, we tested accuracy and completeness of the patient-friendly discharge summary.

6This cross-sectional study, as part of a larger project to improve care delivery in our health system, was deemed exempt from institutional review board review based on the NYU Langone Health self-certification protocol. The study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline.

7This was a cross-sectional review of 50 inpatient discharge summaries. The number 50 was chosen a priori based on feasibility. We used Epic Systems reporting workbench to export a dataset containing metadata for all notes of the Discharge Summary Note type across NYU Langone Health Systems from June 1 to 30, 2023, totaling 5025 summaries. We used the Excel 2016 rand function (Microsoft Corporation) to generate a random number corresponding to each note and selected the 200 notes with the lowest random number. A single reviewer confirmed the identified notes were actual discharge summaries written by the General Internal Medicine service and that the patients were not discharged as dead. For final inclusion in the study, we selected 50 of the remaining notes with the lowest random numbers. Our sample included discharges from all of NYU Langone’s hospital campuses and did not include more than 1 discharge from any single patient. Discharge summaries as they appear in the medical record, which we refer to as original discharge summaries, were used as inputs in our generative AI prompt, and the outputs were what we refer to as patient-friendly discharge summaries.

8Our project team first worked with the organization’s patient family council to understand characteristics that made a discharge summary patient friendly. We then reviewed our institution’s discharge summary standard and reviewed scholarly sources. Using this information, we identified a subset of required elements that we aimed to include in the patient-friendly summary: admission date, discharge date, indication for admission, history of present illness, hospital course, diagnoses, procedures, and discharge physician. Elements of the original discharge summary that we aimed to exclude from the patient friendly version were the billing and coding table, admission source, diagnostic studies, admitting physician, and discharge condition. In line with the best practice for patient education materials, we made the patient-friendly summary 1 page in length. We chose a question-and-answer format. Figure 1 presents an example of a patient-friendly discharge summary format.

9We developed the prompt using the institutionally licensed, Health Insurance Portability and Accountability Act–compliant version of GPT-4 on the Azure Open AI studio platform. The prompt development team included a physician data scientist, 2 board-certified physician informaticists, 2 board-certified internal medicine specialists, a resident physician, and an AI engineer (J.Z., J.M.K., S.B., J.A., Y.A., and J.F.). Working independently, physician team members began with basic prompts testing and refining, evolving their approach through hundreds of iterations, a process known as prompt engineering. The physicians individually kept experiment journals, documenting test results for each iteration, and met weekly for 6 weeks comparing findings with each other and refining their approach. Since much of the prompt design came from several members of our group working independently, we used the following directive to align our efforts: “create a discharge summary for a hospitalization on acute care medicine unit that is readable for a patient at a sixth grade reading level.” At the end of the 6 weeks, the best-performing prompt was chosen. This prompt used few-shot learning, a technique where the AI system learns by example from sample inputs and outputs. To see our prompt, refer to eFigure 1 in Supplement 1. We did not include the original discharge summaries used for prompt engineering in the final study. The process of prompt creation was undertaken from May 24 to July 13, 2023.

10The 50 original discharge summaries were preprocessed to include only elements of the aforementioned discharge summary. Then they were input into the LLM together with the engineered prompt to generate 50 patient-friendly discharge summaries. All discharge summaries were processed by the LLM between July 26 and August 5, 2023. For evaluating readability, we used the Flesch-Kincaid readability tests. These include the Flesch-Kincaid Reading Ease score and Flesch-Kincaid Reading Grade Level, both of which use word and sentence lengths to estimate how difficult an English text is to understand. We used the total word count as a measure of document length. We used Microsoft Word software to measure Flesch-Kincaid scores and total word count.,

11To assess understandability, we used the Patient Education Materials Assessment Tool (PEMAT) understandability scale. The PEMAT is a validated instrument specifically designed to gauge the extent to which patients can comprehend and act on health-related educational resources., The PEMAT consists of 2 independent scales: one scoring materials based on 19 understandability criteria, and another on a 7-point actionability scale.

12The understandability scale focuses on 19 aspects related to clarity, organization, layout, and the use of visual aids. Each element is rated as 0 (does not satisfy the criteria), 1 (meets the criteria), or not applicable. The final score is calculated as a percentage, with higher scores indicating superior understandability. As our organization’s discharge summaries do not include patient actions, which are instead provided in the discharge instructions, the actionability scale component of PEMAT was not included in our analysis.

13We used 2 independent physician reviewers (J.Z. and J.M.K.) to evaluate the understandability of all 50 patient summaries using the 19-point understandability scale. Where statistical testing is performed, the results are taken as the mean of the scores given by the 2 reviewers.

14We separately recruited 25 resident physician volunteers to evaluate accuracy and completeness of the patient-friendly discharge summaries. These physicians regularly compose discharge summaries and are familiar with the standard elements of discharge summaries at our institution. Each discharge summary was reviewed by 2 of the resident physicians independently. Accuracy was measured on a 6-point scale modeled after previous accuracy scales developed for generative AI research. The original discharge summary was used as the criterion standard judgment of accuracy. We then further reviewed free-text description of inaccuracies for patterns to see if categories emerged. For reviews that reported any inaccuracy, we asked whether there was any potential safety concern. For completeness, we asked reviewers to indicate whether each of the prespecified 8 elements of the discharge summary was present (eFigure 2 in Supplement 1). We considered the possibility that the simplified summary could contain useful content that was not included in the criterion standard note. We did not measure this scenario.

15We used paired t tests to compare the differences in word count, Flesch-Kincaid Reading Ease score, and Flesch-Kincaid Reading Grade Level between original and patient-friendly discharge summaries. Since the PEMAT score is not normally distributed, we used the Wilcoxon rank sum test to examine the association with the 2 groups. Additionally, we conducted an analysis of variance to access the association of sex and race and ethnicity reported in the electronic medical record (owing to concerns regarding gender and racial biases in AI-based technologies), length of stay, and original discharge summary word count with accuracy. Interrater reliability was calculated using the percentage agreement. All statistical analyses were performed using R, version 4.3.0 (R Project for Statistical Computing). Two-sided P <.05 was considered statistically significant.

16Among 50 patient discharges included in the analysis, 31 patients (62.0%) identified as female and 19 (38.0%) identified as male, with a median age of 65.5 (IQR, 59.0-77.5) years. Racial and ethnic identity in the electronic health record included 2 (4.0%) Asian patients, 15 (30.0%) Black or African American patients, 25 (50.0%) White patients, and 8 (16.0%) patients of other race or ethnicity (including Filipino; Hispanic, Latino, or Spanish; multiracial Black and White; and not listed) (Table 1). We compared measures of readability between the original and the patient-friendly discharge summaries (Figure 2). We found the patient-friendly discharge summaries had fewer words than the original discharge summary, a difference that was statistically significant (mean [SD], 338 [256] vs 1520 [904] words; P <.001). Readability as measured by the Flesch-Kincaid Reading Ease score was significantly higher in the patient-friendly discharge summaries compared with the original discharge summaries (mean [SD], 69.5 [3.2] vs 35.5 [9.8]; P <.001). Conversely, Flesch-Kincaid Grade Level was significantly improved (ie, was lower) for patient-friendly discharge summaries (mean [SD], 6.2 [0.5] vs 11.0 [1.5] grade levels; P <.001)

17We found a significantly improved understandability based on PEMAT in the patient-friendly discharge summaries, with a score of 81% compared with 13% in original discharge summaries (P <.001). Interrater reliability was good, with 97% agreement.

18With regard to accuracy, 54 of 100 reviews gave a top box rating (Figure 3). None of the reviews rated accuracy as completely incorrect, and only 1 of the 100 was rated as more incorrect than correct. Interrater reliability of top box accuracy was poor at 48%. For the reviews that were not given the top box rating of accuracy, 18 of 46 (39.1%) were considered to have potential safety risks. Free-text descriptions of inaccuracy were found to fall into 3 categories after our review: omissions, hallucinations, and other. The most common inaccuracy was omission, making up 24 of 46 (52.1%), while hallucinations made up 4 of 46 (8.7%), and the rest fell into the other category (18 of 46 [39.1%]). For examples of inaccuracies, see Table 2.

19As for completeness, 56 of the 100 reviews rated as entirely complete, 26 had only 1 category incomplete, 11 had 2 categories incomplete, 6 had 3 categories incomplete, and 1 had 4 categories incomplete (eFigure 3 in Supplement 1). The most common category graded as incomplete was procedures. Of the 43 reviews that indicated a procedure was performed, 17 (39.5%) were graded as incomplete. The second most common category graded as incomplete was history of present illness, with 25 of 100 (25.0%) being incomplete. The next most common was procedure, with 17 of 100 reviews indicating that this was incomplete. Interrater reliability was good across categories at 88%.

20We did not find any association between word count in the original discharge summary and accuracy nor between length of stay and accuracy. Given the presence of bias reported in AI, we explored whether accuracy was associated with race or sex and did not find such an association (eTable in Supplement 1).

21In this cross-sectional study of 50 discharge summaries, we found that generative AI successfully transformed discharge summaries into a format that was more readable and understandable for patients. Conventional standards of readability recommend creating materials appropriate for a sixth grade reading level. We were able to show that our patient-friendly discharge summaries were consistently at a sixth or seventh grade reading level. This is markedly different from original discharge summaries, in which reading ease varies widely but is typically at the 11th grade reading level. We measured understandability using the PEMAT instrument and found patient-friendly discharge summaries to have high levels of understandability. This is also a marked difference from original discharge summaries.

22It is notable that in regard to word count, we found a marked difference, with original discharge summaries having a high mean word count of 1520. While we did not parse out why these word counts were high, we suspect that unnecessarily duplicated material makes up a large portion, which has been suggested in the literature. These high word counts may be a reason that patient records are inaccessible and illustrate where generative AI can be helpful in sorting through large amounts of information.

23While others have used various AI models to develop discharge summaries, to our knowledge, our effort represents the first use of new-generation AI platforms on real patients’ notes to transform them into a patient-friendly format. This also accords with the mission of certain patient advocacy groups, such as OpenNotes, to empower patients to own and understand their own health information and increase transparency.

24Despite this success, our balancing measures of accuracy and completeness need to be addressed before any future deployment. Only 54 of 100 reviews had top box accuracy, and safety concerns were reported in 18. While previous studies have highlighted bias and hallucination as a cause of generative AI inaccuracies, most inaccuracies in our study were attributed to omission of key information (52.1%) vs hallucination (8.7%). Since all omitted information must be contained in the original discharge summary that is made available to patients, we think that safety concerns due to omissions will be of less risk during deployment. Our bigger worry consisted of inaccuracies due to hallucinations. It is also worth noting we did not find an association between accuracy and either sex or race to suggest bias, though it is important to keep in mind that our study was not powered for this question. In regard to completeness, we found that components most frequently missing from the patient-friendly discharge summaries were history of present illness and procedures.

25We think a major source of inaccuracy due to omission and incompleteness comes from prompt engineering that optimized for readability and understandability. For example, limiting the number of words in a sentence or a document is considered more understandable. This makes it difficult to provide a detailed, comprehensive description of a complex patient condition. Future iterations will have to explore the trade-off between readability and understandability on one hand and completeness on the other.

26Our study has several limitations. First, we limited our patient-friendly discharge summaries to English. This is a major issue with original discharge summaries as well and is a barrier to transparent, equitable care of non–English-speaking patients and caregivers., We hope to include non-English outputs in future iterations of our initiative. Another limitation was validation of survey instruments. Given the novelty of using generative AI on medical records, there is a paucity of validated instruments to measure accuracy and completeness. We developed an original instrument to do this; therefore, it does not have robust validation. Furthermore, while PEMAT is well validated for patient educational materials, it has not been validated in this particular use. Interrater reliability was good for PEMAT and completeness measurements; however, accuracy showed low levels of interrater reliability. To further test the significance of our survey results, future directions might include using human-generated patient-friendly discharge summaries as a comparator group for accuracy and completeness.

27Another limitation is the lack of generalizability. We looked at only 50 discharge summaries from general medicine floors during a single month in 1 health system, so our results may not be generalizable to other settings. Furthermore, the prompt was specifically designed for use within our health care system.

28In this cross-sectional study, the strong performance of our generative AI prompt in creating highly readable and understandable discharge summaries shows the promise of using generative AI to make patient information more accessible to patients themselves. However, limitations in accuracy and completeness, which in turn affect safety, remain a hurdle for full-scale implementation. To address these limitations, we will further explore prompt strategies such as chain of thought and use of additional inputs from the patient records, such as progress notes. Initial implementation will require robust clinical safeguards such as physician review before publishing the output to patients.