No retraction, correction or concern found when last checked on Oct 21, 2026.

A framework for human evaluation of large language models in healthcare derived from literature review

Thomas Yu Chow Tam, Sonish Sivarajkumar, Sumit Kapoor and 12 more

npj Digital Medicine · 2024 · CC BY

Abstract

With generative artificial intelligence (GenAI), particularly large language models (LLMs), continuing to make inroads in healthcare, assessing LLMs with human evaluations is essential to assuring safety and effectiveness. This study reviews existing literature on human evaluation methodologies for LLMs in healthcare across various medical specialties and addresses factors such as evaluation dimensions, sample types and sizes, selection, and recruitment of evaluators, frameworks and metrics, evaluation process, and statistical analysis type. Our literature review of 142 studies shows gaps in reliability, generalizability, and applicability of current human evaluation practices. To overcome such significant obstacles to healthcare LLM developments and deployments, we propose QUEST, a comprehensive and practical framework for human evaluation of LLMs covering three phases of workflow: Planning, Implementation and Adjudication, and Scoring and Review. QUEST is designed with five proposed evaluation principles: Quality of Information, Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and Trust and Confidence.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 108

1.
Ouyang L, et al. Training language models to follow instructions with human feedback. Adv. Neural Inf. Process. Syst. 2022;35:27730-27744.
Not in our library
2.
Touvron, H. et al. Llama 2: open foundation and fine-tuned chat models. Preprint at https://arxiv.org/abs/2307.09288 (2023).
Not in our library
3.
Ayers JW, et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern. Med. 2023;183:589-596.
Not in our library10.1001/jamainternmed.2023.1838
4.
Chari S, et al. Informing clinical assessment by contextualizing post-hoc explanations of risk prediction models in type-2 diabetes. Artif. Intell. Med. 2023;137:102498.
Not in our library10.1016/j.artmed.2023.102498
5.
Using AI-generated suggestions from ChatGPT to optimize clinical decision support · 2023
No notice
6.
Alapati R, et al. Evaluating insomnia queries from an artificial intelligence chatbot for patient education. J. Clin. Sleep. Med. JCSM. Publ. Am. Acad. Sleep. Med. 2024;20:583-594.
Not in our library10.5664/jcsm.10948
7.
Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. 40th Annual Meeting of the Association for Computational Linguistics (eds. Isabelle, P., Charniak, E. & Lin, D.) 311–318 (Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 2002). 10.3115/1073083.1073135.
Not in our library
8.
Lin, C.-Y. ROUGE: a package for automatic evaluation of summaries. Text Summarization Branches Out (Association for Computational Linguistics, Barcelona, Spain, 2004).
Not in our library
9.
Liang, P. et al. Holistic evaluation of language models. CoRR10.48550/arXiv.2211.09110 (2022).
Not in our library
10.
Large language models encode clinical knowledge · 2023
Corrected
11.
Sivarajkumar S, Kelley M, Samolyk-Mazzanti A, Visweswaran S, Wang Y. An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing: algorithm development and validation study. JMIR Med. Inf. 2024;12:e55318.
Not in our library10.2196/55318
12.
Chiang, C.-H. & Lee, H. Can large language models be an alternative to human evaluations? In Proc. the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds. Rogers, A., Boyd-Graber, J. & Okazaki, N.) 15607–15631 (Association for Computational Linguistics, Toronto, Canada, 2023). 10.18653/v1/2023.acl-long.870.
Not in our library
13.
Wei Q, et al. Evaluation of ChatGPT-generated medical responses: a systematic review and meta-analysis. J. Biomed. Inform. 2024;151:104620.
Not in our library10.1016/j.jbi.2024.104620
14.
Assessing the research landscape and clinical utility of large language models: a scoping review · 2024
No notice
15.
Yuan, M. et al. Large language models illuminate a progressive pathway to artificial intelligent healthcare assistant. Med. Plus1, 100030 (2024).
Not in our library
16.
Awasthi, R. et al. HumanELY: Human evaluation of LLM yield, using a novel web-based evaluation tool. 2023.12.22.23300458 Preprint at 10.1101/2023.12.22.23300458 (2023).
Not in our library
17.
Mongan J, Moy L, Kahn CE. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers. Radiol. Artif. Intell. 2020;2:e200029.
Not in our library10.1148/ryai.2020200029
18.
Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol · 2021
No notice
19.
Martindale APL, et al. Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines. Nat. Commun. 2024;15:1619.
Not in our library10.1038/s41467-024-45355-3
20.
Norgeot B, et al. Minimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist. Nat. Med. 2020;26:1320-1324.
Not in our library10.1038/s41591-020-1041-y
21.
Lechien JR, Georgescu BM, Hans S, Chiesa-Estomba CM. ChatGPT performance in laryngology and head and neck surgery: a clinical case-series. Eur. Arch. Oto-Rhino-Laryngol. J. Eur. Fed. Oto-Rhino-Laryngol. Soc. EUFOS Affil. Ger. Soc. Oto-Rhino-Laryngol. - Head. Neck Surg. 2024;281:319-333.
Not in our library10.1007/s00405-023-08282-5
22.
Seth I, et al. Exploring the role of a large language model on carpal tunnel syndrome management: an observation study of ChatGPT. J. Hand Surg. 2023;48:1025-1033.
Not in our library10.1016/j.jhsa.2023.07.003
23.
Maillard A, et al. Can Chatbot artificial intelligence replace infectious diseases physicians in the management of bloodstream infections? A prospective cohort study. Clin. Infect. Dis. Publ. Infect. Dis. Soc. Am. 2024;78:825-832.
Not in our library10.1093/cid/ciad632
24.
Yaneva V, Baldwin P, Jurich DP, Swygert K, Clauser BE. Examining ChatGPT performance on USMLE sample items and implications for assessment. Acad. Med. J. Assoc. Am. Med. Coll. 2024;99:192-197.
Not in our library10.1097/acm.0000000000005549
25.
Wu Y, et al. Evaluating the performance of the language model ChatGPT in responding to common questions of people with epilepsy. Epilepsy Behav. 2024;151:109645.
Not in our library10.1016/j.yebeh.2024.109645
26.
Ghosh A, Bir A. Evaluating ChatGPT’s ability to solve higher-order questions on the competency-based medical education curriculum in medical biochemistry. Cureus. 2023;15:e37023.
Not in our library10.7759/cureus.37023
27.
Choi W. Assessment of the capacity of ChatGPT as a self-learning tool in medical pharmacology: a study using MCQs. BMC Med. Educ. 2023;23:864.
Not in our library10.1186/s12909-023-04832-x
28.
Kavadella A, Dias da Silva MA, Kaklamanos EG, Stamatopoulos V, Giannakopoulos K. Evaluation of ChatGPT’s real-life implementation in undergraduate dental education: mixed methods study. JMIR Med. Educ. 2024;10:e51344.
Not in our library10.2196/51344
29.
Baglivo F, et al. Exploring the possible use of AI chatbots in public health education: feasibility study. JMIR Med. Educ. 2023;9:e51421.
Not in our library10.2196/51421
30.
Hatia A, et al. Accuracy and completeness of ChatGPT-generated information on interceptive orthodontics: a multicenter collaborative study. J. Clin. Med. 2024;13:735.
Not in our library10.3390/jcm13030735
31.
Kienzle A, Niemann M, Meller S, Gwinner C. ChatGPT may offer an adequate substitute for informed consent to patients prior to total knee arthroplasty-yet caution is needed. J. Pers. Med. 2024;14:69.
Not in our library10.3390/jpm14010069
32.
Peng W, et al. Evaluating AI in medicine: a comparative analysis of expert and ChatGPT responses to colorectal cancer questions. Sci. Rep. 2024;14:2840.
Not in our library10.1038/s41598-024-52853-3
33.
Xie Y, Seth I, Rozen WM, Hunter-Smith DJ. Evaluation of the artificial intelligence chatbot on breast reconstruction and its efficacy in surgical research: a case study. Aesthet. Plast. Surg. 2023;47:2360-2369.
Not in our library10.1007/s00266-023-03443-7
34.
Evaluating large language models on medical evidence summarization · 2023
No notice
35.
Moramarco F, et al. Towards more patient friendly clinical notes through language models and ontologies. AMIA Annu. Symp. Proc. AMIA Symp. 2021;2021:881-890.
Not in our library
36.
Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care Questions · 2023
No notice
37.
Diagnostic Accuracy of Differential-Diagnosis Lists Generated by Generative Pretrained Transformer 3 Chatbot for Clinical Vignettes with Common Chief Complaints: A Pilot Study · 2023
No notice
38.
Medical Specialties & Subspecialties | ABMS. American Board of Medical Specialtieshttps://www.abms.org/member-boards/specialty-subspecialty-certificates/. Accessed 19 Sep, 2024.
Not in our library
39.
Schmidt S, Zimmerer A, Cucos T, Feucht M, Navas L. Simplifying radiologic reports with natural language processing: a novel approach using ChatGPT in enhancing patient understanding of MRI results. Arch. Orthop. Trauma Surg. 2024;144:611-618.
Not in our library10.1007/s00402-023-05113-4
40.
Truhn D, et al. A pilot study on the efficacy of GPT-4 in providing orthopedic treatment recommendations from MRI reports. Sci. Rep. 2023;13:20159.
Not in our library10.1038/s41598-023-47500-2

All 108 references

Reads

Read by people on this page

None yet.

Checked or read by AI through naulon

None yet.

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.

Are you an author?

Is this your paper?

Claim it to add your note and be paid when agents cite it. Free.