No retraction, correction or concern found when last checked on Oct 21, 2026.

A framework for human evaluation of large language models in healthcare derived from literature review

Thomas Yu Chow Tam, Sonish Sivarajkumar, Sumit Kapoor and 12 more

npj Digital Medicine · 2024 · CC BY

Abstract

With generative artificial intelligence (GenAI), particularly large language models (LLMs), continuing to make inroads in healthcare, assessing LLMs with human evaluations is essential to assuring safety and effectiveness. This study reviews existing literature on human evaluation methodologies for LLMs in healthcare across various medical specialties and addresses factors such as evaluation dimensions, sample types and sizes, selection, and recruitment of evaluators, frameworks and metrics, evaluation process, and statistical analysis type. Our literature review of 142 studies shows gaps in reliability, generalizability, and applicability of current human evaluation practices. To overcome such significant obstacles to healthcare LLM developments and deployments, we propose QUEST, a comprehensive and practical framework for human evaluation of LLMs covering three phases of workflow: Planning, Implementation and Adjudication, and Scoring and Review. QUEST is designed with five proposed evaluation principles: Quality of Information, Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and Trust and Confidence.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 108

1.
Ouyang L, et al. Training language models to follow instructions with human feedback. Adv. Neural Inf. Process. Syst. 2022;35:27730-27744.
Not in our library
2.
Touvron, H. et al. Llama 2: open foundation and fine-tuned chat models. Preprint at https://arxiv.org/abs/2307.09288 (2023).
Not in our library
3.
Ayers JW, et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern. Med. 2023;183:589-596.
Not in our library10.1001/jamainternmed.2023.1838
4.
Chari S, et al. Informing clinical assessment by contextualizing post-hoc explanations of risk prediction models in type-2 diabetes. Artif. Intell. Med. 2023;137:102498.
Not in our library10.1016/j.artmed.2023.102498
5.
Using AI-generated suggestions from ChatGPT to optimize clinical decision support · 2023
No notice
6.
Alapati R, et al. Evaluating insomnia queries from an artificial intelligence chatbot for patient education. J. Clin. Sleep. Med. JCSM. Publ. Am. Acad. Sleep. Med. 2024;20:583-594.
Not in our library10.5664/jcsm.10948
7.
Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proc. 40th Annual Meeting of the Association for Computational Linguistics (eds. Isabelle, P., Charniak, E. & Lin, D.) 311–318 (Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 2002). 10.3115/1073083.1073135.
Not in our library
8.
Lin, C.-Y. ROUGE: a package for automatic evaluation of summaries. Text Summarization Branches Out (Association for Computational Linguistics, Barcelona, Spain, 2004).
Not in our library
9.
Liang, P. et al. Holistic evaluation of language models. CoRR10.48550/arXiv.2211.09110 (2022).
Not in our library
10.
Large language models encode clinical knowledge · 2023
Corrected
11.
Sivarajkumar S, Kelley M, Samolyk-Mazzanti A, Visweswaran S, Wang Y. An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing: algorithm development and validation study. JMIR Med. Inf. 2024;12:e55318.
Not in our library10.2196/55318
12.
Chiang, C.-H. & Lee, H. Can large language models be an alternative to human evaluations? In Proc. the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (eds. Rogers, A., Boyd-Graber, J. & Okazaki, N.) 15607–15631 (Association for Computational Linguistics, Toronto, Canada, 2023). 10.18653/v1/2023.acl-long.870.
Not in our library
13.
Wei Q, et al. Evaluation of ChatGPT-generated medical responses: a systematic review and meta-analysis. J. Biomed. Inform. 2024;151:104620.
Not in our library10.1016/j.jbi.2024.104620
14.
Assessing the research landscape and clinical utility of large language models: a scoping review · 2024
No notice
15.
Yuan, M. et al. Large language models illuminate a progressive pathway to artificial intelligent healthcare assistant. Med. Plus1, 100030 (2024).
Not in our library
16.
Awasthi, R. et al. HumanELY: Human evaluation of LLM yield, using a novel web-based evaluation tool. 2023.12.22.23300458 Preprint at 10.1101/2023.12.22.23300458 (2023).
Not in our library
17.
Mongan J, Moy L, Kahn CE. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers. Radiol. Artif. Intell. 2020;2:e200029.
Not in our library10.1148/ryai.2020200029
18.
Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol · 2021
No notice
19.
Martindale APL, et al. Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines. Nat. Commun. 2024;15:1619.
Not in our library10.1038/s41467-024-45355-3
20.
Norgeot B, et al. Minimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist. Nat. Med. 2020;26:1320-1324.
Not in our library10.1038/s41591-020-1041-y
21.
Lechien JR, Georgescu BM, Hans S, Chiesa-Estomba CM. ChatGPT performance in laryngology and head and neck surgery: a clinical case-series. Eur. Arch. Oto-Rhino-Laryngol. J. Eur. Fed. Oto-Rhino-Laryngol. Soc. EUFOS Affil. Ger. Soc. Oto-Rhino-Laryngol. - Head. Neck Surg. 2024;281:319-333.
Not in our library10.1007/s00405-023-08282-5
22.
Seth I, et al. Exploring the role of a large language model on carpal tunnel syndrome management: an observation study of ChatGPT. J. Hand Surg. 2023;48:1025-1033.
Not in our library10.1016/j.jhsa.2023.07.003
23.
Maillard A, et al. Can Chatbot artificial intelligence replace infectious diseases physicians in the management of bloodstream infections? A prospective cohort study. Clin. Infect. Dis. Publ. Infect. Dis. Soc. Am. 2024;78:825-832.
Not in our library10.1093/cid/ciad632
24.
Yaneva V, Baldwin P, Jurich DP, Swygert K, Clauser BE. Examining ChatGPT performance on USMLE sample items and implications for assessment. Acad. Med. J. Assoc. Am. Med. Coll. 2024;99:192-197.
Not in our library10.1097/acm.0000000000005549
25.
Wu Y, et al. Evaluating the performance of the language model ChatGPT in responding to common questions of people with epilepsy. Epilepsy Behav. 2024;151:109645.
Not in our library10.1016/j.yebeh.2024.109645
26.
Ghosh A, Bir A. Evaluating ChatGPT’s ability to solve higher-order questions on the competency-based medical education curriculum in medical biochemistry. Cureus. 2023;15:e37023.
Not in our library10.7759/cureus.37023
27.
Choi W. Assessment of the capacity of ChatGPT as a self-learning tool in medical pharmacology: a study using MCQs. BMC Med. Educ. 2023;23:864.
Not in our library10.1186/s12909-023-04832-x
28.
Kavadella A, Dias da Silva MA, Kaklamanos EG, Stamatopoulos V, Giannakopoulos K. Evaluation of ChatGPT’s real-life implementation in undergraduate dental education: mixed methods study. JMIR Med. Educ. 2024;10:e51344.
Not in our library10.2196/51344
29.
Baglivo F, et al. Exploring the possible use of AI chatbots in public health education: feasibility study. JMIR Med. Educ. 2023;9:e51421.
Not in our library10.2196/51421
30.
Hatia A, et al. Accuracy and completeness of ChatGPT-generated information on interceptive orthodontics: a multicenter collaborative study. J. Clin. Med. 2024;13:735.
Not in our library10.3390/jcm13030735
31.
Kienzle A, Niemann M, Meller S, Gwinner C. ChatGPT may offer an adequate substitute for informed consent to patients prior to total knee arthroplasty-yet caution is needed. J. Pers. Med. 2024;14:69.
Not in our library10.3390/jpm14010069
32.
Peng W, et al. Evaluating AI in medicine: a comparative analysis of expert and ChatGPT responses to colorectal cancer questions. Sci. Rep. 2024;14:2840.
Not in our library10.1038/s41598-024-52853-3
33.
Xie Y, Seth I, Rozen WM, Hunter-Smith DJ. Evaluation of the artificial intelligence chatbot on breast reconstruction and its efficacy in surgical research: a case study. Aesthet. Plast. Surg. 2023;47:2360-2369.
Not in our library10.1007/s00266-023-03443-7
34.
Evaluating large language models on medical evidence summarization · 2023
No notice
35.
Moramarco F, et al. Towards more patient friendly clinical notes through language models and ontologies. AMIA Annu. Symp. Proc. AMIA Symp. 2021;2021:881-890.
Not in our library
36.
Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care Questions · 2023
No notice
37.
Diagnostic Accuracy of Differential-Diagnosis Lists Generated by Generative Pretrained Transformer 3 Chatbot for Clinical Vignettes with Common Chief Complaints: A Pilot Study · 2023
No notice
38.
Medical Specialties & Subspecialties | ABMS. American Board of Medical Specialtieshttps://www.abms.org/member-boards/specialty-subspecialty-certificates/. Accessed 19 Sep, 2024.
Not in our library
39.
Schmidt S, Zimmerer A, Cucos T, Feucht M, Navas L. Simplifying radiologic reports with natural language processing: a novel approach using ChatGPT in enhancing patient understanding of MRI results. Arch. Orthop. Trauma Surg. 2024;144:611-618.
Not in our library10.1007/s00402-023-05113-4
40.
Truhn D, et al. A pilot study on the efficacy of GPT-4 in providing orthopedic treatment recommendations from MRI reports. Sci. Rep. 2023;13:20159.
Not in our library10.1038/s41598-023-47500-2
41.
Large language model (ChatGPT) as a support tool for breast tumor board · 2023
No notice
42.
Allahqoli L, Ghiasvand MM, Mazidimoradi A, Salehiniya H, Alkatout I. Diagnostic and management performance of ChatGPT in obstetrics and gynecology. Gynecol. Obstet. Invest. 2023;88:310-313.
Not in our library10.1159/000533177
43.
Yapar D, Demir Avcı Y, Tokur Sonuvar E, Eğerci ÖF, Yapar A. ChatGPT’s potential to support home care for patients in the early period after orthopedic interventions and enhance public health. Jt. Dis. Relat. Surg. 2024;35:169-176.
Not in our library10.52312/jdrs.2023.1402
44.
Huespe IA, et al. Clinical research with large language models generated writing—clinical research with AI-assisted writing (CRAW) study. Crit. Care Explor. 2023;5:e0975.
Not in our library10.1097/cce.0000000000000975
45.
Shao C-Y, et al. Appropriateness and comprehensiveness of using ChatGPT for perioperative patient education in thoracic surgery in different language contexts: survey study. Interact. J. Med. Res. 2023;12:e46900.
Not in our library10.2196/46900
46.
Bazzari FH, Bazzari AH. Utilizing ChatGPT in telepharmacy. Cureus. 2024;16:e52365.
Not in our library10.7759/cureus.52365
47.
Qu RW, Qureshi U, Petersen G, Lee SC. Diagnostic and management applications of ChatGPT in structured otolaryngology clinical scenarios. OTO Open. 2023;7:e67.
Not in our library10.1002/oto2.67
48.
Agarwal M, Goswami A, Sharma P. Evaluating ChatGPT-3.5 and Claude-2 in answering and explaining conceptual medical physiology multiple-choice questions. Cureus. 2023;15:e46222.
Not in our library10.7759/cureus.46222
49.
Wilhelm TI, Roos J, Kaczmarczyk R. Large language models for therapy recommendations across 3 clinical specialties: comparative study. J. Med. Internet Res. 2023;25:e49324.
Not in our library10.2196/49324
50.
How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment · 2023
No notice
51.
Dennstädt F, et al. Exploring capabilities of large language models such as ChatGPT in radiation oncology. Adv. Radiat. Oncol. 2024;9:101400.
Not in our library10.1016/j.adro.2023.101400
52.
ChatGPT outperforms humans in emotional awareness evaluations · 2023
No notice
53.
Sallam M, Al-Salahat K, Al-Ajlouni E. ChatGPT performance in diagnostic clinical microbiology laboratory-oriented case scenarios. Cureus. 2023;15:e50629.
Not in our library10.7759/cureus.50629
54.
Varshney D, Zafar A, Behera NK, Ekbal A. Knowledge grounded medical dialogue generation using augmented graphs. Sci. Rep. 2023;13:3310.
Not in our library10.1038/s41598-023-29213-8
55.
Choi J, et al. Availability of ChatGPT to provide medical information for patients with kidney cancer. Sci. Rep. 2024;14:1542.
Not in our library10.1038/s41598-024-51531-8
56.
Krishna, K. et al. LongEval: guidelines for human evaluation of faithfulness in long-form summarization. In Proc. 17th Conference of the European Chapter of the Association for Computational Linguistics (eds. Vlachos, A. & Augenstein, I.) 1650–1669 (Association for Computational Linguistics, Dubrovnik, Croatia, 2023). 10.18653/v1/2023.eacl-main.121.
Not in our library
57.
Moramarco, F. et al. Human evaluation and correlation with automatic metrics in consultation note generation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers) (eds Muresan, S Nakov, P & Villavicencio, A.) 5739–5754 (Association for Computational Linguistics, 2022).
Not in our library
58.
Radford, A. et al. Learning transferable visual models from natural language supervision. Preprint at 10.48550/arXiv.2103.00020 (2021).
Not in our library
59.
Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1 (Long and Short Papers) (eds Burstein, J Doran, C & Solorio, T.) 4171–4186 (Association for Computational Linguistics, 2019).
Not in our library
60.
Liu, Y. et al. Roberta: a robustly optimized BERT pretraining approach. Preprint at https://arxiv.org/abs/1907.11692 (2019).
Not in our library
61.
Draschl A, et al. Are ChatGPT’s free-text responses on periprosthetic joint infections of the hip and knee reliable and useful? J. Clin. Med. 2023;12:6655.
Not in our library10.3390/jcm12206655
62.
Khlaif ZN, et al. The potential and concerns of using AI in scientific research: ChatGPT performance evaluation. JMIR Med. Educ. 2023;9:e47049.
Not in our library10.2196/47049
63.
Rogasch JMM, et al. ChatGPT: can you prepare my patients for [18 F]FDG PET/CT and explain my reports? J. Nucl. Med. 2023;64:1876-1879.
Not in our library10.2967/jnumed.123.266114
64.
Sallam M, Barakat M, Sallam M. A Preliminary Checklist (METRICS) to standardize the design and reporting of studies on generative artificial intelligence–based models in health care education and practice: development study involving a literature review. Interact. J. Med. Res. 2024;13:e54704.
Not in our library10.2196/54704
65.
Jenko, N. et al. An evaluation of AI generated literature reviews in musculoskeletal radiology. Surg. J. R. Coll. Surg. Edinb. Irel. 00008–8 (2024) 10.1016/j.surge.2023.12.005.
Not in our library10.1016/j.surge.2023.12.005
66.
Deiana G, et al. Artificial intelligence and public health: evaluating ChatGPT responses to vaccination myths and misconceptions. Vaccines. 2023;11:1217.
Not in our library10.3390/vaccines11071217
67.
Roosan D, et al. Effectiveness of ChatGPT in clinical pharmacy and the role of artificial intelligence in medication therapy management. J. Am. Pharm. Assoc. 2024;64:422-428.e8.
Not in our library10.1016/j.japh.2023.11.023
68.
Ayub I, Hamann D, Hamann CR, Davis MJ. Exploring the potential and limitations of chat generative pre-trained transformer (ChatGPT) in generating board-style dermatology questions: a qualitative analysis. Cureus. 2023;15:e43717.
Not in our library10.7759/cureus.43717
69.
An Y, Fang Q, Wang L. Enhancing patient education in cancer care: Intelligent cancer patient education model for effective communication. Comput. Biol. Med. 2024;169:107874.
Not in our library10.1016/j.compbiomed.2023.107874
70.
Babayiğit O, Tastan Eroglu Z, Ozkan Sen D, Ucan Yarkac F. Potential use of ChatGPT for patient information in periodontology: a descriptive pilot study. Cureus. 2023;15:e48518.
Not in our library10.7759/cureus.48518
71.
Gordon EB, et al. Enhancing patient communication With Chat-GPT in radiology: evaluating the efficacy and readability of answers to common imaging-related questions. J. Am. Coll. Radiol. 2024;21:353-359.
Not in our library10.1016/j.jacr.2023.09.011
72.
Kuşcu, O., Pamuk, A. E., Sütay Süslü, N. & Hosal, S. Is ChatGPT accurate and reliable in answering questions regarding head and neck cancer? Front. Oncol. 13, 1256459 (2023).
Not in our library10.3389/fonc.2023.1256459
73.
Iannantuono, G. M. et al. Comparison of large language models in answering immuno-oncology questions: a cross-sectional study. Oncologist oyae009 10.1093/oncolo/oyae009 (2024).
Not in our library10.1093/oncolo/oyae009
74.
Zhou Y, Moon C, Szatkowski J, Moore D, Stevens J. Evaluating ChatGPT responses in the context of a 53-year-old male with a femoral neck fracture: a qualitative analysis. Eur. J. Orthop. Surg. Traumatol. Orthop. Traumatol. 2024;34:927-955.
Not in our library10.1007/s00590-023-03742-4
75.
Lahat A, Shachar E, Avidan B, Glicksberg B, Klang E. Evaluating the utility of a large language model in answering common patients’ gastrointestinal health-related questions: are we there yet? Diagnostics. 2023;13:1950.
Not in our library10.3390/diagnostics13111950
76.
Cadamuro J, et al. Potentials and pitfalls of ChatGPT and natural-language artificial intelligence models for the understanding of laboratory medicine test results. An assessment by the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) Working Group on Artificial Intelligence (WG-AI). Clin. Chem. Lab. Med. 2023;61:1158-1166.
Not in our library10.1515/cclm-2023-0355
77.
Nachalon, Y., Broer, M. & Nativ-Zeltzer, N. Using ChatGPT to generate research ideas in dysphagia: a pilot study. Dysphagia10.1007/s00455-023-10623-9 (2023).
Not in our library10.1007/s00455-023-10623-9
78.
Yun JY, Kim DJ, Lee N, Kim EK. A comprehensive evaluation of ChatGPT consultation quality for augmentation mammoplasty: a comparative analysis between plastic surgeons and laypersons. Int. J. Med. Inf. 2023;179:105219.
Not in our library10.1016/j.ijmedinf.2023.105219
79.
Sallam M, et al. ChatGPT output regarding compulsory vaccination and COVID-19 Vaccine conspiracy: a descriptive study at the outset of a paradigm shift in online search for information. Cureus. 2023;15:e35029.
Not in our library10.7759/cureus.35029
80.
Hristidis V, Ruggiano N, Brown EL, Ganta SRR, Stewart S. ChatGPT vs Google for queries related to dementia and other cognitive decline: comparison of results. J. Med. Internet Res. 2023;25:e48966.
Not in our library10.2196/48966
81.
Al-Sharif, E. M. et al. Evaluating the accuracy of ChatGPT and Google BARD in fielding oculoplastic patient queries: a comparative study on artificial versus human intelligence. Ophthal. Plast. Reconstr. Surg. 10.1097/IOP.0000000000002567. (2024)
Not in our library10.1097/iop.0000000000002567
82.
Kaneda Y, Namba M, Kaneda U, Tanimoto T. Artificial intelligence in childcare: assessing the performance and acceptance of ChatGPT responses. Cureus. 2023;15:e44484.
Not in our library10.7759/cureus.44484
83.
Song H, et al. Evaluating the performance of different large language models on health consultation and patient education in urolithiasis. J. Med. Syst. 2023;47:125.
Not in our library10.1007/s10916-023-02021-3
84.
Chee J, Kwa ED, Goh X. ‘Vertigo, likely peripheral’: the dizzying rise of ChatGPT. Eur. Arch. Oto-Rhino-Laryngol. J. Eur. Fed. Oto-Rhino-Laryngol. Soc. EUFOS Affil. Ger. Soc. Oto-Rhino-Laryngol. Head. Neck Surg. 2023;280:4687-4689.
Not in our library10.1007/s00405-023-08135-1
85.
Hillmann, H. A. K. et al. Accuracy and comprehensibility of chat-based artificial intelligence for patient information on atrial fibrillation and cardiac implantable electronic devices. Eur. Eur. Pacing Arrhythm. Card. Electrophysiol. J. Work. Groups Card. Pacing Arrhythm. Card. Cell. Electrophysiol. Eur. Soc. Cardiol. 26, euad369 (2023).
Not in our library10.1093/europace/euad369
86.
Currie G, Robbie S, Tually P. ChatGPT and patient information in nuclear medicine: GPT-3.5 Versus GPT-4. J. Nucl. Med. Technol. 2023;51:307-313.
Not in our library10.2967/jnmt.123.266151
87.
Tie, X. et al. Personalized impression generation for PET reports using large language models. J. Imaging Inform. Med. 10.1007/s10278-024-00985-3 (2024).
Not in our library10.1007/s10278-024-00985-3
88.
Madrid-García A, et al. Harnessing ChatGPT and GPT-4 for evaluating the rheumatology questions of the Spanish access exam to specialized medical training. Sci. Rep. 2023;13:22129.
Not in our library10.1038/s41598-023-49483-6
89.
Cankurtaran RE, Polat YH, Aydemir NG, Umay E, Yurekli OT. Reliability and usefulness of ChatGPT for inflammatory bowel diseases: an analysis for patients and healthcare professionals. Cureus. 2023;15:e46736.
Not in our library10.7759/cureus.46736
90.
Sievert M, et al. Risk stratification of thyroid nodules: assessing the suitability of ChatGPT for text-based analysis. Am. J. Otolaryngol. 2024;45:104144.
Not in our library10.1016/j.amjoto.2023.104144
91.
Gobira M, et al. Performance of ChatGPT-4 in answering questions from the Brazilian National Examination for Medical Degree Revalidation. Rev. Assoc. Med. Bras. 1992. 2023;69:e20230848.
Not in our library10.1590/1806-9282.20230848
92.
Saibene AM, et al. Reliability of large language models in managing odontogenic sinusitis clinical scenarios: a preliminary multidisciplinary evaluation. Eur. Arch. Oto-Rhino-Laryngol. J. Eur. Fed. Oto-Rhino-Laryngol. Soc. EUFOS Affil. Ger. Soc. Oto-Rhino-Laryngol. Head. Neck Surg. 2024;281:1835-1841.
Not in our library10.1007/s00405-023-08372-4
93.
Evaluation of the Performance of Generative AI Large Language Models ChatGPT, Google Bard, and Microsoft Bing Chat in Supporting Evidence-Based Dentistry: Comparative Mixed Methods Study · 2023
No notice
94.
Parasuraman A, Berry LL, Zeithaml VA. Refinement and reassessment of the SERVQUAL scale. J. Retail. 1991;67:420.
Not in our library
95.
Shoemaker, S. J., Wolf, M. S. & Brach, C. Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ. Couns. 96, 395–403 (2014).
Not in our library10.1016/j.pec.2014.05.027
96.
Cheong RCT, et al. Artificial intelligence chatbots as sources of patient education material for obstructive sleep apnoea: ChatGPT versus Google Bard. Eur. Arch. Otorhinolaryngol. 2024;281:985-993.
Not in our library10.1007/s00405-023-08319-9
97.
Biggs, J. B. & Collis, K. F. Evaluating the Quality of Learning: The SOLO Taxonomy (Structure of the Observed Learning Outcome) (Academic Press, 2014).
Not in our library
98.
Sinha, R. K., Roy, A. D., Kumar, N., Mondal, H. & Sinha, R. Applicability of ChatGPT in assisting to solve higher order problems in pathology. Cureus15, e35237 (2023).
Not in our library10.7759/cureus.35237
99.
Wang RY, Strong DM. Beyond accuracy: what data quality means to data consumers. J. Manag. Inf. Syst. 1996;12:5-33.
Not in our library10.1080/07421222.1996.11518099
100.
Riedel, M. et al. ChatGPT’s performance in German OB/GYN exams–paving the way for AI-enhanced medical education and clinical practice. Front. Med. 10, 1296615 (2023).
Not in our library10.3389/fmed.2023.1296615
101.
Sallam, M., Barakat, M. & Sallam, M. METRICS: establishing a preliminary checklist to standardize design and reporting of artificial intelligence-based studies in healthcare. JMIR Prepr. 10, (2023).
Not in our library
102.
Sallam, M., Barakat, M. & Sallam, M. Pilot testing of a tool to standardize the assessment of the quality of health information generated by artificial intelligence-based models. Cureus15, e49373 (2023).
Not in our library10.7759/cureus.49373
103.
Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J. Epidemiol. Community Health. 1999;53:105-111.
Not in our library10.1136/jech.53.2.105
104.
Seth, I. et al. Comparing the efficacy of large language models ChatGPT, BARD, and Bing AI in providing information on rhinoplasty: an observational study. Aesthet. Surg. J. Open Forum.5, ojad084 (2023).
Not in our library10.1093/asjof/ojad084
105.
Mu X, et al. Comparison of large language models in management advice for melanoma: Google’s AI BARD, BingAI and ChatGPT. Ski. Health Dis. 2024;4:e313.
Not in our library10.1002/ski2.313
106.
Xie Y, Seth I, Hunter‐Smith DJ, Rozen WM, Seifman MA. Investigating the impact of innovative AI chatbot on post‐pandemic medical education and clinical assistance: a comprehensive analysis. ANZ J. Surg. 2024;94:68-77.
Not in our library10.1111/ans.18666
107.
Anastasio AT, Mills FB, Karavan MP, Adams SB. Evaluating the quality and usability of artificial intelligence–generated responses to common patient questions in foot and ankle surgery. Foot Ankle Orthop. 2023;8:24730114231209919.
Not in our library10.1177/24730114231209919
108.
Chou R, et al. AHRQ Series Paper 4: assessing harms when comparing medical interventions: AHRQ and the Effective Health-Care Program. J. Clin. Epidemiol. 2010;63:502-512.
Not in our library10.1016/j.jclinepi.2008.06.007

Reads

Read by people on this page

None yet.

Checked or read by AI through naulon

None yet.

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.

Are you an author?

Is this your paper?

Claim it to add your note and be paid when agents cite it. Free.