Towards accurate differential diagnosis with large language models

Abstract

A comprehensive differential diagnosis is a cornerstone of medical care that is often reached through an iterative process of interpretation that combines clinical history, physical examination, investigations and procedures. Interactive interfaces powered by large language models present new opportunities to assist and automate aspects of this process 1 . Here we introduce the Articulate Medical Intelligence Explorer (AMIE), a large language model that is optimized for diagnostic reasoning, and evaluate its ability to generate a differential diagnosis alone or as an aid to clinicians. Twenty clinicians evaluated 302 challenging, real-world medical cases sourced from published case reports. Each case report was read by two clinicians, who were randomized to one of two assistive conditions: assistance from search engines and standard medical resources; or assistance from AMIE in addition to these tools. All clinicians provided a baseline, unassisted differential diagnosis prior to using the respective assistive tools. AMIE exhibited standalone performance that exceeded that of unassisted clinicians (top-10 accuracy 59.1% versus 33.6%, P = 0.04). Comparing the two assisted study arms, the differential diagnosis quality score was higher for clinicians assisted by AMIE (top-10 accuracy 51.7%) compared with clinicians without its assistance (36.1%; McNemar’s test: 45.7, P < 0.01) and clinicians with search (44.4%; McNemar’s test: 4.75, P = 0.03). Further, clinicians assisted by AMIE arrived at more comprehensive differential lists than those without assistance from AMIE. Our study suggests that AMIE has potential to improve clinicians’ diagnostic reasoning and accuracy in challenging cases, meriting further real-world evaluation for its ability to empower physicians and widen patients’ access to specialist-level expertise.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 37

1. 1.

   Kanjee Z, Crowe B, Rodman A. Accuracy of a generative artificial intelligence model in a complex diagnostic challenge. JAMA. 2023;330:78-80.

   Not in our library10.1001/jama.2023.8288

2. 2.

   Szolovits P, Pauker SG. Categorical and probabilistic reasoning in medical diagnosis. Artif. Intell. 1978;11:115-144.

   Not in our library10.1016/0004-3702(78)90014-0

3. 3.

   Liu Y, et al. A deep learning system for differential diagnosis of skin diseases. Nat. Med. 2020;26:900-908.

   Not in our library10.1038/s41591-020-0842-3

4. 4.

   Rauschecker AM, et al. Artificial intelligence system approaching neuroradiologist-level differential diagnosis accuracy at brain MRI. Radiology. 2020;295:626-637.

   Not in our library10.1148/radiol.2020190283

5. 5.

   Balas M, Ing EB. Conversational AI models for ophthalmic diagnosis: comparison of ChatGPT and the Isabel pro differential diagnosis generator. JFO Op. Ophthalmol. 2023;1:100005.

   Not in our library10.1016/j.jfop.2023.100005

6. 6.

   GPT-4 Technical Report 2303.08774 (OpenAI, 2023).

   Not in our library

7. 7.

   Anil, R. et al. Palm 2 technical report. Preprint at 10.48550/arXiv.2305.10403 (2023).

   Not in our library

8. 8.

   Scao, T. L. et al. BLOOM: a 176b-parameter open-access multilingual language model. Preprint at 10.48550/arXiv.2211.05100 (2022).

   Not in our library

9. 9.

   Touvron, H. et al. Llama 2: open foundation and fine-tuned chat models. Preprint at 10.48550/arXiv.2307.09288 (2023).

   Not in our library

10. 10.
    Large language models encode clinical knowledge · 2023

    Corrected
11. 11.

    Nori, H., King, N., McKinney, S. M., Carignan, D. & Horvitz, E. Capabilities of GPT-4 on medical challenge problems. Preprint at 10.48550/arXiv.2303.13375 (2023).

    Not in our library

12. 12.

    Eriksen, A. V., Moller, S. & Ryg, J. Use of GPT-4 to diagnose complex clinical cases. NEJM AI10.1056/AIp2300031 (2023).

    Not in our library

13. 13.

    Buckley, T., Diao, J. A., Rajpurkar, P., Rodman, A. & Manrai, A. K. Multimodal foundation models exploit text to make medical image predictions. Preprint at 10.48550/arXiv.2311.05591 (2024).

    Not in our library10.1038/s41467-026-74207-5

14. 14.
    Towards conversational diagnostic artificial intelligence · 2025

    No notice
15. 15.

    Bond WF, et al. Differential diagnosis generators: an evaluation of currently available computer programs. J. Gen. Intern. Med. 2012;27:213-219.

    Not in our library10.1007/s11606-011-1804-8

16. 16.

    Ledley RS, Lusted LB. Reasoning foundations of medical diagnosis: symbolic logic, probability, and value theory aid our understanding of how physicians reason. Science. 1959;130:9-21.

    Not in our library10.1126/science.130.3366.9

17. 17.

    Hong, J., Levine, S. & Dragan, A. Zero-shot goal-directed dialogue via RL on imagined conversations. In Proc. Foundation Models for Decision Making Workshop at NeurIPS 2023 (eds Yang, S. et al.) 77 (NeurIPS, 2023).

    Not in our library

18. 18.

    Kossen, J. et al. Active acquisition for multimodal temporal data: a challenging decision-making task. Preprint at 10.48550/arXiv.2211.05039 (2023).

    Not in our library

19. 19.

    Mackie, I., Chatterjee, S. & Dalton, J. Generative relevance feedback with large language models. In Proc. 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (eds Chen, H.-H. et al.) 2026–2031 (Association for Computing Machinery, 2023).

    Not in our library

20. 20.

    Fritz P, et al. Evaluation of medical decision support systems (ddx generators) using real medical cases of varying complexity and origin. BMC Med. Inform. Decis. Mak. 2022;22:254.

    Not in our library10.1186/s12911-022-01988-2

21. 21.

    Miller, R. A., Pople Jr, H. E. & Myers, J. D. Internist-i, an experimental computer-based diagnostic consultant for general internal medicine. In Computer-assisted medical decision making (eds Reggia, J. A. & Tuhrim, S.) 139–158 (Springer, 1985).

    Not in our library

22. 22.

    Li, C. et al. Llava-med: training a large language-and-vision assistant for biomedicine in one day. In Proc. Advances in Neural Information Processing Systems 36 (eds Oh, A. et al.) 28541–28564 (NeurIPS, 2023).

    Not in our library

23. 23.

    Tu, T. et al. Towards generalist biomedical AI. NEJM AI1, AIoa2300138 (2024).

    Not in our library

24. 24.

    Muehlematter UJ, Daniore P, Vokinger KN. Approval of artificial intelligence and machine learning-based medical devices in the USA and europe (2015–20): a comparative analysis. Lancet Digit. Health. 2021;3:e195-e203.

    Not in our library10.1016/s2589-7500(20)30292-2

25. 25.

    Roschewitz M, et al. Automatic correction of performance drift under acquisition shift in medical image classification. Nat. Commun. 2023;14:6608.

    Not in our library10.1038/s41467-023-42396-y

26. 26.

    Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366:447-453.

    Not in our library10.1126/science.aax2342

27. 27.
    Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations · 2021

    No notice
28. 28.

    Samorani M, Harris SL, Blount LG, Lu H, Santoro MA. Overbooked and overlooked: machine learning and racial bias in medical appointment scheduling. Manuf. Serv. Oper. Manag. 2022;24:2825-2842.

    Not in our library10.1287/msom.2021.0999

29. 29.
    Do as AI say: susceptibility in deployment of clinical decision-aids · 2021

    No notice
30. 30.

    Umapathi, L. K., Pal, A. & Sankarasubbu, M. Med-HALT: medical domain hallucination test for large language models. In Proc. 27th Conference on Computational Natural Language Learning (CoNLL) (eds Jiang, J. et al.) 314–334 (Association for Computational Linguistics, 2023).

    Not in our library

31. 31.

    Sikka R, Morath JM, Leape L. The quadruple aim: care, health, cost and meaning in work. BMJ Qual. Saf. 2015;24:608-610.

    Not in our library10.1136/bmjqs-2015-004160

32. 32.

    Cai CJ, Winter S, Steiner D, Wilcox L, Terry M. ‘Hello AI’: uncovering the onboarding needs of medical practitioners for human-ai collaborative decision-making. Proc. ACM Hum. Comput. Interact. 2019;3:1-24.

    Not in our library10.1145/3359206

33. 33.

    Yin, Z. et al. Do large language models know what they don’t know? In Proc. Findings of the Association for Computational Linguistics: EACL 2023 (eds Vlachos, A. & Augenstein, I.) 8653–8665 (Association for Computational Linguistics, 2023).

    Not in our library

34. 34.

    Dvijotham K, et al. Enhancing the reliability and accuracy of ai-enabled diagnosis via complementarity-driven deferral to clinicians. Nat. Med. 2023;29:1814-1820.

    Not in our library10.1038/s41591-023-02437-x

35. 35.
    MIMIC-III, a freely accessible critical care database · 2016

    No notice
36. 36.

    Jin D, et al. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Appl. Sci. 2021;11:6421.

    Not in our library10.3390/app11146421

37. 37.

    Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-174.

    Not in our library10.2307/2529310

Cited by papers we hold 3

• Towards conversational diagnostic artificial intelligence · 2025
  No notice
• Large Language Model Influence on Diagnostic Reasoning · 2024
  No notice
• Evaluation and mitigation of the limitations of large language models in clinical decision-making · 2024
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.