Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information

1Patients are increasingly turning to the internet for information about cancer, with 80% of US adults reportedly using the internet to seek health information. Within the online communication environment, cancer misinformation and harmful information remain a serious concern. Chatbots and other artificial intelligence (AI) systems have become increasingly popular in recent years for providing information and assistance to users in online spaces; recently, 1 such AI system receiving a lot of public and news attention is ChatGPT, which uses natural language processing to generate responses to user input. Given the importance of accurate information in the field of cancer research and treatment, determining the accuracy of AI (mis)information outputs from chat platforms such as ChatGPT is critical to clinicians and, more broadly, health and medical communicators. As an initial step in monitoring these public platforms, we engaged in a small-scale study of the information ChatGPT generated.

2Using the National Cancer Institute’s (NCI’s) web page “Common Cancer Myths and Misconceptions”, on December 20, 2022, we asked ChatGPT (December 15, 2022, version) 13 questions about cancer that are common points of confusion among the public (per NCI). The NCI’s answers and ChatGPT answers to each question were blinded, and then evaluated for accuracy based on the established knowledge of 5 scientific reviewers with expertise in cancer treatment and cancer misinformation (B.K., C.B., A.K., E.W., and S.A.) between December 20, 2022, and January 3, 2023. Expert reviewers were not compensated. Accuracy ratings (yes vs no) were evaluated independently for each question, and then compared between the blinded NCI and ChatGPT answers. Interrater agreement was evaluated by Fleiss ĸ. Statistical analyses were performed using Stata, version 17 (StataCorp LP).

3Following expert review, 13 of 13 NCI answers were rated as accurate by 5 of 5 expert reviewers, demonstrating 100% interrater agreement. ChatGPT outputs were determined to be accurate by 5 of 5 reviewers for 11 of 13 questions, demonstrating 96.9% overall agreement. The remaining 2 questions (question 5, “Does my attitude—positive or negative—determine my risk of getting or surviving cancer?” and question 9, “Do power lines cause cancer?”) were rated as accurate by 4 of 5 reviewers, with a single unique reviewer rating each question as inaccurate (ĸ = ‒0.03, SE = 0.08, P =.6). Table 1 displays all questions and associated answers from NCI and ChatGPT, respectively, as well as word count and Flesch-Kincaid readability grade level for each individual response. There were few noticeable differences in the number of words or the readability of the answers from NCI or ChatGPT. Answers from both sources were written at a less readable rate than would likely be recommended by health literacy advocates, for example. Although our team did not code for specific message features, multiple observers noted their perception that the ChatGPT messages often used terms associated with hedging and uncertainty that could lead to the adoption of a belief or practice that could be harmful. Overall, the results suggest that output from ChatGPT provides accurate information about common cancer myths and misconceptions without providing misinformation or harmful information to users. To evaluate whether repeated questioning could result in outputs with semantic variations that could lead to misinformation, we completed 5 runs repeatedly from the same input for each of the 13 questions (Supplementary Table 1, available online). On review, although subtle sentence structure changes were noted, the answers remained consistent, suggesting that answers remain accurate with repetitive questioning.

4This brief report offers important insights into the potentially positive capabilities of ChatGPT and other AI systems in the context of cancer-related (mis)information. As a team, we were initially uncertain about the utility of ChatGPT for health information and thought that the AI system could unintentionally convey misinformation or harmful information to users. This possibility is still important to consider for any AI chat system, given that users can problematically shape chatbot responses in a short period; researchers have found that AI systems can “enact malignant stereotypes”; and, specific to ChatGPT, there is reasonable skepticism about the tool’s utility when considering a breadth of topics beyond the basic cancer information we asked about in our study. ChatGPT could be the future of AI chat technology (or perhaps the tool is just a popular example of the state of the technology’s capabilities); regardless of this particular tool’s features, however, we must consider how best to monitor and evaluate the use of these tools in the online communication environment. Future systematic work to evaluate use and outputs of such platforms requires an infrastructure to monitor these sites, their algorithms and information sources, and potential bias in providing information equitably to diverse populations. There is concern that algorithms reinforce current health disparities and inequities, although the extent to which this is true about health information specifically is currently unknown. Other practical considerations about how people will interpret AI-generated chat responses in terms of trust and credibility exist, particularly as they relate to medical and health information, behaviors, and services compared with consumer products, behaviors, and services.

5The limitations of this study are that we evaluated only common cancer misinformation in the English language. More obscure cancer myths may have incomplete or English-only information, resulting in the model’s inability to be trained on a sufficiently large and diverse data set to answer fewer common questions or non–English-language questions accurately. Finally, although this study obtained contemporary ChatGPT outputs, there is a possibility that these outputs were trained on models that could be scientifically outdated. Currently, ChatGPT is limited to data collected before 2021, so it is possible that as new scientific information emerges, ChatGPT may not be an accurate source of information or, at the least, its accuracy will be delayed.

6Collectively, we found ChatGPT outputs in response to common cancer misinformation to be accurate and similar to answers that the NCI provided. In the context of these questions, there does not appear to be a clear area where this system may be susceptible to misinformation. Future research is needed to determine whether other chatbots and AI-driven systems provide accurate cancer information consistently, whether these findings apply to more diverse claims about cancer across the continuum, and what the ideal infrastructure might be for future monitoring to ensure accuracy of cancer information within the online information ecosystem.