Artificial Intelligence Education and Tools for Medical and Health Informatics Students: Systematic Review

1Artificial intelligence (AI) is one of the most disruptive innovations in health care, and the topic has attracted the attention of physicians, clinicians, researchers, and medical device industry professionals. Recent advancements in machine learning (ML) and deep learning (DL) algorithms and cloud computing have increased the adoption of AI. Consequently, applications that can handle a large number of unstructured data sets and solve complex problems have become a part of daily clinical practice.

2Most AI applications process data and run self-learning algorithms behind the scenes. Although some AI applications provide data-driven recommendations to clinicians, others may not offer an option to accept, reject, or modify the output. The recommendations AI applications provide through statistical correlations may not be the best option because human-made AI algorithms may be flawed. To use and screen AI-based decisions, clinicians and health informaticians who develop AI applications should have an excellent understanding of the underlying AI concepts. This paper will focus on the emerging need for formal AI education in medicine and health informatics.

3Intelligence requires the capacity to perceive contexts, associate contexts to actions, and act. Even though the concept of machines that imitate intelligent human behavior is not new, AI has recently become a topic of interest. As an academic discipline, the Dartmouth College Artificial Intelligence Conference that was organized by John McCarthy in 1956 was considered the birth of this field.

4AI, ML, and DL are closely related, and the absence of universal definitions might be confusing; however, the difference between AI, ML, and DL is simple. AI is defined as “the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages”. AI-based devices can perceive the environment, simulate human intelligence, and solve problems. Their ability to adapt through progressive learning algorithms is what differentiates AI technologies from robotic and hardware-driven automation. In other words, computers can mimic human intelligence using AI techniques. ML is the subset of AI that allows systems to learn from data and develop self-learning algorithms. ML applications can learn from data without being explicitly programmed; make predictions and recommendations using various tools; and enable computer applications to improve their performance. DL is a subfield within ML that allows machines to use algorithms inspired by the structure of neural networks. A computer can learn how to classify images and how to assign labels to words in a sentence (semantic labeling) by using DL algorithms. DL programming uses large quantities of unstructured data, calculates complex statistical models, and predicts outcomes without being explicitly programmed. Virtual assistants, chatbots, and facial recognition algorithms are some other practical examples of DL.

5The main purpose of this study was to investigate peer-reviewed publications focused on AI education and to determine objective assessment methods for AI skills and competency training for medical and health informatics professionals. The impact of AI on the learning experience was evaluated to assess the need for AI education. As medical education, clinical informatics education, and health informatics education are closely related, medical and health informatics education trends were analyzed together. The American Medical Informatics Association has been working closely with the Commission on Accreditation for Health Informatics and Information Management Education to determine health informatics competencies; clinical informatics became a medical subspecialty in 2011. Although clinical and health informatics programs are designed for students who plan to pursue different career pathways, they use similar competencies. The implementation of ML in health care could result in unintended challenges and biased decisions, depending on the algorithms, data sources, and methodologies used. Physicians who are not familiar with the evidence standards for AI might not be able to use the right approaches to integrate AI into clinical care. Even though there are several studies that explored how AI algorithms were helping enhance education, the number of peer-reviewed publications that focused on artificial intelligence education in medicine is limited (Figure 1).

6Using a replicable systematic search strategy, a full-text review was performed between November 2019 and February 2020. PRISMA-P (Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols) systematic review methodology introduced by Moher et al was used to identify and analyze reliable literature. The PRISMA-P method uses a structured procedure that consists of a 17-item checklist to facilitate systematic review protocols.

7The combination of five groups of keywords was used to search PubMed, IEEE (Institute of Electrical and Electronics Engineers) Xplore Digital Library, CINAHL (Cumulative Index to Nursing and Allied Health Literature) Plus, and ScienceDirect databases: (1) medical education, (2) medical training, (3) artificial intelligence, (4) machine learning, and (5) deep learning (Figure 2 and Table 1). Overall, 2082 articles matched the search criteria. After removing duplicate studies and performing an abstract review, 76 full-text articles were selected for the review. All search results were entered into EPPI-Reviewer 4 text mining software (the EPPI-Centre, University of London), and the studies that met the inclusion criteria were identified. Two researchers performed the extraction independently and assessed quality.

8Based on the following inclusion and exclusion criteria, the selection process was applied. Peer-reviewed research articles, review papers, conference papers, case reports, correspondences, discussions, viewpoint papers, editorials, mini-reviews, and short communications papers that focused on AI tools to enhance the learning experience in medical and health informatics education or teach AI as a new competency published after 1990 were included (Table 1). Book chapters, news, and extended abstracts published before 1990 in languages other than English were excluded. To establish validity, disagreements were discussed until a consensus was reached. Overall, 26 articles matched the inclusion criteria of this research (Figure 3).

9One of the goals of this systematic literature review was to evaluate existing studies and determine the current state of AI education. The selected papers were used to identify the answers to three research questions.

10The first research question is the following: what topics are discussed in peer-reviewed publications that focus on medical and health informatics education and AI? To answer this question, the selected publications were classified based on their education foci and the characteristics of included studies were summarized in Table 2. The publications that focused on the use of AI applications in medical and health informatics education were categorized as Category 1. These studies used various AI-based tools to enhance the learning experience. Several case studies and new initiatives to teach specific AI skills, such as ML programming languages and big data analytics software, were also identified. The publications that evaluated AI education were classified as Category 2. Category 1 studies discussed different AI applications to enhance education and summarized the impact of AI on medical and health informatics education, while Category 2 studies focused on the teaching of AI concepts.

11Of 26 publications, 16 (61%) investigated the use of AI applications in medical education (Category 1) and 10 publications (39%) evaluated AI education in medicine (Category 2; Figure 4).

12The first publications about the use of AI for medical applications were published in the early 1990s, and the capabilities of AI applications were restricted by technological limitations at that time. Klar and Bayer's paper, published in 1990, was one of the first publications about the application of AI, and they discussed the integration of expert knowledge into computer-assisted teaching in medicine. Lillehaug and Lajoie proposed greater integration of AI in their 1998 paper, and they were among the first researchers who advocated for intelligent decision support systems and AI-based applications for medical education. Frize and Frasson examined the role of decision support and intelligent tutoring systems in medical education, and recommended multidisciplinary studies in 2000.

13Most Category 1 studies explored innovative applications designed to improve the learning experience. For example, Chan and Zary evaluated existing AI applications in medicine, determined that the primary reason to use AI in medical education was to provide feedback, and identified that significant challenges included the assessment of effectiveness and management of technical difficulties. Another systematic review conducted by Chary et al identified 30 articles that assessed the application of natural language processing (NLP) to medical education. NLP is a subfield of AI and refers to intelligent communication methods using natural languages. The study revealed the benefits of NLP training in residency education and recommended strategies for its application.

14Simulation-based learning has evolved over the last decade, and the virtual environment has become essential for education. Our review identified multiple case studies about advanced virtual environments. For instance, Winkler-Schwartz et al analyzed virtual reality simulators that use AI, and developed a checklist to assess studies using ML algorithms to evaluate technical skills. This study concluded that the checklist had the potential to decrease the knowledge gaps in the use of AI in surgical education. Similarly, Zhao et al concluded that virtual patient systems and other distance education systems that used AI increased the efficiency of medical education. Another case study that focused on virtual patient simulators identified educational and technical challenges to enhancing the learning process with AI virtual reality applications. The case study published by Bowyer et al described the role of advanced virtual environments in surgical training. Moreover, Conde et al recommended the use of AI applications for training and education when simulated human patients were not an option.

15Augmented reality is another form of virtual reality, and this technology superimposes images on top of the video viewer. Sitterding et al described the differences between virtual reality, augmented reality, serious gaming, and gamification, and shared the preliminary findings of their pilot study, which determined that the simulation experience was similar to a real-life environment.

16In our research, we noted different innovative AI applications for special learning activities. In their 2017 article, Guimarães et al reviewed current education models for anatomy education, and recommended the use of AI analytic tools to personalize the learning process. Boulet and Durning discussed the application of AI to replace human ratings and assess medical education competencies. Another study that used augmented reality and AI algorithms evaluated a case study about the development of a laparoscopic box trainer to assess surgical psychomotor skills, and concluded the proposed system had potential benefits.

17Incorporating intelligent techniques into an adaptive e-learning system was another research group's focus. Kabassi et al designed a web-based educational system for medical education, incorporated intelligent algorithms to individualize the learning experience, and shared the findings. The authors recommended further studies with more participants. Yang and Shulruf designed a pilot study to demonstrate the value of AI-assisted tutoring, and determined that this additional tutoring significantly enhanced the performance of medical students.

18Our second research question involved determining the highest level of evidence of current research. The articles were classified according to the Oxford Centre for Evidence-Based Medicine (OCEBM) Levels of Evidence ranking scheme. Of 26 articles, 18 (69%) were classified as Level IV evidence (case series), and 8 (31%) were classified as Level V evidence (expert opinions). The majority of the publications had Level IV and V evidence, which are considered poor reference standards. Hence, these findings emphasize the need to design new research studies.

19Finally, our third research question was the following: what is the status of AI education in medicine and health informatics? In our study, 10 articles discussing AI education were identified. Since this topic is a relatively new area in medical and health informatics education, our findings were consistent with recent developments.

20In their review, Paranjape et al summarized multiple initiatives for AI in medical education, in which students worked with data experts and solved health care problems. The authors recommended familiarizing students with AI-based clinical applications, and introducing linear algebra, calculus, and probability during different stages of medical education. Similarly, Chin-Yee and Upshur discussed the random error and biased data generated by AI and ML applications, and emphasized the effect of medical education for appraising clinical judgments.

21A review conducted by Park et al emphasized the importance of understanding AI to be able to validate the clinical accuracy of AI algorithms. Furthermore, multiple publications stated the need to move beyond traditional medical education, suggested a reform to align education with new practice requirements, and emphasized the role of academics and teachers in the development of appropriate AI application skills.

22As discussed earlier, ML is an AI technique to process massive amounts of data and make predictions using computers. Kolachalama and Garg proposed the integration of ML-related content in medical student, resident, and fellow education. The authors recommended the integration of real-world clinical examples into ML courses as well as practical guidelines for choosing the right tools.

23The radiology community embraced AI and ML long before other medical specialties, and pioneered the usage of AI algorithms in advanced imaging applications. A multicenter survey of undergraduate medical students determined students' optimistic views about the implications of AI applications for radiology. The French Radiology Community developed principles to regulate the use of AI tools, and recommended specific education to evaluate AI technologies. Similarly, the Canadian Association of Radiologists recommended the integration of computer science, health informatics, and statistics training during residency education.

24The technological advancements in computer and software technologies; digitization of health care data; and methodological developments in information science, philosophy, mathematics, linguistics, and psychology disciplines accelerated medical research programs that focus on ML and commercialization. Maturation of AI technologies changed the roles of clinicians, and novel decision-making processes in medical settings and innovative AI-based protocols have the potential to provide diagnostic and treatment decisions by analyzing complex data sets.

25Over the last decade, many AI researchers have concentrated on developing a proof of concept system for clinicians and patients. There is an increase in the number of studies that evaluate the effectiveness of intelligent reasoning. Intelligent monitoring technologies require new algorithms to detect anomalies, predict patterns, and make decisions.

26A recent independent report prepared for the UK Secretary of State for Health and Social Care explored how the health care workforce could be prepared to use digital technology. The report emphasized the skills gap in the workforce, and made some recommendations about the integration of digital health care technologies, AI, and data analytics in undergraduate curricula.

27Clinicians should have a realistic view of AI, and become familiar with the right tasks for AI in health care. Formal training for medical and health informatics students should enable them to develop AI algorithms, use AI technologies in a competent manner, and keep bias out of AI tools. Any new AI technology might encounter something new for which it has no experience, and therefore the physician should be able to assess problematic decisions and take necessary precautions when needed. Recent discussions about the need for medical education reform emphasize the shortcomings of the current model of education. Overreliance on ML and AI technologies might have unintended severe adverse consequences, such as failure to recognize invalid test results.

28While machine learning and AI algorithms are able to handle high-dimensional data classification problems and medical image interpretation, their success rates in risk prediction and diagnosis are lower. Consequently, there is a need to determine the most appropriate application areas for AI in health care.

29There are different ways to implement AI in clinical practice, and clinicians and health informaticians need formal training to use the right approaches. Health informaticians and physician champions who design and develop AI-based protocols need to have a good understanding of complex algorithms, methodologies for data quality assessment, probabilistic forecasting, and comparative model assessment to work with engineers and develop reliable AI applications. Moreover, clinicians who use AI applications should become familiar with potential challenges.

30AI-based relational time pattern analysis replaced simple threshold-based diagnostic rules. Current medical education and health informatics curricula still do not provide the ability to understand AI communications and necessary skill sets to develop AI systems that can detect and analyze relational time patterns.

31The ability to interpret AI algorithms' mistakes and formulate the best strategies to correct these applications requires specialized training. Consequently, medical and health informatics education must emphasize algorithm-based platforms, and include relevant data analytics and AI topics in their curricula. Moreover, computer science and health informatics programs should consist of health care–focused digital skills training.

32To the best of our knowledge, no previous research has investigated the use of AI tools to enhance the learning experience and AI education of medical and health informatics students. The main findings of this systematic review are as follows: (1) Although there are several recommendations on the integration of AI into medical and health informatics curricula and some academic institutions implemented experimental training programs, AI and ML education are not a part of traditional medical and health informatics curricula yet. (2) Current medical education and health informatics accreditation standards do not require AI training, and AI competencies have not been determined. (3) Using the OCEBM Levels of Evidence classification table, the majority of studies were classified as Level IV and V, which indicates poor reference standards.

33Several efforts were made to design an optimal systematic review process; however, there were still many limitations. It is probable that some studies might not be listed in the peer-reviewed academic literature databases or might be published in a non-English language. Although this is a systematic review of the field, AI is a new technical discipline, particularly in medicine, and therefore the number of articles that met the inclusion criteria was limited.

34Overall, the selected publications did not provide specific details about different jobs' requirements and curriculum needs. The emergence of intelligent systems in health care requires new learning modalities. Even though several organizations, agencies, and work groups such as the International Medical Informatics Association, the Commission on Accreditation for Health Informatics and Information Management Education, the Health Informatics Society of Australia, the TIGER (Technology Informatics Guiding Education Reform) Initiative, and the Association of American Medical Colleges published skill recommendations for health informatics curricula, they have not determined specific skill sets for AI education. Furthermore, a recent study evaluated health informatics students' skills in developing AI apps and emphasized the need to develop new competencies.

35A specialized AI education framework for various professional fields would be useful; using the results of this systematic review, we propose a framework for specialized AI training for different domains (Figure 5). Medical students need to become familiar with clinical AI applications and predictive modeling techniques to assess biased data and evaluate innovative AI technologies. Health informatics students should become familiar with the application of appropriate ML algorithms and development of innovative clinical informatics systems. Furthermore, they should gain the hands-on skills required to extract data, manage large data sets to perform sophisticated data analytics, and develop innovative AI systems. Computer science students need specialized skill sets to work with data scientists and should become familiar with Python, R, and SQL programming languages, and data analytics tools (Figure 5).