Serious Gaming and Gamification Education in Health Professions: Systematic Review

1Innovative approaches and modalities for education in health professions education are constantly sought to improve teaching and learning and ultimately patient care and outcomes. Digital education may be one such innovation. This review focuses on serious gaming and gamification education.

2For the purposes of this review, we have used the terminology defined by Alvarez. The term “serious game” was used to refer only to games designed specifically for the “serious” purpose of providing health professions education via a digital device. The term “serious diverting” was used to refer to the use of games originally designed primarily for entertainment used without modification, as part of health professions education delivered via a digital device. “Serious gaming” was used to refer to any use of digital games for health professions education, thereby encompassing “serious games” and “serious diverting.”

3A related but separate concept—“gamification”—can be defined as “the application of the characteristics and benefits of games to real world processes or problems”. Gamification differs from serious games in terms of the design intention, with gamification interventions involving the application of game elements with a utilitarian purpose and serious games designed as full-fledged games for a purpose other than just entertainment. Wortley suggests that both may be experienced by the user as a complete game, although typically, gamification involves the use of game components outside a game setting, such as rewarding users completing an electronic learning (e-learning) module with badges or points. Gamification has the potential to allow for greater involvement of the user in setting his/her own objectives or outcomes, personalization of the intervention, and cost-effectiveness. Most, but not all, uses of the term refer to interventions involving the use of enabling digital technologies.

4Serious gaming/gamification has the potential to provide learners with opportunities to be part of active learning, solving clinical problems, and gaining experience in risk-free surroundings, without needing to involve patients. Learners may have the opportunity to develop analytical skills, strategic thinking, knowledge, multitasking, decision making, communication, and psychomotor skills, with multiplayer functions providing opportunities for collaborative learning. The motivational properties of gaming have the potential to be harnessed for educational purposes. Serious gaming/gamification can be used at a time and place that suits the learner. The reusable nature of serious gaming/gamification may allow more frequent or longer interactions, free up lecturer time, and provide monetary savings. However, this could lead to reduced opportunities to ask questions, hold discussions, and spend time with patients. Use of such interventions within small groups, with lecturer support, could allow for discussion and interaction but would likely increase lecturer time needed as compared to traditional learning. Serious gaming/gamification, like other kinds of e-learning, may ease the process of updating materials, as modifications to content can be made continuously, unlike with a text book.

5Although serious gaming and gamification interventions appear to have much potential, rigorous evaluation is required to assess whether they can lead to effective learning. There is a potential for the game or game elements to become a distraction rather than a facilitator of learning, with the method “more memorable than the message”; therefore, the quality of learning must be the focus, as opposed to the capabilities of the technology used.

6This systematic review is one of a series of reviews evaluating the scope for implementation and potential impact of a wide range of digital health education interventions for pre- and postregistration health professions. This review was conducted in collaboration with the World Health Organization’s Health Workforce Department. The objective of this work is to compare the effectiveness of serious gaming and gamification education versus various controls in improving learners’ knowledge, skills, professional attitudes, and satisfaction as well as patient outcomes.

7While conducting and reporting the review, we adhered to the gold-standard systematic review methods recommended by the Cochrane Collaboration. For a detailed description of the methodology, please refer to our previous paper.

8We comprehensively searched the following databases between 1990 and August 2017: MEDLINE (Ovid), EMBASE (Elsevier), Web of Science, Educational Resource Information Centre (Ovid), Cochrane Central Register of Controlled Trials (CENTRAL), (The Cochrane Library), PsycINFO (Ovid), and Cumulative Index to Nursing and Allied Health Literature (EBSCO). The search strategy for MEDLINE is presented in the Multimedia Appendix 1. We searched for papers in English but considered eligible studies in any language. We also searched two trials registries, reference lists of all included studies, and relevant systematic reviews and contacted the relevant investigators for further information.

9We included randomized controlled trials (RCTs) and cluster RCTs (cRCTs) of pre- and postregistration health professions using serious gaming/gamification with any type of controls (traditional learning, digital education, or another type of serious gaming/gamification intervention), which measured patient outcomes, knowledge, skills (cognitive and psychomotor), professional attitudes, and satisfaction (primary outcomes) and adverse effects or costs (secondary outcomes). We excluded crossover trials due to the high likelihood of carry-over effect. Participants were not excluded on the basis of sociodemographic characteristics such as age, gender, ethnicity, or any other related characteristics. Outcome definitions are available in the associated paper.

10The search results from different electronic databases were combined in a single EndNote library (X 8.2; Clarivate Analytics, Philadelphia, PA), and duplicate records were removed. Two reviewers independently screened titles and abstracts to identify studies that potentially met the inclusion criteria. The full texts of these articles were retrieved and read. Two review authors independently assessed these articles against the eligibility criteria (SG, AG, and BE). At least two reviewers independently extracted the data for each of the included studies using a structured data-extraction form. We extracted all relevant data on the characteristics of participants, interventions, controls, and outcomes measures. For continuous data, we reported standardized mean differences and SDs. None of the studies reported dichotomous data. Disagreements were resolved through discussion.

11Two reviewers independently assessed the risk of bias of the included studies using the Cochrane Collaboration’s “Risk of bias” tool. Studies were assessed for the risk of bias in the following domains: random sequence generation, allocation concealment, blinding (participants and personnel), blinding (outcome assessment), completeness of outcome data (attrition bias), selective outcome reporting (reporting bias), and other risk of bias. For cRCTs, we also assessed recruitment bias, baseline imbalances, loss of clusters, and incorrect analysis. Judgements concerning the risk of bias for each study were classified as high, low, or unclear.

12Data were synthesized using Review Manager (Version 5.3; The Nordic Cochrane Centre, The Cochrane Collaboration, Copenhagen, Denmark). Included studies were insufficiently homogenous in terms of population, inclusion criteria, interventions, and outcomes for meta-analysis. The decision not to perform a meta-analysis was made by a consensus of review authors. We present a narrative synthesis of findings, with effect sizes calculated for outcomes where there were sufficient data. Where possible, we assessed the quality of studies and size of effect. Results are presented by outcome and separately for each comparison (serious gaming/gamification vs traditional learning, serious gaming/gamification vs digital health education, and serious gaming/gamification vs serious gaming/gamification).

13The results for comparisons between serious gaming/gamification and traditional learning as well as serious gaming/gamification and digital education are presented in the narrative summary of findings tables (Tables 7 and 8). Two authors (SG and AG) rated the overall quality of the evidence as implemented and described in GRADEprofiler (GRADEproGDT online version; Evidence Prime, Inc, Hamilton, ON, Canada) and chapter 11 of the Cochrane Handbook for Systematic Reviews of Interventions. We considered the following criteria to assess the quality of the evidence: limitations of studies (risk of bias), inconsistency of results, indirectness of the evidence, imprecision, and publication bias. We also downgraded the quality, where appropriate. This was done for all primary outcomes reported in the review.

14Our searches yielded a total of 30,532 citations and 30 studies (27 RCTs and 3 cRCTs) including 3634 participants (Figure 1).

15Sample sizes ranged from 14 to 1470 participants. Almost half the included studies had sample sizes below 50.

16Fourteen studies were conducted in Europe, and 11 studies were conducted in North America, one of which recruited participants from 63 countries via the internet. One study was conducted in Singapore. Four studies were conducted in middle-income countries, three of which were conducted in Brazil. One study was conducted by authors based in China and Taiwan, but it was unclear where the study itself was carried out. None of the included studies were conducted in low-income countries. Details of study designs and population for each trial are compared in Tables 1-3 and a summary is given below.

17Eleven studies included only medical students. Five studies included only nursing students and four included only surgical residents. The remaining studies included primary care doctors (n=2); dental students (n=2); anesthesiology residents (n=1); urologists (n=1); speech and language science students (n=1); participants of the Major Incident Medical Management and Support course, which typically includes doctors, nurses, and paramedics with an interest in prehospital care (n=1); nursing and medical students (n=1); and medical students, residents, and specialists in Obstetrics and Gynecology (n=1, reported in one article and one conference abstract).

18Characteristics of the interventions included are compared in Tables 4-6 and a summary is given below.

19Two of the included studies evaluated “gamification” interventions. The remainder evaluated serious gaming interventions. Two of these were group interventions, in which a serious gaming intervention was projected to a traditional classroom of learners who played together.

20A total of 22 interventions had original design purposes other than entertainment, of which 11 interventions were classified as “Message Broadcasting - Educative”; four were classified as “Training - Mental”, three were classified as “Training - Physical”, and four were classified as both “Training - Mental” and “Training - Physical”. All of the interventions with design purposes other than entertainment were classified under “Education” for “Scope.”

21The remaining eight interventions were commercial off-the-shelf games designed only for the purpose of entertainment but used for training of motor skills as part of “Serious Diverting” interventions. These were all classified as “Training - Physical” for “Purpose” and as “Entertainment” and “General Public” for “Scope”.

22Serious gaming/gamification was compared with traditional learning in 19 studies, with digital education in 7 studies, and with other serious gaming/gamification interventions in 6 studies.

23One study addressed patient outcomes. Fourteen studies assessed knowledge. Twenty-three studies addressed outcomes relating to skills. Four studies assessed outcomes related to attitudes. Sixteen studies addressed participant satisfaction.

24Figures 2 and 3 summarize the risk-of-bias assessments for the included studies. A total of 25 of the included studies were considered to be at high risk of bias according to Cochrane guidelines, because they had a high or unclear risk of bias for either the sequence generation or allocation concealment domains. All three cRCTs were rated high for incorrect analysis, as none accounted for clustering in the analysis.

25Effects of the interventions are compared in Multimedia Appendices 2-4 and a summary is given below.

26One study measured patient-related outcomes. This study compared serious gaming/gamification with an online posting intervention for primary care physicians and reported significantly shorter time to control blood pressure in the intervention group for only a subgroup of participants whose patients were already on antihypertensive medication at the start of the study (P=.02), although this may not be clinically significant (117 vs 125 days). Data were insufficient for calculation of standardized mean differences (SMD). The quality rating assessed using Grading of Recommendations, Assessment, Development and Evaluations (GRADE) was low.

27Figure 4 summarizes the results of studies reporting knowledge outcomes.

28Four RCTs reported higher posttest scores in a serious gaming/gamification group than in a traditional learning group, with a mostly moderate magnitude of effect. Interventions included a videogame to prepare students to enter the operating room (SMD 1.05, 95% CI: 0.68-1.41), a urology educational adventure game (SMD: 0.69, 95% CI: 0.35-1.03), pediatric respiratory disease-assessment game for nurses (SMD: 0.65, 95% CI 0.23-1.07), and the InsuOnline serious game (SMD: 0.40, 95% CI: 0.06-0.73). Comparisons were made between usual learning, written script, and traditional lectures.

29One RCT of a group intervention, where speech and language science students played a serious game together in a classroom via a projector, found no difference in anatomy and physiology knowledge compared to a self-study control (SMD: 0.05, 95% CI: –0.74 to 0.83).

30Two cRCTs were also included. One showed evidence of a large magnitude of effect for a blood transfusion serious game as compared to usual education, although the effect may not have been statistically significant (SMD: 1.95, 95% CI: –0.20 to 4.11). The second study showed no evidence of effect for a geriatric game compared with standard education, although this study measured perceived knowledge rather than an objective measure (SMD: 0.01, 95% CI: –1.50 to 1.61).

31All the individually played games with an objective assessment of knowledge suggested that serious gaming/gamification was superior to traditional learning. The quality rating assessed using GRADE was low for this outcome and comparison (Table 7).

32Five studies found no evidence of a difference. Studies included comparison of serious gaming on dentin bonding and an online lecture control and serious gaming with digital education on patient safety and Web-based vascular anatomy study aids with and without game elements. One study of a group serious gaming intervention found no difference in pediatric knowledge between groups who played a projected board game in teams in a conference room with Web-based pediatric flashcards. One study found that compared with an online educational posting, serious gaming (an online spaced-education game) may improve knowledge (large magnitude of effect) (SMD: 0.82, 95% CI: 0.42-1.22). The quality rating assessed using GRADE was low (Table 8).

33One study of a spaced-education game found that interventions with greater question spacing (four questions every 4 days rather than two questions every 2 days) resulted in higher posttest scores, with a moderate magnitude of effect (SMD: 0.50, 95% CI: 0.38-0.64). The quality rating assessed using GRADE was moderate, as the one included study had a low risk of bias in all but one domain.

34A total of 24 studies addressed skill outcomes. Figure 5 summarizes the results of studies reporting skill outcomes.

35Twelve RCTs and three cRCTs compared serious gaming/gamification to traditional learning in this outcome category. The results were inconsistent, and studies were generally of low quality, making it difficult to draw conclusions about the efficacy of these interventions.

36Six studies reported significant differences between groups for overall skill assessments in favor of serious gaming, with a magnitude of effect ranging from small to large. However, SMDs for two of these studies could not be calculated due to missing data. Interventions included games with scenarios simulating clinical environments and serious diverting interventions for improving practical skills.

37Three studies comparing serious gaming/gamification with traditional learning used multiple measures for assessing skill outcomes; differences in favor of serious gaming/gamification were observed for some, but not all, of these skill measures, and the studies did not present an overall estimate of the effect. Effect sizes could not be estimated, as SDs were not reported and attempts to contact the authors for further data were unsuccessful.

38Two studies reported no significant difference in skill outcomes when comparing serious gaming/gamification and traditional learning and another reported no differences in pre- and posttest scores in either group.

39Three cRCTs were also included. One showed evidence of an effect of small magnitude, favoring a blood transfusion game group (SMD: 0.33, 95% CI: –1.19 to 1.86); the second study found evidence of a moderate magnitude of effect, favoring the standard educational activity group, although skill measures were self-perceived as opposed to objective (SMD: –0.77, 95% CI: –2.53 to 1.00); and the third showed no evidence of effect for a triage trainer game (SMD: –0.18, 95% CI: –2.37 to 2.02). Each of these results may not be statistically significant.

40There is some evidence that serious gaming/gamification interventions are more effective for improving skills than traditional learning. The quality rating assessed using GRADE was low, as the risk of bias was unclear for multiple domains and all the included studies had fewer than 400 participants.

41Five studies comparing skill outcomes for serious gaming/gamification and other modalities of digital education found no evidence of a difference in outcomes between groups. In these studies, serious gaming was compared with an online video on dentin bonding and with an electronic module (e-module) on patient safety and management of an acutely unwell patient. Another study reported higher postintervention skill score in a virtual reality control group than a commercial off-the-shelf intervention, with a large magnitude of effect for the time taken to complete surgical skill tasks (peg transfer and bimanual carrying; SMD: –1.56, 95% CI: –0.31 to –2.81), but reported no difference for distance travelled with surgical instruments when completing these tasks. The quality rating assessed using GRADE was low.

42We are uncertain whether any particular type of serious gaming/gamification is more effective than the other for improving skills. In three of the five studies comparing two serious gaming/gamification interventions, games involving motor skills, visuospatial skills, and manual dexterity may be more effective than interventions involving cognitive skills for improving laparoscopic surgical skills, but the quality of available evidence is very low.

43Figure 6 summarizes the results of studies including professional attitudes outcomes.

44Two RCTs compared a serious gaming/gamification intervention with traditional learning and measured outcomes related to professional attitudes. There was some evidence of a small magnitude of effect for a serious game, preparing students to go into the operating theatre for the first time, compared with traditional learning (SMD: 0.49, 95% CI: 0.14-0.84). A study comparing an insulin-prescribing game with an onsite learning activity for primary care reported insufficient data for comparisons between groups.

45One cRCT was also included. When reanalyzed with the number of clusters as the sample size to account for clustering in the analysis, there was evidence of intervention effectiveness, but this may not have been statistically significant and the analysis was likely underpowered (SMD: 1.23, 95% CI: –0.55 to 3.02). The quality of evidence for this outcome and comparison was rated very low according to the GRADE assessment.

46One study compared a serious game and an e-module on patient safety and reported no difference between groups in perceived patient safety behavior or reported stress.

47The quality of evidence was rated low according to the GRADE assessment.

48One study (reported in two papers) compared two serious diverting interventions, one was a first-person shooter (FPS) and one was a non-FPS, and reported no significant differences in self-efficacy or positive engagement modes. Data were insufficient for calculation of effect sizes. The quality of the evidence was very low according to GRADE assessment.

49Eleven RCTs and two cRCTs measured outcomes relating to satisfaction. Seven studies did not measure satisfaction in a comparison group, Diehl et al measured satisfaction with the intervention and comparison group using different scales, and Kerfoot et al reported results for participants in both groups combined; therefore, these studies did not meet the inclusion criteria for this review. The remaining studies showed mixed evidence and are compared below.

50One study reported significantly better attitudes toward learning among a serious gaming group (a urology adventure game group) compared with a written script. The quality rating assessed using GRADE was low.

51Three studies reported higher satisfaction scores for serious gaming/gamification on managing acutely unwell patients, patient safety, and training during a pediatric clerkship compared with an e-module or Web-based flashcards covering the same topics. The quality rating assessed using GRADE was low.

52Results of a participant survey suggested that more participants in the FPS gaming group than in the non-FPS gaming group found the intervention beneficial for their performance on a surgical simulator. No significance test was reported. The quality rating assessed using GRADE was very low.

53No studies measured economic outcomes of education or adverse effects of the intervention.

54The objective of this systematic review was to evaluate the effectiveness of serious gaming and gamification interventions for delivering pre- and postregistration health professions education. A total of 30 studies, most at high risk of bias according to Higgins, were identified, with high levels of heterogeneity in terms of populations and outcomes.

55Serious gaming/gamification has the potential to reach a global audience and hence has been identified as a possible educational strategy that could contribute to transformation of health professions education. Results from our review show that serious gaming/gamification in pre- and post- registration health professions education could result in increased knowledge, skills, and satisfaction when compared to traditional education and, perhaps, other modalities of digital education.

56Most of the current literature on the effectiveness of serious gaming/gamification has been performed in high-income countries, which limits the applicability of this review’s findings to low- and middle-income countries. This is a key gap in the evidence, as low-and middle-income countries are most affected by the worldwide shortage of trained health workers. Other limitations of the evidence base include the lack of studies assessing patient outcomes, or clinician behavior.

57The cost of serious gaming devices might be a barrier for use compared with traditional lectures or text books. For example, some of the included studies used game consoles, which many health care workers, particularly in low- and middle-income settings, may not have access to. Other included studies used lower-cost modes of delivery, such as projecting a serious game to a group of students who played together. However, none of the eligible studies provided any information about economic outcomes of education or adverse or unintended effects of the intervention, which limits our understanding of the feasibility of implementing these interventions in practice and our understanding of the applicability of serious gaming/gamification as a cost-effective solution.

58Considering the types of interventions that may be effective, based on classification of interventions by original design intention, there were no clear patterns suggesting differing effectiveness between custom designed games and commercial off-the-shelf games for skill outcomes. Only custom-designed interventions were used to improve knowledge.

59There was considerable heterogeneity in the results, particularly for skill outcomes, which we were unable to explain by systematic consideration of the types of intervention, population, and comparison group.

60As serious gaming/gamification is an emerging field in the education sector, there are few previous reviews of the literature on its role in health professions education. Wang et al conducted a systematic review of serious games for training health care professionals focused on game development and evaluation methodologies and reported a growing number of interventions and diversity of game genres over time. Similar to our review, they found that study designs and methodological quality were heterogeneous and that best practices for development, evaluation, and use of such interventions are still being defined. A scoping review of serious gaming/gamification in health professions highlighted the need for economic evaluation of interventions, particularly when studies show no difference in efficacy between a serious game and traditional learning. Our review contributes to the literature by providing an up-to-date summary of the evidence, focused on intervention effectiveness with a comprehensive systematic search. This is the first systematic review of the evidence indicating that serious gaming/gamification may improve participant knowledge compared with traditional learning.

61There is a broad range of literature on serious gaming beyond health professions education. Meta-analyses have suggested that these interventions could significantly enhance learning among school students, adult workforce trainees, and mixed-age groups with regard to cognitive and attitudinal outcomes and knowledge acquisition. These reviews also suggested that games were more effective if they were supplemented with other methods of instruction, had multiple sessions, and involved active rather than passive learning. It was unclear whether playing as a group or alone was more effective. Systematic reviews have also suggested that serious gaming may have a role in the management of various medical conditions such as depression and chronic conditions in young people and in improving health outcomes. The body of evidence on gamification interventions for education is smaller, with a systematic mapping study suggesting that most studies focused on the role of such interventions in student engagement and were published only as conference papers rather than full peer-reviewed articles. A systematic review identified some evidence that gamification can be beneficial for health behavior change and well-being.

62This review suggests that serious gaming may have the potential to advance education by improving knowledge, and possibly skill, outcomes for health professions compared with traditional learning. It may be able to provide educational interventions that are of equivalent educational value to other kinds of digital education, but with improved learner satisfaction. If this approach is equivalent to other kinds of education in terms of outcome but more cost-effective or able to offer greater access, it may provide further reasons to recommend serious gaming/gamification interventions, but no studies assessing these factors were identified.

63Only two studies assessed gamification interventions. One suggested that the intervention was more effective than an online posting in improving knowledge by a large magnitude. The other suggested greater improvements in patient outcomes for questions spaced with four questions every 4 days rather than two questions every 2 days. These findings suggest that it may be worthwhile to incorporate gamification techniques into education, where possible, particularly for interventions aimed at improving knowledge, although further evidence is needed to establish the effectiveness among different groups of health professions for a wider range of patient outcomes and skill- and attitudes-related outcomes.

64This review adopted a detailed and comprehensive search strategy without language limitations, followed by robust screening, data extraction, and risk-of-bias assessments, adhering to the Cochrane guidelines. Thirty studies were found to be eligible, but most of them were at high risk of bias according to Higgins, with high levels of heterogeneity in terms of populations and outcomes. This heterogeneity of the included studies made it inappropriate to perform meta-analysis for any outcomes. Evidence for the majority of the outcomes and comparisons in the review was considered of low quality. Many studies have small sample sizes that were unlikely to provide sufficient power to detect an effect, provided insufficient detail for complete risk of bias assessment, and did not report all data for all outcomes assessed; in addition, statistical analysis was often not performed appropriately for the data (eg, not accounting for clustering), reducing confidence in the results (Figures 2 and 3). Only two studies of gamification interventions were identified.

65Serious gaming has the potential to contribute to the field of health professions education, but given that most studies to date are of low quality and carried out in high-income countries, future research should seek to use an RCT or cRCT design following a published protocol; evaluate interventions with a robust theoretical underpinning; be adequately powered; involve participants from low- and middle-income countries; appropriately randomize participants and blind outcome assessors, where possible; use validated outcome-assessment tools, facilitating comparability between interventions and studies; compare both serious gaming and gamification interventions with each other and with controls (other types of digital health education or traditional learning); and assess patient outcomes, participant behavior, attitudes, economic outcomes of education, and adverse events.

66There is some evidence that serious gaming/gamification may improve health professionals’ knowledge after the intervention compared with traditional education. In addition, some low-quality evidence shows that serious gaming/gamification may improve or be equivalent to traditional education for skills and to other modalities of digital education for knowledge and skills. Future research should evaluate theory-grounded interventions and assess patient outcomes, economic outcomes of education, and adverse events.