Recommendations for Methodology of Virtual Reality Clinical Trials in Health Care by an International Working Group: Iterative Study

1Therapeutic virtual reality (VR) is an innovative treatment modality to manage a broad range of health conditions and is gaining considerable attention. Users of VR wear a head-mounted display (HMD) with a close-proximity screen that creates a sense of being transported into life-like, three-dimensional worlds. VR has been used to assess and treat a wide variety of medical, surgical, psychiatric, and neurocognitive conditions including pain, addiction, anxiety disorders, schizophrenia, eating disorders, stroke rehabilitation, vestibular disorders, and movement disorders. One of the first published uses of HMD-based therapy was the treatment of acrophobia in 1995. There have also been functional magnetic resonance imaging studies demonstrating the effect of VR on the brain during receipt of a painful stimuli. VR is thought to work through a combination of distraction, extinction learning, cognitive-behavioral principles, mindful meditation, stress reduction, gate-control theory, and the spotlight theory of attention. Importantly, VR has become increasingly portable, immersive, and vivid, which has enabled the technology to be used in a broad range of inpatient and outpatient applications.

2As the use of therapeutic VR expands, it is essential that guidelines are established to ensure scientific rigor in the development and evaluation of VR applications, similar to established standards for pharmacotherapies. VR developers would benefit from systematic guidance on best practices for designing and conducting VR clinical trials. To fulfil this unmet need, we garnered input from an international working group, called the Virtual Reality Clinical Outcomes Research Experts (VR-CORE) committee. This paper presents the resulting best practice framework informed by expert input, along with specific recommendations on ways to conduct high-quality VR treatment trials. Although the focus of this paper is VR, the framework also applies to other emerging “XR” technologies, including augmented reality and mixed reality, as the methodologic considerations for clinical trials are largely similar across XR platforms.

3We performed a systematic review of randomized controlled trials (RCTs) using therapeutic VR to help identify eligible VR-CORE committee members through review of author lists. To cover the largest breadth of studies, the literature search focused on existing meta-analyses of therapeutic VR RCTs identified through search of PubMed, Google Scholar, and the Cochrane Database of Systematic Reviews using a combination of keywords: (“virtual reality” OR “VR”) AND (“review [pt]” OR “systematic review [pt]” OR “meta-anal*” OR “metaanaly*”). Based on our literature search, and supplemented by recommendations from established experts, we developed a multidisciplinary group for the VR-CORE, including experts in fields relevant to developing and testing VR treatments such as user-centered design principles, software design, epidemiology, statistics, and clinical trial methodology. The committee was formulated to balance expertise across clinical disciplines (medicine, pediatrics, surgery, psychology, psychiatry, neuroscience, anesthesia, nursing, and rehabilitation) and reflect multinational perspectives.

4To obtain systematic feedback from the committee, a series of electronic meetings were held to collect and synthesize structured input. An iterative approach was modeled after similar processes were employed by our previous working groups in other fields of health care. Using an online meeting platform that allows users to view and react to each other’s comments, committee members initially responded to open-ended “think aloud” prompts (eg, “When you think about the current state of the clinical VR research, what comes to your mind?”), followed by increasingly specific probes prepared by the moderators (eg, “What should be the role of human centered design principles in developing VR treatments?”). The full set of questions and responses is listed in Multimedia Appendix 1. The active members of the VR-CORE at the time of this discussion are listed in the Acknowledgments section. Emergent themes and proposed methodologic best practices were culled from the online dialogue, and the resulting recommendations were distributed to the members for synthesis and iterative rephrasing.

5Multimedia Appendix 1 provides excerpted transcripts of the VR-CORE responses to discussion topics. Key themes drawn from the online dialogue are summarized in the following sections.

6Committee members described the current state of clinical VR research as the “Wild West” with a “lack of clear guidelines and standards.” The state of current VR research was described as “heterogeneous,” often focused “more on the tech rather than the theories behind it.” Committee members expressed concern that much of the current research is “merely descriptive” in nature, often insufficiently powered, focused on small case reports and retrospective analyses, and often does not employ experimental designs.

7The committee believed it is vital to “include the patients’ voice early and often in the development of VR treatments” and that developers must “carefully, systematically, and meticulously seek the patients’ feedback” through participatory research and design thinking that involves multidisciplinary collaboration. The committee acknowledged the importance of including the voice of providers as well. The committee also called for better definitions and standardization of therapeutic VR study designs.

8The committee described various stages for developing and validating VR treatments, beginning with content development in partnership with end-users, progressing through initial clinical testing and safety evaluation, and ending with properly powered RCTs. The committee outlined a wide range of considerations for each stage (Multimedia Appendix 1), including the importance of standardizing control groups, selecting clinically relevant outcome measures, reporting which equipment was used in the trial, accounting for dropouts and disqualified participants, and allowing for pragmatic features of each study design.

9Although there are fundamental best practices in study design that apply to all biomedical intervention trials, the committee identified VR-specific attributes that are unique considerations for VR trials. Using the Food and Drug Administration Phase I-III pharmacotherapy model as guidance and combining the results of literature synthesis with VR-CORE input, a framework emerged to support three phases of VR clinical study designs, named VR1, VR2, and VR3.

10VR1 studies focus on content development by working with patient and provider end-users through principles of human-centered design. VR2 trials conduct early testing with a focus on feasibility, acceptability, tolerability, and initial clinical efficacy. VR3 trials are RCTs that compare clinically important outcomes between intervention groups and a control condition. Each study should undergo ethical review before initiation. Figure 1 summarizes each phase of the VR-CORE model. Best practice recommendations for each trial design are described below.

11The committee strongly believes that therapeutic VR applications should be designed with direct input from patient and provider end-users. Lack of patient involvement, poor requirement definitions, and nonadaptation to user feedback are some of the common factors that explain failures of digital interventions. Incorporating patients into the design process enables developers to increase the relevance and effectiveness of VR treatments. The committee stresses that VR treatments should be created with acknowledgment of patients’ knowledge, attitudes, beliefs, preferences, and expectations of therapeutic VR. VR-CORE refers to a VR1 study as one that results in the development of VR treatment in partnership with patient and provider end-users and follows best practices for patient-centered design.

12After their review of the literature on human-centered design both generally and in relation to digital and VR interventions, the committee identified three key principles that are fundamental for developing “desirable, feasible and viable” VR treatments. These principles promote empathy, team collaboration, and continuous user feedback (Table 1). The committee believes that the use of these principles allows development teams to better identify users’ needs, incorporate user feedback, and institute rapid cycle improvements that generate more relevant products at lower cost. The key principles for VR1 studies are outlined in Table 1.

13The committee believes that the more attuned a development team is to the specific perspective and needs of patients, the more likely they are to design meaningful VR treatments. Promoting empathy toward the design process involves carefully listening to and elucidating patients’ social environment, needs, fears, desires, habits, hopes, aspirations, and expectations. The committee recommends initiating the design process with an inspiration step, or exercise focused on culling patients’ voice and understanding their needs, struggles, and experiences. Table 1 describes best practices for sparking inspiration within the framework of empathy. Different patient profiles and scenarios should be included in this first step. Many techniques can be used to develop empathy and inspiration of the design team. These include qualitative assessments, observations, spending time with users, and conducting interviews and user experiments. In addition, a patient journey map can be used to illustrate the interpretation of a story from a patient’s perspective. The working group also recommends seeking input from relevant nonpatient end-users, including health care providers who may prescribe the VR treatment or interact with patient users.

14The committee believes that team collaboration is fundamental for collectively designing a VR treatment and synthesizing data collected during the inspiration step. Brainstorming helps generate ideas from the initial corpus of data and findings. Table 1 describes best practices for ideation within the framework of team collaboration. The process of ideation allows team members to think expansively and divergently. As a range of ideas is generated, some ideas will be extreme or ambitious, whereas others will be achievable. Depending on the time and the available budget, the team decides what ideas should be prototyped further.

15An effective VR treatment should be developed through continuous user feedback and iterative prototyping, thereby enabling the team to rapidly test their ideas during real-time assessment from end-users. Table 1 describes best practices for VR treatment prototyping within the framework of user feedback. Prototypes should be refined with continuous testing by patient end-users, and failures are viewed as a way to learn and improve the prototype to better meet users’ needs. Hence, the number of defects will tend to be lower and less costly in the future. To help facilitate the learning process for patients, it is recommended, when feasible, that the research team use a “mirroring” program to allow the research staff to see what the patient is viewing through the VR headset and help them learn the user interface.

16Briefly, the committee believes that the VR1 treatment design process should start with end-users. VR-CORE recommends specifying who the real users are and what they say, see, feel, and do. Hence, implementation of a patient-design approach is an important way to place users at the center of the VR design process. For researchers who are developing an open-source VR intervention that they would like to share with the academic community for collaborative V1 development process, the use of a software-development platform such as GitHub.com and citation of the latest version of the program within the methods section of VR1 research papers are recommended. The committee also recommends use of the Integrate, Design, Assess, and Share checklist developed by Mummah and colleagues as a supplemental, structured guide for conducting a VR1 study.

17Once the research team has developed a VR treatment in partnership with end-users, the resulting product should undergo initial assessment in the target patient population within a representative clinical setting, herein termed a VR2 trial. Modeled after the work of Mosadeghi and colleagues, the purpose of VR2 trials is to conduct early testing with a focus on acceptability, feasibility, tolerability, and initial clinical efficacy prior to initiating a more definitive VR3 clinical trial. Although developers may opt to bypass a VR2 trial in lieu of a VR3 trial, there is a risk of subjecting an incompletely tested intervention to a larger and costlier RCT, and best practices in digital intervention development suggest an intermediary stage between initial VR design and definitive testing. The following sections describe the features of a VR2 trial.

18In contrast to a VR1 study, which is focused on collaborative content development in a design environment, the VR2 trial evaluates what happens when the VR treatment is placed in the hands of target patients within the intended clinical setting. For example, a VR treatment focused on management of inpatient pain should be tested in an inpatient environment. A VR treatment targeting outpatient stroke rehabilitation should be evaluated in locations where patients receive rehabilitation, such as in a physical therapy center or, if intended, at home. In short, a comprehensive VR2 trial evaluates the VR treatment in the natural setting(s) where the product is intended to be used. Table 2 summarizes the best practices for VR2.

19In the context of a VR2 trial, acceptability refers to a patient’s willingness to use the VR treatment. Previous research on therapeutic VR reveals a drop off in the relation between patient eligibility to receive VR and patient willingness to try VR. The disconnect emphasizes that many patients are uninterested in using novel health technologies such as VR, particularly when hospitalized or under duress. Among those who are eligible for a VR trial, some choose not to participate for a wide variety of reasons. Patients may express varying degrees of skepticism, fear, vulnerability, and concern regarding psychological consequences or simply not want to be bothered by the equipment. In a VR2 trial, investigators collect data regarding patient willingness to try the VR treatment, including reasons why they did or did not find the intervention to be acceptable for use. Researchers should collect and report acceptability data using techniques such as focus groups, cognitive interviews, or structured questionnaires.

20In the context of a VR2 trial, feasibility is the degree to which the VR treatment can be successfully integrated within the flow of usual care. The committee noted that even the best designed VR treatments can face implementation challenges when applied on the front lines of health care delivery. It is wise for developers to understand potential barriers early and often, identify workarounds and solutions to these barriers, and only then consider testing their interventions in VR3 RCT trials. For example, patients and providers often seek information regarding the frequency and “dosing” of a VR treatment; these details could be manually collected in the context of a VR2 trial. Similarly, treatments deployed in a clinical environment may be unfamiliar to doctors, nurses, and other health care providers, giving researchers an opportunity to study the interaction among staff and proactively identify areas of confusion or misuse. The committee recommends including a table that enumerates patient, provider, technical, and operational barriers to use; identifies root causes; and offers solutions to enhance effectiveness in future clinical applications.

21The VR2 trial offers an early opportunity to evaluate patient tolerability of the VR treatment, including both hardware and software components. Researchers should measure and report the prevalence of patient-reported physical (eg, vertigo, nausea, and “cybersickness”) and emotional (eg, fear and anxiety) adverse effects of the VR treatment, along with any discomfort or inconvenience related to the VR equipment (eg, ill-fitting headset, facial or nasal pain, inability to explore the three-dimensional environment fully due to limited mobility).

22Cybersickness (or VR sickness) is a unique side effect of VR. There a several different terms used interchangeably within the literature, such as simulator sickness or “sim sickness,” although some believe they are different types of motion sickness. When the vestibular system and oculomotor system notice a discrepancy between reality and the virtual environment, one or more of following symptoms ensue: eyestrain, nausea, fatigue, headache, blurred vision, and postural instability. The specific mechanism of cybersickness is still unknown.

23Recommendations for developers already exist: appropriately accelerate within the program, anticipate changes in direction, affect changes in the field of view, establish realistic virtual avatar movements, reduce drops in the frame rate below 60 fps, blur the display with movement, and provide other solutions at the level of program design.

24There are also several strategies for medical staff and researchers including habituation, assessment of the risk of side effects before the intervention, use of oculomotor exercises before the intervention, and diaphragmatic breathing during the intervention. One of the most useful strategies is to limit the total duration of each treatment session, particularly early in the process.

25The VR-CORE recommends assessing for side effects at every phase (VR1, VR2, and VR3). Regarding assessment scales, the Simulator Sickness Questionnaire is the most commonly used scale in the literature.

26Although the VR2 trial is not designed to definitively test whether a VR treatment is efficacious or effective, it offers an early opportunity to measure efficacy within the context of a small clinical trial. There is no requirement in a VR2 trial to include a control group, although uncontrolled case series carry a higher risk of bias than controlled studies; even studies with nonrandomized concurrent controls, “wait list” controls, or retrospective controls may reduce the risk of bias as compared to an uncontrolled series.

27Regardless of the inclusion of a control group, investigators should identify a clinically relevant and validated patient- reported outcome (PRO) to evaluate the evidence of efficacy. For example, a study evaluating pain might include a standard 11-point numeric rating scale before and after exposure to the VR treatment. A study evaluating stroke rehabilitation might measure physical function with the National Institutes of Health Patient Reported Outcomes Measurement Information System. Selection of the most appropriate PRO is at the discretion of the research team, but should be carefully justified and capture the most salient features of patient-reported health that might improve with the VR treatment.

28The most definitive clinical validation of a VR treatment is the VR3 trial, which is a prospective, adequately powered, methodologically rigorous RCT evaluating clinical outcomes and safety in target patients receiving the VR treatment as compared to a control condition. Although the therapeutic mechanism of action may be studied as a secondary goal in a VR3 trial (eg, through neuroimaging, blood biomarkers, and physiologic testing), the principal goal is to evaluate the treatment’s impact on a clinically meaningful patient outcome rather than surrogate markers.

29Although the committee acknowledged understandable costs and resource barriers involved in conducting VR3 trials, there was broad agreement that RCTs are of equal scientific importance in therapeutic VR as any other form of treatment and should be prioritized whenever possible. Multicenter collaborations may facilitate VR3 trials by combining patients and resources through shared protocols. The features of a VR3 trials are described below and summarized in Table 3.

30Having been developed in a VR1 study and initially tested in a VR2 trial, the study intervention should be clearly described in preparation for a VR3 trial. Researchers should provide details regarding the equipment used; visualizations employed (with representative screenshots or videos); and frequency, duration, and timing of use. Optimally, the intervention should be manualized, and at the very least, enough details should be provided to allow other investigators to repeat the trial, if desired. The Template for Intervention Description and Replication checklist provides a useful framework for describing study interventions and should be applied to VR treatments. The target patient population should be clearly described, including explicit inclusion and exclusion criteria employed. Certain exclusion criteria may be standardized among VR trials, such as a history of significant motion sickness, active nausea, and vomiting or epilepsy.

31The committee acknowledged that there is no perfect or standardized control condition for all VR treatment trials; the optimal control depends on the patient population, proposed mechanism of action of the intervention, and clinical setting, among other considerations. Selection of the control is at the discretion of the research team but should be justified and explained. The committee described a hierarchy of control conditions, ranging from “usual care” without any active intervention to passive visualizations on a two-dimensional screen and nonimmersive visualizations within a headset, immersive but passive experiences within a headset, and immersive and active experiences within a headset. Selection of the optimal control may be guided by considering the hypothesized target of engagement and the proposed mechanism of action.

32Randomization should be described and ideally achieved using an appropriate computer program (eg, MS Excel Random Number Generator) or random number tables without involvement of the investigators who enrolled the patients.

33The committee acknowledged that blinding and concealment can be challenging, but they identified techniques to incorporate these RCT principles within the constraints of VR research. For example, Spiegel and colleagues (2017) achieved concealment of allocation in an RCT comparing a library of VR content to a “health and wellness” television channel in hospitalized patients experiencing pain. At the time of consent, the researchers explained to patients that the study compared “two different audiovisual experiences designed to reduce pain,” but did not describe the details of the competing interventions. Patients randomized to the television intervention did not know that VR was the other condition and vice versa. This approach may reduce the “novelty effect” of receiving VR rather than a familiar experience like television. Equipoise may also be achieved by exposing patients in both arms to headsets, but varying the content viewed within the headset (eg, immersive vs nonimmersive, active vs passive). At a minimum, study analysts should be blinded to patient group allocation, allowing for unbiased evaluation of the data without the knowledge of the study group. Patients should be asked not to reveal details of the program they experienced to decrease the chance of unblinding the study analysts. The measurement of perceived group assignment at the end of the study can help assess the success of blinding within the study. This should be done at the discretion of the research team.

34Like the VR2 trial, VR3 trials must prespecify a clinically relevant and validated PRO as the primary endpoint. The study must be appropriately powered to demonstrate a minimally clinically important difference (MCID) in that endpoint between the VR treatment and control arms. The psychometrics of PRO measurement are beyond the scope of this document, but existing references may assist investigators in protocol development. Secondary endpoints may include a variety of clinical, imaging, biometric, and physiologic surrogate markers, as deemed appropriate by the study team. Like VR2 trials, potential adverse events must be prospectively measured and reported.

35VR3 studies should monitor patients for a sufficient period to determine whether the VR treatment meaningfully impacts clinically important outcomes. One-time, short-term evaluations may be insufficient to evaluate the true clinical value of an intervention. Follow-up over several days may be appropriate if the study only focuses on hospital stay, but measurement over weeks, or even months, may be necessary to assess the impact on long-term clinical benefits.

36VR-CORE recommends that the primary outcome be reported as the before and after difference in difference between study arms, with accompanying 95% CIs. For example, the change in the mean PRO score before and after the VR intervention should be compared against the change in the mean PRO score before and after the control intervention. In addition, the panel recommends predefining a binary response criterion, guided by the MCID of the primary endpoint. The proportion achieving the MCID should be reported and compared between groups, and the resulting number needed to treat should be calculated.

37The primary analyses should use the intention-to-treat population, including all patients randomized regardless of follow-up or receipt of study interventions. However, per-protocol analysis may be appropriate in certain situation, such as if patients refuse the VR treatment after randomization; in this instance, reporting the rate of refusal would be important, but investigators might also seek to compare therapeutic responses only among those receiving the intervention.

38Multivariable analysis may be useful in adjusting for prespecified confounding factors (especially if not equally distributed in the study groups) and exploring independent predictors of outcomes. To perform a multivariable analysis, it is optimal to have at least 10 (preferably, 20) observations for each independent variable included in the multivariable model.

39VR3 trials must be registered in a publicly accessible registry (eg, such as ClinicalTrials.gov). All completed trials should be published, regardless of whether they are positive or negative. The Consolidated Standards for Reporting Trials (CONSORT) guidelines provide the framework for reporting RCTs and should be followed in VR3 trials. VR3 trials must include a CONSORT diagram to demonstrate the flow of patients through each stage of the trial, including the number screened to the number randomized into each study group and the number analyzed.

40To improve methodological quality in the therapeutic VR literature, the VR-CORE international working group presents a three-part framework for best practices in developing and testing VR treatments. This framework may be used to facilitate development of high-quality, effective, and safe VR treatments that meaningfully improve patient outcomes. Patients, providers, payers, and regulators should consider this framework when assessing the validity of VR treatments.