From Arksey and O’Malley and Beyond: Customizations to enhance a team-based, mixed approach to scoping review methodology

1Specifications table

2Scoping reviews are a method of knowledge synthesis that identify trends and gaps within an existent knowledge base, or scope of knowledge, for the purpose of informing research, policy, and practice; Levac; Tricco et al.. Since the debut of Arksey and O’Malley's seminal article on scoping review methodology, the method continues to evolve. The purpose of this manuscript is to summarize, synthesize, and offer customizations to enhance scoping review methodology within the context of a mixed method, team-based approach.

3Arksey and O’Malley were the first to highlight the rationale and provide methodological recommendations for conducting a scoping review. They articulate four common reasons to undertake a scoping review: 1) examine the state of research activity on a selected topic, 2) screen for the need for and/or feasibility of undertaking a systematic review, 3) synthesize and share research findings, and/or 4) highlight key gaps in the literature. To assist authors through the scoping review process, the Arksey and O’Malley Framework propose six stages of conduct: 1) specify the research question, 2) identify relevant literature, 3) select studies, 4) map out the data, 5) summarize, synthesize, and report the results, and 6) include expert consultation.

4Scholars continue to contribute meaningful insights to build upon the Arksey and O’Malley Framework. Levac et al were the first to call attention to the need for universal language to consistently describe the method. Given the inconsistent use of nomenclature such as “scoping review,” “scoping exercise,” “scoping study,” or “scoping review of the literature” in the literature, scholars continue to advocate for the use of “scoping review” as a universal term of reference,,,. Ambiguity remains a main critique of the initial Arksey and O’Malley framework. Levac et al address this by offering detailed descriptions and the challenges associated with each stage of the Arksey and O’Malley framework. Further methodological recommendations on the conduct of scoping reviews are published by Colquhoun et al, Daudt, van Mossel, and Scott and the Joanna Briggs Institute in. Colquhoun et al call to improve transparency in reporting and suggest using an adaptation of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) and conversations continue towards constructing a standardized reporting checklist via the Enhancing the QUAlity and Transparency of Health Research (EQUATOR). By comparing and contrasting scoping and systematic reviews, Joanna Briggs Institute clarified the purpose and essential characteristics associated with each type of review and essential elements to include when creating a protocol for a scoping review.

5As knowledge and use of scoping review methodology increased, so have the number of published scoping review manuscripts. A PubMed search of the term “scoping review” yielded 27,345 results as of October 23, 2020 and revealed a trend of scoping reviews on scoping reviews,,,. Findings synthesized from the scoping reviews on scoping reviews revealed critiques including wide variation in methodological rigor,,; inconsistencies in how findings are reported,,; lack of reference to quality,; and limited guidance on team-based applications of the method,. Furthermore, the inclusion of quality appraisal or risk of bias remains under debate,; Levac et al, 2014;,,, as does the question of whether scoping reviews as a methodology would benefit from the construction and application of standardized reporting guidelines,,,.

6Method details: Enhancing the Arksey and O’Malley Framework

7This paper builds upon the existent literature describing scoping review methodology, with customizations that emphasize a team-based, mixed method approach. Our recommended enhancements, as derived from our synthesis of literature on scoping review methodology and findings from scoping reviews about scoping reviews, are organized into tables per each stage of the Arksey and O’Malley Framework. We present our rationale for these methodological customizations, illustrating how we applied these customizations within “Outcomes and outputs affiliated with children’s advocacy centers in the United States: A scoping review” (2020).

81) Arksey and O’Malley Framework Stage 1: Specify the research question

9Clear articulation of the research question is the first stage of conducting a scoping review. While the main challenges associated with this stage involve the broad nature of the research question(s) and purpose, key strategies identified in prior literature for success in this stage include: embracing the process of question specification for scoping review as iterative, refining the question, clarifying its relationship to the study purpose, and determining the relevant population and context in conversation with learning from existent literature (Table 1A). To these recommendations, we add “identify framework within which to refine specification of research question and purpose.” (Table 1B).

10Research question and purpose: “Outcomes and outputs affiliated with children’s advocacy centers in the United States: A scoping review”

11We first conceptualized our research questions and purpose based on a rapid review of the literature. This broad search revealed a few important insights. While a systematic review exploring the efficacy of the Children’s Advocacy Center (CAC) model was published in 2016, the results of the systematic review were limited because very few articles were able to meet the rigorous inclusion criteria required of a systematic review. Additionally, the rapid review of the literature suggested that a wide variety of disciplines publish articles about CACs. This is not an uncommon trend within research about child abuse and neglect, as caring for children and families who experience abuse often involves multiple sectors, disciplines, and professionals. Given this, it became clear that a team-based approach (the term team-based approach will be used in this manuscript to include all types of teams: multidisciplinary, interdisciplinary, transdisciplinary, and interprofessional) would be necessary in order to best capture diverse perspectives within the literature. The rapid review also revealed a need to clearly articulate the definitions of key concepts for the scoping review as different disciplines define outputs and outcomes in different ways.

12We selected a population, concepts, and context (PCC) framework to refine the research question and purpose.

13Population: Children aged 0-18 years old who experienced child abuse, non-offending family members of children who experienced child abuse, and multidisciplinary team members who serve children and families who experience child abuse (law enforcement, prosecution, medial team, mental health, victim advocates, CAC staff, social work) in the United States.

14Concept: 1. Outcome and output measures affiliated with CACs

152. Outcome measures were defined as person-centered

163. Output measures were defined as programmatic or service characteristics

17Context: Must involve CACs

18The research questions were:What types of person-oriented outcomes are being measured among children and non-offending family members who seek services and resources at CACs in the United States?What types of output measures are affiliated with CACs in the United States?What types of research designs are being used to study CACs?

19What types of person-oriented outcomes are being measured among children and non-offending family members who seek services and resources at CACs in the United States?

20What types of output measures are affiliated with CACs in the United States?

21What types of research designs are being used to study CACs?

22We articulated our research purpose as follows:

23A greater understanding of the scholarly work surrounding outcome and output measures used by CACs will identify gaps in the research, clarify key themes/concepts, and highlight opportunities to enhance outcomes for children and families who experience abuse and receive services via CACs.

242) Arksey and O'Malley Framework Stage 2: Identify relevant literature

25The second stage of the Arksey and O’Malley Framework focuses on identifying relevant literature, as guided by the research question and purpose. While practical issues such as timing and resources are cited as barriers that may limit the scope of the literature, Table 2A depicts recommendations to navigate such challenges.

26A team-based approach in scoping review methodology

27While Arksey and O’Malley broadly addressed the importance of integrating team-based approach in scoping review methodology, Levac et al specifically called for the explicit incorporation of teams into the method as a means to provide content expertise. As an example, Daubt et al applied the Arksey and O’Malley Framework within the context of an inter-professional team, which they defined as including members with both clinical and research training. Some of the identified challenges of implementing a team-based approach w within the scoping review process include decreased productivity by increasing challenges in coordinating with multiple members, reaching consensus, and navigating a lack of common language. Despite these challenges, the rich insights from diverse perspectives will meaningfully enhance the scoping review process and findings,,. Daubt et al suggest that regular consistent communication, regular meetings, designated team leads, and a smaller research team help to facilitate successful inter-professional teamwork for scoping reviews.

28Assembling the research team for “Outcomes and outputs affiliated with children’s advocacy centers in the United States: A scoping review”

29Data from the rapid review in stage 1 clearly identified the need for a team-based approach in answering the proposed research questions. Rather than providing a specific name for the type of team, a framework was developed to clearly emphasize the var various roles, expertise, skills, and contexts embodied within the members of our team. Given the diverse systems that respond to child abuse and neglect, the research team was selected to include representation from nursing (KKW), criminology (WR), public health (MM, KL), clinical psychology (BVW), medicine (SDR, LM), and social work (HL). Most of the members of the research team possessed both research and clinical expertise. Two team members were selected for their content expertise specifically related to CACs (KL, JJ).

30Research: Four of the team members possess doctoral degrees (KKW, HL, WR, BVW), one of the members is a doctoral candidate (SDR), and the two team members with public health degrees are employed in research positions (MM, KL).

31Clinical Practice: A general pediatrician (SDR), a board-certified child abuse pediatrician (LM), a pediatric nurse practitioner with experience as a sexual assault forensic investigator and working on children protection teams (KKW), a child and family psychologist (BVW), and a criminologist with expertise in collaborating with law enforcement (WR).

32Policy: KL is affiliated with the National Children’s Alliance (NCA), the national accreditation body for CACs. JJ is the Director of the Canopy Child Advocacy Center in Cleveland, Ohio.

33Descriptive analysis of our sample revealed that the most predominant disciplinary backgrounds of the first authors included psychology (29%), social work (22%), and nursing (11%) (Supplementary Material A); thereby validating the importance of having a psychologist, social worker, and nurse as members of the research team.

34Search Strategy and Abstract Selection: “Outcomes and outputs affiliated with children's advocacy centers in the United States: A scoping review”

35In keeping with the Arksey and O’Malley Framework goal of comprehensiveness, we developed our search strategy to be inclusive of publications within the full time frame applicable to our research question (1985 to 2019), relevant to the setting of interest (U.S.), and population of interest (children). We searched multiple indexing services that addressed disciplines relevant to our research question using the key terms of “Child Advocacy Center(s),” “Children’s Advocacy Center(s),” and “CAC(s)”, including: CINAHL (KKW), PubMed (KKW, AG), SocIndex (WR), Ovid (KKW), and PsychINFO (KKW). The search was extended to include a backwards review of citations from key peer reviewed and grey literature (KKW), hand search (KKW), and recommendations from expert consultants (JH, TC, and WW).

36Article Storage and Data Collection: “Outcomes and outputs affiliated with children’s advocacy centers in the United States: A scoping review”

37While recommendations to enhance stage 2 of the Arksey and O’Malley Framework focus on identifying the relevant literature to include in the scoping review, sparse details are provided about how to store and organize the articles. Full text article review and data extraction/collection can easily become challenging when scoping reviews involve multiple articles and reviewers. Prior to selecting the studies for stage 3, it may be helpful to establish a plan for article storage and data collection (Table 2B).

38We used Research Electronic Data Capture software (REDCap) to store articles and facilitate data extraction. REDCap is a secure, web-based platform that supports data capture for research,. REDCap supports the inclusion of multiple users within projects, accommodates multiple types of data, provides audit trails for tracking data manipulation and exportation, and facilitates data exportation into many types of statistical software,. These functions allow REDCap to efficiently accommodate the process of full text review and data extraction for large research teams (Table 2B).

393) Arksey and O’Malley Framework Stage 3: Select studies

40The third stage of the Arksey and O’Malley Framework refers to the process of selecting studies for the scoping review. Different than the linear format of systematic reviews, the study selection process for a scoping review is iterative and therefore more challenging to clearly capture. Table 3A describes recommendations from the literature to enhance transparency and reproducibility of study selection for scoping reviews.

41Full text review: “Outcomes and outputs affiliated with children’s advocacy centers in the United States: A scoping review”

42Once the final sample of articles eligible for full text analysis was determined, articles were assigned by the lead author (KKW) to match with the disciplinary, research, clinical, or policy expertise of each reviewer. This was done purposefully to maximize the diverse strengths of our review team. Each team member screened their assigned articles per the inclusion criteria for full text review: the article must involve CACs, feature child victims less than 18 years of age, take place in the United States, and be written in English. Discrepancies involving article selection were discussed with the lead author (KKW) until consensus was achieved to include or exclude the article in the full text analysis. The search strategy and article selection process were reported using a PRISMA flowchart. The PRISMA flowchart is accessible as Figure 1 in our companion article, “Outcomes and outputs affiliated with children’s advocacy centers in the United States: A scoping review” (Table 3B).

43Variables for Data Extraction Framework: “Outcomes and outputs affiliated with children’s advocacy centers in the United States: A scoping review”

44Data extraction fields and variable selection selectoion were informed by the Arksey and O’Malley Framework (2005), discussion with expert contributors, and research by Peters et al, Herbert and Bromfield, and Cross et al. (2008). Variables were identified a priori to capture both the characteristics and the content of the manuscripts selected for inclusion in full text review. Manuscript characteristic variables included: title, journal, author, discipline of lead author, year of publication, and funding status. Manuscript content variables addressed the domains of: study purpose/aims, research design, population, setting, outcomes, outputs, findings, and future recommendations (Supplementary Material B).

45We defined outcome measures for CACs as those measures that are person-centered, meaning involving a person and the subject for the outcome. For each article included in full text review, we first assessed whether or not person-centered outcomes were addressed. If yes, we then documented a conceptual description of the outcome, the operational definition for the outcome, the subject of the outcome (child, family member, family unit, CAC staff or MDT member, other), and the type of outcome (mental health, physical health, forensic interview, disclosure of abuse, professional practice, knowledge, prosecution, attitudes, policy, satisfaction, other (Supplementary Material C).

46Distinct from person-centered outcomes, we defined output measures as those as findings involving programmatic or service characteristics that demonstrate functionality of the CAC model. For each article included in full text review, we assessed whether any CAC-relevant outputs were reported. If yes, we documented the conceptual definition of the output, the operational definition of the output, and the service line associated with the output (CAC staff, prosecution, forensic interview, law enforcement, economic, medical services, victim advocates, mental health, child protection services, combination, or other (Supplementary Material D).

474) Arksey and O’Malley Framework Stage 4: Extracting, mapping, and charting the data

48Stage four of the Arksey and O’Malley Framework refers to the use of a type of descriptive analytical method to conduct data charting or mapping. While this process is known as data extraction within the context of a systematic review or would be defined per specific statistical techniques within the context of a meta-analysis, a standard process of mapping or charting the data for a scoping review has not been clearly delineated. Arksey and O’Malley describe data charting/mapping as a technique to synthesize and interpret qualitative data, however the steps to replicate this technique are not articulated in the Framework. Additionally, navigating how much data to include or not include can be challenging, as can the question of how to integrate numeric data with qualitative findings. Table 4A presents a synthesis of current recommendations for stage four of the Arksey and O’Malley Framework.

49A mixed methods approach applied to scoping review methodology

50As defined by Creswell and Plano-Clark (2018), mixed method research combines methods, designs, and philosophical orientation to: 1) collect and analyze quantitative and qualitative data to answer a research question or explore a hypothesis, 2) integrate the two types of data, 3) utilize specific research designs to conduct the research, and 4) ground the research within philosophy and theory. Researchers often employ mixed method approaches to explore and corroborate results, obtain a comprehensive understanding of context prior to conducting a study, describe and compare data from multiple sources, or enhance one method by using another, such as conducting a qualitative study to better understand the results from a quantitative study. Given this, a mixed method approach is well aligned with the broad purpose of scoping reviews. Previous scholars,, address using content and numerical analyses in scoping reviews, however this has not yet been formally referred to as a mixed method approach, nor incorporated as a stage within the Arksey and O’Malley Framework (2005). Some of the advantages of incorporating a mixed approach (quantitative and qualitative) into scoping review methodology are listed below in Table 4B.

51Assessment of Interrater reliability in scoping review methodology

52Interrater reliability represents consistency or agreement among raters or reviewers. Often expressed as a correlation coefficient such as a kappa or pooled kappa, it can also be expressed as a percentage of agreement. While Daubt et al discuss a customized three-tiered approach to the study selection process, Tricco et al are the first to recommend including an assessment of interrater reliability as a means to enhance the validity of the scoping review process.

53Extracting, mapping, and charting the data: “Outcomes and outputs affiliated with children’s advocacy centers in the United States: A scoping review”

54A pilot of the full text data extraction process was conducted to assess the feasibility and acceptability of using the REDCap database. A few minor modifications were made to the REDCap database to enhance usability (drop down menus and other comment boxes) and further clarify the definitions of study variables (implementation science/research, outputs, and outcomes). Reviewers independently performed data extraction from their individually assigned articles that met the inclusion criteria and entered it into the REDCap database. Similar to the process for article selection, reviewer inquiries about data fields or data extraction were discussed with either KKW or MM. Reviewer feedback was repeatedly assessed and utilized to inform improvements in the structure and use of the REDCap database in this review.

55Interrater Proportion Agreement: “Outcomes and outputs affiliated with children’s advocacy centers in the United States: A scoping review”

56Interrater proportion agreement was assessed to determine the consistency among reviewer data entry into REDCap. A total of 10 articles were randomly selected from the sample of articles that underwent full text analysis. Each of the randomly selected articles underwent an additional full text analysis, data extraction, and data entry process by the lead author of this paper (KKW). Consistency of data entry into REDCap was compared between the original reviewer and secondary reviewer, with agreement status measured by yes or no. The interrater proportion agreement was 83%. (Supplementary Material E).

575) Arksey and O’Malley Framework Stage 5: Summarize, synthesize, and report the results

58Stage five of the Arksey and O’Malley Framework refers to the process of constructing an analytic framework that describes the breadth of the literature and identifies priority areas within that literature. Given that there are multiple steps involved in the collating, synthesizing, and reporting process, the main challenge of stage 5 is that it lacks clear guidance on how to systematically accomplish this.

59Levac et al build upon the Arksey and O’Malley Framework by suggesting three distinct steps for stage five: 1) conduct quantitative descriptive analysis and qualitative thematic analysis, 2) report the results within the context of the research question(s) and purpose, and 3) interpret the findings within the context of future research, practice, and policy. These recommendations in Table 5A address the benefit of including both quantitative and qualitative data within scoping reviews, and support the consideration of integrating a mixed method framework into scoping review methodology. Table 5B highlights additional customizations.

60Data Analysis: “Outcomes and outputs affiliated with children’s advocacy centers in the United States: A scoping review”

61Data were exported from REDCap into a Microsoft Excel spreadsheet, inspected (KKW), cleaned (KKW), and separated into quantitative and qualitative databases prior to analysis (KKW). Guided by our convergent mixed method design, the qualitative and quantitative data were analyzed separately. For the quantitative strand, we used SPSS software version 26 to calculate frequencies and percentages. For the qualitative strand, we applied Braun and Clarke’s method of thematic analysis to identify and analyze patterns in our qualitative data. The qualitative data were independently sorted and coded by two members of the research team (KKW, WR), with a third team member serving as a thematic auditor (BVW). Preliminary themes were discussed with the research team and then grouped into thematic maps. Findings from each strand were integrated and discussed within the context of trends, gaps, and implications for research, practice, and policy. (Supplementary Materials F and G).

626) Arksey and O’Malley Framework Stage 6: Integrate expert consultation

63The importance of integrating expert consultation within the scoping review process is consistently emphasized within the literature describing scoping review methods and scoping reviews of scoping reviews,,,,. Many scholars agree that inviting expert consultation into the process provides key feedback for scoping reviews and integrates knowledge that may not be accessible via academic literature, such as cutting-edge trends that may not yet be published and valuable historical context. However, recommendations vary in the literature on the timing and extent of engaging with expert consultants across the scoping review process,. Additionally, the initial Arksey and O'Malley Framework (2005) presents expert consultation as an optional component, whereas Levac et al consider this to be an essential component for scoping review methodology. Table 6A presents a synthesis of the literature for stage 6 of the Arksey and O’Malley Framework.

64Expert consultation: “Outcomes and outputs affiliated with children’s advocacy centers in the United States: A scoping review”

65The rapid review of the literature from the first stage of the Arksey and O’Malley Framework identified leading authors in the field of research on Children’s Advocacy Centers. Three experts were selected and consulted early and throughout the scoping review process. All three experts (JH, WW, TC) kindly offered valuable personal insights as well as additional relevant literature to add to the search that positively impacted the trajectory of the scoping review (Table 6B).

66During the initial planning phase for “Outcomes and outputs affiliated with children’s advocacy centers in the United States: A scoping review”, we strongly felt the need to integrate a team based, mixed methods approach. The existent body of literature on scoping review methodology and scoping reviews on scoping reviews informed our process, however the literature rarely focused on actionable steps to enhance teamwork or clearly articulated a means to move beyond separate analyses of numeric and thematic data. While many scholars indirectly refer to using mixed methods within scoping reviews, few articles clearly described the scoping review process as a mixed method approach. Like Daubt et al, we found the process of engaging in a team-based approach to be extremely helpful in gaining a deeper understanding of the key themes surrounding outcomes and outputs affiliated with CACs. We also found that designing a framework that collected quantitative and qualitative variables and created the space to compare the results from numerical and content analyses enriched our findings. In conclusion, this article contributes to the discourse towards advancing scoping review methodology by including enhancements to the scoping review methodology that emphasize a team-based, mixed method approach. Table 7 represents a synthesis of customizations that can be used to enhance a team-based, mixed approach to scoping methodology.

67X The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.