What is an evidence map? A systematic review of published evidence maps and their definitions, methods, and products

1There is growing variation in evidence synthesis methodology to meet the different objectives evidence synthesis can support. The classic systematic review and meta-analysis are both rigorous and produce detailed information about narrow questions, but they are resource intense and the work burden limits the scope of what can be covered. To meet a variety of user needs, offshoots of the classic model have been developed within the evidence synthesis realm; for example, rapid reviews cater to more urgent deadlines but may not adhere to all the methods of a systematic review, scoping reviews accommodate larger bodies of literature for which detailed synthesis is not needed, and realist reviews specialize in exploring how complex interventions work and frequently include evidence excluded from classic systematic reviews.

2These new variants on the classical systematic review are at various phases in development. Determining the unique contributions and methods of each of these new synthesis method offshoots is a challenge. Systematic reviews and meta-analyses have a standardized process for conduct and reporting, codified in the Institute of Medicine standards and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting guidelines. The Realist And Meta-narrative Evidence Syntheses: Evolving Standards (RAMESES) publication standards for realist syntheses and meta-narrative reviews were published in 2013, and scoping review reporting guidance is underway as of 2014 using the Enhancing the QUALity and Transparency Of health Research (EQUATOR) Network. Rapid reviews, also sometimes referred to as evidence summaries, have received increased interest, with a journal series and dedicated summit in 2015, as well as multiple articles on rapid review methodology, but no official standards have been released.

3Evidence mapping is the newest of these new evidence review products. In 2002, there were no published evidence maps, and as recently as 2010, only ten such publications could be identified. In addition, evidence mapping has yet to undergo the scrutiny and development of these other methodologies, and it is not clear if those authors who are using the term are using a methodology unique from other developing methods. Both evidence maps and scoping reviews set out to map the literature. A scoping review recently was described as “a form of knowledge synthesis that addresses an exploratory research question aimed at mapping key concepts, types of evidence, and gaps in research related to a defined area or field by systematically searching, selecting, and synthesizing existing knowledge”. They recommend a process described by Arksey and colleagues and enhanced by Levac and colleagues; these steps were also referenced in other publications comparing methods of evidence synthesis as the standard for scoping reviews. Thus, there is some consensus surrounding the scoping review method and its components. A 2013 publication attempting to distinguish between the two methodologies concluded that scoping reviews include “a descriptive narrative summary of the results” whereas evidence maps identify evidence gaps, and both use a tabular format to depict a summary of literature characteristics. Furthermore, multiple publications lay out differing recommendations for evidence maps, complicating this effort. Hence, earlier attempts by Schmucker and Snilstveit to characterize evidence map methods and products relied on small numbers of evidence map publications in determining their results and reached conflicting conclusions. Schmucker and colleagues' review included seven evidence maps; Snilstveit discussed three examples while primarily focusing on distinguishing these from what they termed an evidence gap map. Since then, more evidence maps have been published, increasing the ability to determine what constitutes an “evidence map,” either in terms of methods or products. In 2014 alone, eleven evidence maps were published.

4Our objectives are to identify and systematically review published evidence maps and to assess commonality and heterogeneity and to determine whether additional work is warranted to try to standardize methods and reporting. We compare and contrast the presented definitions of evidence mapping, the domains used to classify data in evidence maps, and the form of the evidence map takes.

5The librarian on our team, RS, conducted an electronic search of ten databases (PubMed, PsycINFO, Embase, CINAHL, Cochrane Database of Systematic Reviews (CDSR), Cochrane Central Register of Controlled Trials (CENTRAL), Cochrane Database of Abstracts of Reviews of Effects (DARE), Cochrane Methodology Register (CMR), SCOPUS, and Web of Science) from inception to 8/21/2015 for publications relating to our objective by using the search terms "evidence map" OR "evidence mapping" OR "evidence maps" OR "mapping evidence" OR “evidence map*” in a search of titles and abstracts. Because evidence mapping is a relatively new method in the biomedical literature synthesis repertoire, no Medical Subject Headings (MeSH) term is available. There were no language restrictions or restrictions on study design. Additional studies were identified through reference mining of identified studies and expert recommendations.

6Two reviewers independently screened titles and abstracts for relevance and obtained full text articles of publications deemed potentially relevant by at least one reviewer. Full text articles were screened against predetermined inclusion criteria by two independent reviewers, and any disagreements were reconciled through team discussion. To be included, authors must have presented results (i.e., no protocols were included) with a process called evidence mapping or figure called an evidence map. Because of the methodological focus of this review, any patient population, intervention, comparator, outcome, and setting were included.

7For all included publications, the following data were abstracted: use and definition of the term “evidence map,” research question or aim, search methods and years, number of citations included in the evidence map, and country of origin. The unit of analysis was also abstracted, since some maps considered all literature citations for inclusion, whereas others included systematic reviews only or aggregated all publications originating from the same study into one unit. For publications presenting maps, the domains used to classify studies in the map were also abstracted (e.g., interventions, outcomes, and literature size). Data were abstracted by one reviewer using a standardized form and verified by the second reviewer. The form was piloted and refined by both reviewers prior to abstraction. We applied no quality assessment criteria since we are unaware of any that exist for evidence map methods.

8Data were narratively synthesized in three sections, discussing key characteristics of definitions presented, domains used to classify literature in the mapping process (e.g., interventions), and what form an evidence map or evidence mapping methodology takes. Within this latter section, publications were grouped by whether they (1) presented a figure or table explicitly called an “evidence map” or called the results evidence mapping in the publication itself, (2) referred to an online database as the evidence map, or (3) said they used a mapping methodology but did not present a figure or table. No statistical analysis was planned or executed, given the focus on reporting of methods rather than a specific outcome and the heterogeneous nature of the included health topics. We created tables to summarize data on included articles to support the narrative synthesis.

9The total search results identified 145 titles through 8/21/2015. Reference mining and expert recommendations yielded one title each for a total of 147 titles for screening. Of these titles, we included 53 potentially relevant publications after title and abstract screening. Fourteen of these publications were rejected after full text review. Four of the excluded publications identified themselves as another type of synthesis (i.e., scoping review, realist synthesis, or systematic review) in their titles. Three excluded publications presented discussions of evidence mapping and have been incorporated into discussion where relevant. Two publications used the term “evidence map” outside the evidence synthesis methodology context (e.g., in the context of reporting the results of a hazard assessment). Another two publications used evidence from an evidence map that was created in a separate project, one full text publication was not available, one publication was a duplicate citation with another previously screened citation, and the final study excluded explicitly stated that they used a standard systematic review protocol developed according to systematic review guidelines. Having met inclusion criteria, thirty-nine publications were included at the full text review. Of these, 34 publications presented evidence maps explicitly and five publications used a mapping methodology without presenting a map (see Fig. 1 for literature flow and Additional files 1 and 2 for PRISMA checklist and flow). The publications with explicit maps included those that presented the map in the publication (N = 26), and those that discussed a map only available online (N = 8).

10Included publications came from the USA (N = 19), Australia (N = 10), the UK (N = 7), Canada (N = 2), and Japan (N = 1). Most publications are very recent, with no publications before 2003 (see Fig. 2). Since the last systematic review of evidence maps, conducted by Schmucker and colleagues during 2013, an additional 24 evidence maps have been published, doubling the literature quantity. More details on data extracted from included publications can be found in Table 1.

11Although a variety of topics are represented in the 39 included evidence map publications, four broad topical areas account for nearly three quarters of the publications: mental health and learning disability related topics account for 11 of the publications (28 %), complimentary alternative medicine and nutrition-related topics account for seven publications each (18 % each), and traumatic brain injury and spinal cord injury account for four publications (10 %). Within the mental health group, four of the 11 publications come from the headspace research group, described in more detail below. Six of the seven complementary and alternative medicine publications come from the Veterans Affairs Evidence Synthesis Program, of which the authors of this review are a part. Three of the seven nutrition-related publications come from a research group based at Tufts University, and all four of the traumatic brain injury/spinal cord injury publications are a part of the Global Evidence Mapping (GEM) project. Taken together, these four research groups account for 44 % of the published evidence maps included in this review (17/39).

12Of the 39 included studies, eight were associated with two evidence mapping projects: GEM and headspace; these publications were grouped by project in the following discussion since publications from the same project employ the same definition. In two instances, there were two publications from a single mapping project, and these were also grouped, resulting in two evidence maps from these four publications. One publication provided neither definition nor citation. Most studies did not explicitly outline a definition of “evidence map,” and often the research aims were used to capture an implicit definition (see Table 1 for research questions or aims for all included studies).

13Thus, of the 31 evidence maps with elements of a definition, the most commonly stated component of the definition was a review of evidence to identify gaps or future research needs (67 %, 21/31). Another common component was that the process engage the audience and/or produce user-friendly products (58 %, 18/31). In emphasizing the user-friendly aspect, one definition stated that evidence maps should “provide access to user-friendly summaries of the included studies”, while another described evidence maps as “mak[ing] these vast bodies of literature accessible, digestible, and useable”. Many definitions also qualified evidence maps as capturing a broad field (55 %, 17/31). Two components were less often explicitly stated in the definition, having a systematic process and visual depiction (48 %, 15/31 and 23 %, 7/31, respectively), but all included publications used a systematic process (i.e., documented search strategy and inclusion criteria) and most incorporated a visual depiction of the data as well (84 %, 26/31).

14Only one evidence map definition explicitly stated all five of these components (i.e., identify gaps or needs, audience engagement/user-friendly products, broad field, systematic process, and visual depiction). Three other evidence maps met all criteria when their inclusion of a visual depiction was considered a part of the definition implicitly. Seventeen of the evidence maps included four of the five definition components, including those that used a systematic process or visual depiction without explicitly stating this. Thus, 68 % (21/31) evidence maps with definitions included four or five of the five most common components.

15Most of the 31 evidence maps did not provide any citations when referencing evidence mapping methodology (N = 15). Citations occurring more than twice included Arksey and colleagues (N = 8), the GEM project (N = 7), the headspace project (N = 5), and Katz and colleagues (N = 3). The publication by Arksey and colleagues provides a description of scoping review methods but was still the most cited article. The GEM and headspace projects are both large initiatives that have funded the exploration of a very broad topic, neurotrauma and mental health disorders affecting youth, respectively. The publication by Katz and colleagues, the oldest of the publications we identified, was published in 2003, at least 4 years before any of the other identified mapping publications. This publication introduces the term, as well as a nine-step process: identify and convene the appropriate experts, apply expert opinion to define the region of evidence to be mapped, establish the coordinates to be used for positioning within the map, define the map boundaries in terms of pertinent coordinates, search the relevant “terrain,” draw the map, study the map to identify any needed revisions and to establish priorities for detailed assessments, perform detailed assessments in priority areas, and generate reports summarizing the “lay of the land.”

16Only publications that presented an evidence map as a figure or online database were considered relevant in determining domains, or data elements used to classify literature, since these publications were the ones with data presented for such domains. Of the 34 publications, 26 presented the maps in the publication itself, and eight were associated with an online map. As in the definition section, duplicate publications from the same evidence mapping project were collapsed together, leaving a total of 26 evidence maps with domains for this synthesis.

17Thirteen evidence maps categorized their literature by the amount of literature relevant to a particular domain (literature size domain, 50 %, see Table 2). Other popular domains included intervention (N = 12, 46 %), study design (N = 10, 38 %), sample size (N = 10, 38 %), disorder/condition (N = 9, 35 %), and outcomes (N = 9, 35 %). Some maps grouped literature in subdomains within these larger domains, such as groups of conditions and then individual conditions within those groups. The average number of domains captured in an evidence map was four, the smallest number being two (i.e., outcomes and elements of service delivery) and the largest being seven (i.e., population characteristics, intervention, outcomes, setting, study design, sample size, and disorder/condition).

18Three main versions of an evidence map were found: publications with a visual representation of data in the publication (N = 26); publications that referenced a database housing data virtually that can be queried (N = 8); and publications that employed a process or methodology leading to recommendations or synthesized result (N = 5).

19Twenty-six publications explicitly include figure(s) or table(s) which are referred to as evidence maps. They display a range of potential map formats, some quite similar to a classic systematic review evidence table or literature flow diagram. These formats incorporate various numbers of characteristics and types of characteristics. Details of map formats can be found in Table 2. As in prior sections, two projects with two publications each are collapsed into two groups, and thus, 24 evidence maps are discussed in this section.

20Most of the 24 published evidence maps used some variant of a cross-tabular format for their main findings, with counts or sums of publications arrayed across various domains (N = 10, 42 %). Eight bubble plots (33 %), two flow charts (8 %), and two bar charts (8 %) were used as the main findings diagram in evidence mapping publications. The last two publications arrayed included studies on a conceptual framework (8 %) and mapped the evidence on the relationships described in the model. Most publications presented data in more than one table or graphic (N = 16, 67 %), either with subsets of data in each map (e.g., one table for condition A and another for condition B) or with different domains covered in the different maps (e.g., one graphic describing population demographics covered by the literature and another with outcomes cross-tabulated with interventions of included studies). Figure 3 presents a figure simulating a bubble plot style evidence map as an example of one of the more commonly used formats.

21Eight publications discuss evidence maps in the context of online searchable databases. Four publications discuss GEM and four publications discuss headspace. Of the four publications discussing GEM, one provides the overview of the initiative and lays out the GEM mapping method, one discusses search strategies that the GEM group has developed, and the final one discusses the process of developing priority research areas for which evidence maps can be generated. For this group of publications, there is one product: the searchable online GEM database.

22In the case of the headspace publications, there is an overview publication that lays out the history and scope of the headspace program and their mapping methods, but each of the three additional publications presents data on a subsection of the overall evidence map. While the overall program aims to improve evidence-based practice in youth mental health, the subtopics of interventions for depression, suicidal and self-harming behaviors, and psychosis are explored in depth in the latter publications, making it less clear if the overall database is the evidence map, or if the data presented within the topic-specific publications are the evidence maps. Although none of the figures are titled “evidence map,” they appear similar to some of those included in the prior section.

23Five publications used a mapping methodology without presenting an explicit map. With or without producing either of the first two conceptions of an evidence map, the third idea was presented or discussed in all included publications. The process, either discussed or executed, usually included a search of multiple databases with standard search terms, discussion of inclusion criteria, and inclusion of stakeholders in defining and/or refining the scope of the product.

24The principal conclusion of our evaluation of studies that call themselves “evidence maps” is that the implied definition, as defined by a majority of studies reporting individual components, of what constitutes an evidence map is a systematic search of a broad field to identify gaps in knowledge and/or future research needs that presents results in a user-friendly format, often a visual figure or graph, or a searchable database. However, only four of 31 studies (12 %) satisfied all five of these components. Thus, the heterogeneity in methods and stated definitions means that stakeholders cannot necessarily know what to expect if they commission an evidence map or seek to identify existing maps.

25Of all literature synthesis methods, this evidence mapping definition shares many similarities with the definition or goals of a scoping review. Both seek to review broad topics, often with the stated goal of identifying research gaps and areas for future research. In evidence mapping publications, the most often cited article for evidence mapping methods is a scoping methods publication. Compared to a scoping review, using the framework suggested by Colquhoun and colleagues, the methods used in the evidence mapping publications are, on a whole, very similar. The main distinctions seem to be the involvement of stakeholders early in the research process, the rigor of the search strategy (e.g., all mapping publications describing systematic searches of online databases), and the production of a visual or searchable database, with the stated goal that such products are more “user-friendly” or digestible. Because neither methodology has established reporting guidelines, it is difficult to determine where one method ends and the other begins.

26In terms of the “map,” the most common ways of organizing the data into a visual representation were using a cross-tabular format and categorizing literature according to interventions and/or study designs present. However, the domains chosen to display and means of presentation will necessarily vary for any particular map according to the aims of the review.

27Because there is no standard search term or repository for evidence maps, we may not have been able to identify all evidence maps through our search methods. Thus, our findings may be biased to represent those maps that were readily available on the ten searched databases or were cited in other sources we found.

28With growing numbers of publications using the term “evidence map,” clarifying scoping and evidence mapping methods is an important topic for stakeholders so that they will know what to expect when commissioning, conducting, and interpreting results of an “evidence map.” A key part of this effort will be developing reporting guidelines for these methods. This is already underway for scoping reviews, but given the similarities between the two methods, working on both methods in tandem may produce more clear and distinct results.

29A key strength of the evidence mapping method is the use of visuals or interactive, online databases. Keeping these data up to date and available online may prove to be a challenge in rapidly growing fields, and new audiences may be exposed to these resources who are unfamiliar with how to best make use of the products. As evidence synthesis methods evolve to meet modern demands, they must also meet new challenges.

30The principal conclusion of our evaluation is that the implied definition of what constitutes an evidence map is a systematic search of a broad field to identify gaps in knowledge and/or future research needs that presents results in a user-friendly format, often a visual figure or graph, or a searchable database. However, there is diversity in methods and stated definitions, so that, as it stands now, stakeholders cannot necessarily know what to expect if they commission an evidence map or seek to identify existing maps.