Approaches, tools and methods used for setting priorities in health research in the 21st century

1Apart from the continuing need to prioritize investments in health systems and health interventions, there is also a need to prioritize health research. Health research is difficult to prioritize, because the number of possible competing ideas for research is large, the outcome of research is inherently uncertain, and the impact of research is difficult to predict and measure. A systematic and transparent process to assist policy makers and research funding agencies in making investment decisions is a permanent need.

2At national level several methods have been tried: some of the best examples are the Council on Health Research for Development’s approach (COHRED) in Brazil, Cameroon, Peru and Philippines; the Essential National Health Research (ENHR) approach in Cameroon and South Africa; and the Combined Approach Matrix (CAM) in Malaysia, Pakistan and Argentina. COHRED, ENHR and CAM were all developed by committees set up by international agencies, such as the World Health Organization (WHO) or the Global Forum for Health Research (GFHR). These methods are useful for organizing the available information so that the research prioritization can take place.

3To obtain a better understanding of the landscape of approaches, tools and methods used to prioritize health research I conducted a methodical review of the PubMed database covering the period 2001–2014. My primary aim was not to perform an exhaustive review of the field, which would include searching all available scientific databases and grey literature. Instead, I was interested in identifying the methods and tools that are being commonly used in the papers that are most readily accessible through databases in the public domain such as PubMed, and to assess their relative importance and applicability. The review of PubMed for the period between 2001 and 2014 achieves this aim, because this limits the search of priority–setting tools to health topics only, which is the main interest of this analysis, while drawing on a very large database which is publically available and which should contain the vast majority of relevant studies.

4My search terms included “research priorit* OR priorit* research”. These terms were chosen as the most informative combination of search terms after experimenting with several versions of search terms. The search terms identified 343 publications, 138 of which were excluded from the analysis because their contents were irrelevant to health research priority setting. A further 40 studies were excluded because they were review articles which did not attempt to set priorities. In total, 165 relevant studies were identified and retained for the analysis. Figure 1 shows a flowchart of the review on all research priority setting exercises conducted between 2001 and 2014.

5Approximately 12 exercises were initiated each year between 2001 and the end of 2014. Since 2012, there has been a steady increase in the number of exercises published with the peak in 2014 with 34 exercises published (Figure 2). Of the 165 publications identified, the most frequently used was the CHNRI method (26%), followed by the Delphi method (24%), James Lind Alliance method (8%), the Combined Approach Matrix (CAM) method (2%) and the Essential National Health Research method (<1%). COHRED method, although frequently mentioned and clearly described in the historic context of national–level research priority setting, was not underlying any specific priority–setting process in the time period which I studied. Online surveys that were not accompanied by face–to–face meetings were used in 8% of studies, while 9% used a combination of literature review and questionnaire to scrutinise the research options for prioritization among the participating experts. About 3% of studies reported no clear process and provided very little information on how priorities were set. A further 19% used a combination of expert panel interview and focus group discussion (“consultation process”) but provided few details, while a further 2% used approaches that were clearly described, but not established as a replicable method (Figure 3). At this point, I would like to clarify that “replicable” refers to the method's description in sufficient detail, so that all other users could apply it in the same way. It does not refer to method's property to yield the same results when repeated, which is a different meaning of the term “replicable” when assigned to a method.

6Tables 1 to 6 provide a brief description of the approaches and processes used by the specific methods mentioned in Figure 3. The methods range from those that are not described at all, through vaguely described processes of group decision making, to those that follow a certain structure/process and use transparent criteria. Their output is typically quite general, ie, pointing to broad research areas in which more research activity is needed. As described above, COHRED, ENHR and CAM are used in assembling the evidence that can be used for the consultation but not for the ranking of priorities. Nevertheless, the use of any method, regardless of its limitations, is preferable to the alternative of having no clearly defined approach at all.

7Among the 165 identified studies that set health research priorities, 21% were conducted at global level, 50% of the exercises were focused on High Income Countries (HICs) and 28% were focused on Low and Middle Income Countries (LMIC). At the national level, the countries where research priority exercises were most frequently initiated were the UK (27%), USA (16%), Australia (15%), and Canada (11%) (Table 7).

8Topic areas for which research priorities were identified included non–communicable diseases (18%), followed by child and adolescent health (17%), mental health (10%), nursing/midwifery (8%) and infectious disease (8%). The remaining exercises (39%) covered a wide variety of topics, including policy and health system, occupational health /therapy, reproductive health/women’s health, emergency care, environmental health, occupational health, forensic science and injury prevention (Table 7).

9The number of priority setting exercises in health research published in PubMed–indexed journals is increasing, especially since 2010. These exercises are being conducted at a variety of different levels, ranging from the global level to the level of an individual hospital. With the development of new tools and methods which have a well–defined structure – such as the CHNRI method, James Lind Alliance Method and Combined Approach Matrix – it is likely that the Delphi method and non–replicable consultation processes (see the definition of “replicable” earlier in the text) will gradually be replaced by these emerging tools, which offer more transparency and replicability. This is a process that should be endorsed, as a natural progression of the priority–setting field from the period in which hardly any structured processes existed to fill a need, to the new era which will be increasingly dominated by structured and well–defined tools.

10This review is not the first attempt to assess approaches, tools and methods to set health research priorities. Searching the literature, I identified five earlier attempts to review and discuss priority–setting processes. The first review was published by Rudan and colleagues in 2007 in an attempt to develop an evidence base for the development of conceptual framework and guidelines for implementation of the CHNRI methodology. This paper identified ambitious attempts by several large organizations at the international level to define health research priorities for either the whole developing world, large world regions or nationally. These attempts date back to the year 1990, with the “...Commission on Health Research for Development usually being referred to as the first truly significant international initiative aimed toward systematic approach to setting priorities in global health research.” Other initiatives that followed were the “Ad Hoc Committee (AHC) on Health Research Relating to Future Intervention Options” (in 1994), the “Global Forum for Health Research” (in 1998), the “Council on Health Research and Development (COHRED)” (in 2000), “The Grand Challenges” proposed at the World Economic Forum in Davos, Switzerland (in 2003) and the “Combined Approach Matrix” as the first specific priority–setting tool for health research (in 2004). The paper concluded that the processes, initiatives and tools fell short of being informative on what the specific research priorities should be and how exactly are they derived.

11In 2010, Viergever et al. reviewed the articles that set health research priorities and they specifically reviewed exercises coordinated by World Health Organization Headquarters since 2005. This resulted in the total of 230 documents or reports, many of them unpublished (hence, not included in my review). The authors concluded that, at that point in time, there was no “gold standard” approach for health research prioritisation. This was not surprising, given the heterogeneity in the context of research prioritization exercises and different levels at which they were being conducted. Nevertheless, the authors observed several common themes of “good practice” and proposed a generic framework – in the form of “checklist”, like a form of “guidelines” – which also suggested various options for each step of the process. Nine themes were identified through a review of the previously conducted priority–setting processes. They were categorized as the “themes during the preparatory work” (defining context, use of comprehensive approach, ensure inclusiveness of participants, information gathering, planning for implementation), followed by the steps in the process of deciding on the priorities (defining the criteria, methods for deciding on priorities), and two steps in the last phase after the priorities have been set (plan the timing of evaluation in terms of how the research priorities are being used, and write the clear report of the methodology used to ensure the transparency in the process). The authors proposed that the provision of the framework should be of assistance to policy makers and researchers. It could have a dual role: it could not only assist priority–setting process, but also planning the follow up and implementation of the priorities.

12In the same year, in 2010, the World Health Organization's Department for Research Policy and Cooperation held a consultation between methodology–developing experts to identify optimal characteristics of priority–setting methods that could be applicable at the national level. The aim was to empower low and middle–income countries to take more ownership of their own health research agenda. Tomlinson reviewed the progress made at this meeting and published the main conclusions in 2011. Three methods emerged as applicable at the national level: the Combined Approach Matrix (CAM), the Council on Health Research and Development (COHRED) and the Child Health and Nutrition Research Initiative (CHNRI). The authors presented and discussed strengths and weaknesses of each method. They also noted that, across the countries surveyed, genuine engagement of stakeholders was difficult to achieve and was typically missing. Countries also varied in the extent to which they would document priority–setting processes, with not a single country having an appeal process for outlined priorities. Another problem was that the identified priorities usually outlined broad disease categories, rather than more specific research questions. The authors concluded that priority–setting processes should aim to include mechanisms for publicizing results, effective procedures to translate and implement decisions and processes to ensure that the revision of priorities eventually does occur.

13In a more recent report, an independent team from the Kirby Institute in Sydney, Australia, systematically reviewed all studies undertaken in low– and middle–income country (LMIC) settings that attempted to set research priorities over the period from 1966 to 2014. The studies included were not reported but they found 91 studies, including 16 which used the CHNRI method. The authors concluded that almost half of these processes took place at the global level (46%). For regional or national initiatives, a half focused on Sub Saharan Africa (49%), followed by East Asia and the Pacific (20%) and Latin America and the Caribbean (18%). Most commonly, studies were initiated by an international organization or collaboration (46%). Researchers and governments were the most commonly represented stakeholders. The most frequently used process was a conference or workshop to determine priorities (24%), followed by the CHNRI method (18%). The review revealed inconsistent use of existing methods and approaches in health research prioritization processes. It also showed that while there was strong involvement of government and researchers, participation of other key stakeholders was limited. The authors argued that many processes, regardless of the method used, lacked an implementation strategy to translate the result of the process into implementation of research projects. Finally, the authors concluded that research prioritization exercises would often remain “one–time exercises”, given the lack of follow up and implementation strategies involving the funders, researchers and government officials.

14Finally, in 2014, as a part of the Lancet series on increasing value and reducing waste in health research generally, one paper of the series (by Chalmers et al.) explored how to increase value and reduce waste when research priorities are set. The group of authors argued that many basic research endeavours do not lead to knowledge that is useful to the end user of the research results. By using various examples, the authors reiterate the same argument: if research does not meet the needs of the users of research, evidence will have little impact on public health and clinical practice. The authors argue that many research studies that fall in the area of basic (fundamental) research were duplicative. Although a replication of positive findings is a welcome process, an excessive repetition of conducting similar research can be prevented by either: (i) conducting systematic reviews and also involving the end user of the research as well as clinicians in the process (where they used the example of hospital based research priority setting exercise using the James Lind Alliance method); and (ii) mapping research portfolios of major agencies, that could help to prevent duplication in the nature of supported research. The main message of the article is, therefore, a need for better co–ordination among the researchers and the funders over the research that is being conducted and increased focus on the translational value of the information that is being generated through research.

15It is evident from my own methodical review, and from the systematic review undertaken by the researchers from the Kirby Institute, that there is a need for a transparent, replicable, systematic and structured approach to research priority setting, because the large majority of the previous exercises were not based on processes meeting all of these criteria. The review by McGregor et al. shows how, although a very recent addition to the set of tools, the CHNRI method is set to become the most widely used approach.

16The results of my review broadly confirmed the observations of all previous reviews, with an additional insight into time trend – showing an increase in the number of exercises conducted over time, and gradual replacement of poorly defined processes with those that use particular methods and tools, as shown in Figure 2. The next step in the field of health research priority setting should therefore involve monitoring whether any single method may address the need for most exercises conducted at different levels, or if better results may perhaps be achieved through combination of strengths of several methods.