Methods for the guideline-based development of quality indicators--a systematic review

1According to the definition of the Institute of Medicine (1990), quality of care is the "degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge". Increasingly, quality indicators (QIs) are employed to assess and improve the quality of care in many healthcare settings. QIs are measurable items referring to structures, processes, and outcomes of care. They imply a judgment on the quality of care provided. However, the interpretation of such performance assessments can have far-reaching consequences, for instance, in application to pay-for-performance models. Hence, the development of QIs should be based on a systematic approach that ensures transparency and produces high-quality standards. Important attributes of high-quality QIs are their relevance to the selected problem and field of application, their feasibility, and their reliability. They further need to be easily understandable for providers and patients, changeable by behavior, achievable, and measurable with high validity. To ensure content and construct validity, QIs need to be evidence based and should have a strong correlation with the actual quality of care provided, respectively. The reliability of QIs in regard to their level of measurement error can be assessed by an evaluation of the intra- and inter-observer reliability.

2State-of-the-art methodological approaches to QI development have been described in several studies, and a large body of literature exists evaluating their strengths and limitations. However, to date, no study of which we are aware exists that systematically compares different methodological approaches to QI development with respect to their ability to generate QIs that improve the quality of the particular healthcare aspects they were designed for.

3Developing QIs is an expensive and time-consuming process. They are usually specific to certain healthcare settings and, as a result, cannot always be applied to other settings without an adequate adaption process. A time-efficient and resource-saving approach is either to generate QIs from clinical guidelines already available or to couple the process of guideline development with the formulation of appropriate QIs. Due to the aim of clinical practice guidelines to improve quality-of-care processes in practices and care institutions, guideline-based QIs predominantly relate to process quality. However, no gold standard exists for guideline-based QI development.

4Blozik et al. recently conducted a survey among members of the Guideline International Network (G-I-N [Guidelines International Network, Perthshire, Scotland]) that shows that even among working groups specializing in guideline and QI development, a wide variety of methodological approaches are used. A gold standard would help to standardize procedures, foster transparency, and improve efficiency of resources used.

5This review aims to identify, describe, and compare methodological approaches to guideline-based QI development. By pooling the available knowledge and appraising strengths and limitations, we intend to provide the groundwork necessary for defining a gold standard for the development of QIs from clinical practice guidelines. To achieve this, we addressed the following research questions:

61. Which methodological approaches to guideline-based development of QIs have been described so far?

72. What are the strengths and limitations of the methodological approaches described regarding their ability to generate high-quality QIs?

83. Do methodological approaches to the development correlate with the quality of QIs they produce?

9We carried out a systematic literature search across three electronic databases: MEDLINE (US National Library of Medicine, Bethesda, MD, USA), the Excerpta Medica database (Embase [Elsevier B.V., New York, NY, USA]; both via OvidSP® [Ovid Technologies, Inc., New York, NY, USA]) to cover articles in medical journals that are not included in MEDLINE, and the Cumulative Index to Nursing and Allied Health Literature (CINAHL [EBSCO Publishing, Ipswich, MA, USA]) to include articles published in the field of nursing and the allied health professions. The query date of all three databases was April 22, 2010. The search included literature from the earliest records available in the databases up to the search date. Duplicates were eliminated both manually and automatically. To identify articles for review, we linked three search columns using the Boolean operator "and": quality indicators, guidelines, and development. We combined several search terms with the Boolean operator "or" in order to operationalize the search terms (the MEDLINE search algorithm can be found in Additional file 1: Table S1 and was slightly adapted for Embase and CINAHL). We drew several search terms from the controlled vocabularies used for subject indexing in MEDLINE (i.e., Medical Subject Headings [MeSH]), Embase (i.e., EMTREE), and CINAHL (i.e., CINAHL Subject Headings). We searched three databases for ongoing studies (Current Controlled Trials [Springer Science & Business Media, New York, NY, USA], HSRProj [Health Services Research Projects in Progress, US National Library of Medicine, Bethesda, MD, USA], UKCRN-Portfolio [United Kingdom Clinical Research Network, National Institute for Health Research, London, UK]). In addition, we screened the reference lists of all retrieved publications included in the final review. From the relevant literature and the G-I-N database, we derived contact information of institutions and working groups in the field of guideline and QI development. We scanned relevant government and institutional websites in order to obtain web-published documents such as method papers (for details of websites searched, see Additional file 2: Table S2). Finally, we consulted colleagues with a research interest in QI to point out articles not identified during our database, websites, and reference list search.

10Two reviewers independently screened all obtained references for eligibility in a three-stage screening process. Discrepancies were solved by consensus. Articles were considered for inclusion if they reported at least one methodological approach to guideline-based QI development and if they were published in English, French, or German. All study and publication types were included.

11The detailed reporting of the individual development steps (see next paragraph) in publications describing methodological approaches to QI development is indispensable for their reconstruction--be it for the purpose of process evaluation (as we did) or in order to apply methodological approaches to QI development in other settings. We therefore excluded studies at the full-text screening stage that did not describe the extraction of recommendations from clinical guidelines in detail, as this was the process of particular interest to this review. Details of the selection process, including exclusion criteria at the abstract-screening stage, are summarized in Figure 1.

12Two researchers independently extracted data from the relevant literature to a predesigned data extraction form (see Additional file 3: Table S3); discrepancies were solved by consensus. In order to describe and to compare methodological approaches to guideline-based QI development, we developed an a priori framework of the QI development process. For this purpose, we identified six steps that most methodological approaches to guideline-based QI development have in common with regard to function and succession but that differ in their design from one methodological approach to another. Through a preliminary search and analysis of a select number of key publications, we identified six development steps: (1) topic selection, (2) guideline selection, (3) extraction of recommendations, (4) QI selection, (5) practice test, and (6) implementation (see Figure 2). The data extraction form was specifically designed to include (a) information about the methodological approach to these six development steps and (b) items necessary to perform a quality assessment of the relevant studies. For steps 1 to 4, we extracted information about how and by whom the specific development step was conducted, such as selection criteria for topics, guidelines, and recommendations, as well as participants. The two development steps specific to guideline-based QI development (compared to QI development from other sources) were investigated in more detail, namely, guideline selection and extraction of recommendations. In addition to the above-mentioned selection criteria, we collected information about the selected guidelines (Was some sort of quality assessment conducted? Were all selected guidelines listed in the publication?), as well as the extracted recommendations (Were they reported at all? If yes, were the source guideline and the underlying level of evidence made transparent?). For an overview of all selected information on guideline selection and extraction of recommendations, see Table 1.

13Due to the wide variety of study and publication types and the overlap of the quality assessment and the assessment of methodological approaches, we limited the quality assessment to items covering funding information, the reporting of study and publication type, and the reporting of duration and time frame of the study.

14Following data extraction and identification of the methodological approaches to each of the above-listed development steps, we focused on analyzing the similarities and differences among the identified methodological approaches. The results are presented following further elaboration of the six development steps introduced above. We discuss our results in context of the current literature in the Discussion section.

15We identified a total of 8,697 potentially relevant articles, of which 8,468 were excluded based on their titles or abstracts (see Figure 1 for details regarding the screening process). No additional articles were identified through expert consultation. We conducted full-text reviews of the remaining 229 articles and an additional eight articles identified in reference lists and in the grey literature. The final review included 48 articles.

16Of the 48 articles in the final review, 10 papers described methodological approaches to guideline-based QI development in general (referred to as "method papers"), and 32 articles addressed the guideline-based QI development for a certain clinical topic (referred to as "topic papers"). An additional six papers comprised a detailed description of a method as well as its application for a certain clinical topic (referred to as "method + topic papers"). None of the selected publications was a controlled study comparing one development method to another. All journal articles were published in English; two of the method papers published via institutional websites were written in German.

17In not disclosing the funding source and time frame of the study and in not explicitly reporting the study type, many of the publications did not meet our basic quality-assessment criteria (for details, see Table 2).

18The identified relevant studies originate from many different institutions and working groups, only a few of which have published more than one relevant study on guideline-based QI development (e.g., the Dutch IQ healthcare [University of Radbound, Nijmegen, The Netherlands]).

19Tables 2, 3, and 4 provide an overview of the characteristics of all included publications. Figure 3 provides a comprehensive overview of all methodological approaches identified.

20Unless indicated otherwise, numbers of studies referred to in the following paragraphs always relate to all 48 studies of the final review pool.

21Criteria for the selection of a clinical topic for QI development were detailed in 33 publications. The most frequently reported criteria were

22• the public health relevance of a topic (mentioned in 18 publications),

23• the existence of a gap between potential and actually achieved quality of healthcare (mentioned in 16 publications).

24Other reported criteria were uncertainty about the quality of care provided for a specific healthcare setting (mentioned in six publications), the economical impact of a specific healthcare problem (mentioned in six publications), and the individual impact on the quality of life (mentioned in four publications).

25In 16 studies, QIs were developed from a single guideline, whereas in seven studies more than one guideline was used to derive QIs. Twenty studies detailed other sources, such as existing QI databases, in addition to clinical guidelines.

26Only eight of the authors who developed QIs from more than one source provided a transparent description of the respective sources of final QIs.

27Criteria for the selection of guidelines from which the QIs were derived were reported in 10 publications. Reported criteria were

28• the methodological quality,

29• the up-to-dateness,

30• the eligibility of a guideline for the selected topic (e.g., with regard to the specific setting).

31In 15 publications a critical appraisal of the used guidelines was reported based on the Appraisal of Guidelines Research and Evaluation in Europe (AGREE) instrument or similar quality criteria.

32Whilst participants in guideline selection are often mentioned, at least indirectly, for instance by being referred to as "the authors", criteria for their selection were reported in only four publications. These selection criteria were

33• member of a guideline development committee,

34• having methodological competence,

35• belonging to a profession involved in the selected healthcare process.

36Nine studies extracted all recommendations from selected guidelines. In 25 studies, recommendations were selected during the extraction process and not all recommendations were extracted as potential QIs. Criteria for this selection were reported in 21 of the 25 studies. Criteria for the preselection at the stage of recommendation extraction mentioned by the Agency for Healthcare Research and Quality (AHRQ) are

37• the size of the impact on patient health (the AHRQ considers the impact great when an issue affects a few patients severely or affects many patients),

38• the relevance to obtaining value for money.

39Other criteria for the preselection formulated by Hadorn et al. are

40• the importance to quality of healthcare provided,

41• the feasibility of monitoring.

42Other frequently reported criteria were the level of evidence, the grade of recommendation, and measurability.

43Levels of evidence and grades of recommendation of the recommendations potential QIs were developed from were reported in 24 studies. Only four studies reported criteria for the selection of persons who extracted potential QIs from guidelines. They were similar to those for persons involved in guideline selection (see above); both tasks were usually carried out by the same group of people.

44The AHRQ provides a detailed description of the extraction process, including specifications of participants' necessary skills, as well as criteria for the selection of recommendations to be extracted.

45Four requirements for persons involved in the extraction of potential QIs from guidelines postulated by the AHRQ are

46• clinician and nonclinician management skills,

47• clinical expertise,

48• technical expertise in performance measurement,

49• healthcare information management expertise.

50Another prerequisite for a valid extraction process mentioned in several of the relevant studies requires that the extraction be performed by at least two researchers independently.

51In 35 studies, a consensus method was used to augment the evidence from literature with expert and layperson opinion by letting a panel rate and select a set of final QIs from a set of potential QIs. In 15 of these 35 publications this method was described as the "modified RAND/UCLA method," named after the RAND/UCLA (University of California, Los Angeles) appropriateness method.

52Whereas only a few studies named the individual members of the panels, criteria for their selection (e.g., clinical expertise, methodological expertise, membership in a specialist society) were reported in 31 of 35 studies. Only 25 of 35 studies provided rating criteria for the panel process. Among the frequently named criteria were the usefulness of QIs for improving patient outcomes, their relevance, and the feasibility of monitoring.

53Participation of patients in the development process was reported in six studies. In all of these studies, patients participated in the panels. No study reported patient participation during guideline selection and the extraction of recommendations.

54Only 19 studies reported the conduct of a QI practice test. In two studies, the practice test was conducted after the development process was completed. In 21 studies, a practice test was not mentioned at all.

55An implementation strategy for guideline-based QIs was reported in 26 studies. Among the reported activities were the instruction of key persons ("early adopters") as multipliers, the participation of end users in the development process, the publication of developed QIs by medical associations, supplying the appropriate software, and the adaptation of "global" QIs to more specific settings. Financial incentives and certification were also used to support implementation.

56Authors tended to describe the process of topic selection in insufficient detail. Mostly, selection criteria merely reflected the aims of the application of QIs in general: to measure and improve quality in areas of healthcare where the actual quality of care is either suboptimal or unknown.

57The selected literature describes two different approaches to guideline selection. The first approach identified in the reviewed literature is to develop QIs based on one or only a few preselected guidelines, often with the aim of supporting or evaluating guideline implementation. In certain contexts, such as specific settings in small healthcare systems, only one guideline may be available for QI development. In these cases, guideline-selection processes are of no or only minor relevance, and the number of recommendations to be translated into potential QIs is proportionately low.

58The second approach is to select a clinical topic and, subsequently, to obtain suitable, topic-specific guidelines as a basis for the development of QIs from guideline recommendations. In this case, expert opinion and existing QI sets are sometimes used as alternative sources for QIs. In comparison to the first approach, this approach provides a broader basis for the subsequent development of QIs, bears the potential to produce a balanced set of QIs, carries a reduced risk of selection bias, and increases content validity.

59Many studies do not describe their guideline-selection criteria in sufficient detail and lack critical appraisal of their selected guidelines, both of which may compromise content validity and hence the quality of resulting QI sets. We argue that high-quality QIs can only be derived from high-quality guidelines. To ensure QIs originate from a sound foundation, development committees should (a) conduct a systematic search for relevant guidelines in national and international guideline databases as well as conventional literature databases and (b) conduct a critical appraisal of the methodological quality of selected guidelines (e.g., by using the AGREE instrument).

60As is common practice in other areas of research such as guideline development, the documentation of selection criteria for participating persons as well as the disclosure of their names and potential conflicts of interest could greatly add transparency to the whole development process and, as a result, increase the content validity of QIs.

61The main focus of this review is the extraction of guideline recommendations. This step is both crucial and unique to guideline-based QI development, whereas the other steps could also be applied to the development of QIs from other sources such as primary literature or existing QI sets. We only included studies that provided a detailed description of the recommendation-extraction process. As a result, we excluded a large number of otherwise eligible studies (see Additional file 4: Table 4 for a list of studies excluded for this reason).

62The reviewed literature describes two different approaches to the extraction of guideline recommendations. The first approach is to initially extract all recommendations and to then select QIs using a systematic consensus process. The second approach is to select a limited number of recommendations during the extraction process. We believe the difference between both approaches is of crucial importance to the quality of ensuing QI sets. Predominantly, only a small number of persons conduct the extraction process. Often, those participants were not selected following transparent selection criteria. The extraction of potential QIs itself through this small group of participants usually does not follow any documented selection criteria, either. As a result, the final QI set may suffer from selection bias.

63Subsequent systematic consensus processes to rate and select the extracted potential QIs are usually conducted by larger panels. In comparison to the small group of persons conducting the selection of potential QIs, panel participants are commonly selected to build a balanced panel of different professionals participating in the process of healthcare the QIs are developed for. In addition, the use of predesigned forms containing rating and selection criteria during these systematic consensus processes substantially reduces the risk of selection bias (see "QI selection").

64Another important aspect of the extraction process is the translation of the guideline text into recommendations manageable as potential QIs. It can be difficult to derive appropriate numerators and denominators on the basis of the guideline recommendation wording, which may not be specific enough for this purpose. A whole paragraph of guideline text, for instance, cannot easily be translated into a potential QI without cutting out potentially relevant information. Thus, the translation process is a further potential source of bias.

65Hence, both the selection of participants as such and the documentation of selection criteria for participants are of great importance. We identified a large deficit in the existing literature regarding this: Only five studies reported selection criteria for participants.

66Panel methods are not specific to guideline-based QI development and are frequently used to systematically augment the evidence from guidelines with expert opinion (e.g., the widely used RAND/UCLA appropriateness method). Performed carefully, this reduces the risk of unintentional influence of stakeholders on the results of the development process. Panel methods are an established component of the development process of high-quality guidelines. As our results confirm, they are also widely used in the development of QIs. Many of the reviewed studies showed a lack of transparency regarding the nomination process (e.g., in not providing explicit selection criteria for panel members).

67Our results show that patient participation during QI development is extremely uncommon. In principle, the frequently used panel method offers room for the participation of patients or patient representatives. However, to date, no standardized approach to patient participation during QI development exists. To fill this gap, our working group is currently conducting a systematic review of approaches to patient participation during QI development.

68Practice tests prior to publication and usage of QIs are an essential step in evaluating validity, reliability, feasibility, and other important attributes of QIs (see Background). They are an integral part of any implementation strategy and an essential component of the quality loop. The practice test in a study by Wollersheim et al. showed that between 10% and 20% of the developed QIs were not measurable.

69It could be argued that regular evaluations of the usage of QIs suffice. However, given the impact QIs can have from day one of their application (e.g., if used in pay-for-performance models [see Background]) and the fact that QIs are more widely accepted after an advance test, it is desirable that practice tests under "laboratory conditions" become established components of the development process.

70The importance of implementation strategies is often referred to in the course of critical appraisal of guidelines. As for guideline development, implementation strategies are indispensable for the real-life application of QIs. Our results show that even though a wide variety of implementation strategies are reported, they are not always part of the QI development process. Given the importance of implementation, a thorough discussion and application of implementation strategies should be an integral part of a gold-standard QI development method.

71To our knowledge, this is the first systematic review of methodological approaches to guideline-based QI development. This systematic review has been conducted following a rigorous methodological approach. The identification of methodological approaches to each step of guideline-based QI development allows a detailed description and comparison of the development methods published so far. We summarized the available evidence from systematically retrieved literature to provide a comprehensive overview of guideline-based QI development.

72A major limitation of this study is that we were not able to provide answers to review questions 2 and 3. The selected studies were very heterogeneous in type, in terms of the quality of reporting and in the methodological approaches to guideline-based QI development presented. Because we could not identify any studies comparing different methodological approaches to guideline-based QI development and no gold standard exists to compare the published methodological approaches to, we were not able to provide an evidence-based judgment on the methodological approaches identified. Hence, we were not able to determine whether any of the methodological approaches (as a whole or as single development steps) is "superior" to the others in its ability to generate high-quality QIs.

73However, in describing the methodological approaches used by the different working groups developing QIs, we provide a basis for further research. This research should seek to determine which of these methodological approaches applied to individual steps of the development process are best suited to constitute a development pathway that generates the "best" QIs. In order to achieve this aim in view of limited resources, existing guideline developers network infrastructure (e.g., the G-I-N) should be used to cooperate and formulate a gold standard, as proposed by Blozik et al..

74A wide variety of methodological approaches are described in the literature for guideline-based QI development. It remains unclear which method leads to the best QIs, since no randomized controlled or other comparative studies investigating this issue exist.

75In presenting a comprehensive methodological overview, we provide a groundwork for further research leading to an evidence-based gold standard for guideline-based QI development.

76The authors declare that they have no competing interests.

77TK designed the study; performed literature search and screening, literature retrieval, and data extraction and interpretation; and wrote and revised the paper. EB contributed to the initial study idea, study design, and data interpretation; critically revised the article for important intellectual content; and read and approved the final draft. MS contributed to initial study idea, study conception and design, and data interpretation; critically revised the article for important intellectual content; and read and approved the final draft.