Applying GRADE-CERQual to qualitative evidence synthesis findings—paper 5: how to assess adequacy of data

1The GRADE-CERQual (Confidence in Evidence from Reviews of Qualitative research) approach has been developed by the GRADE (Grading of Recommendations Assessment, Development and Evaluation) working group. The approach has been developed to support the use of findings from qualitative evidence syntheses in decision-making, including guideline development and policy formulation.

2GRADE-CERQual (hereafter referred to as CERQual) includes four components for assessing how much confidence to place in findings from reviews of qualitative research (also referred to as qualitative evidence syntheses): (1) methodological limitations; (2) coherence; (3) adequacy of data; and (4) relevance. This paper focuses on one of these four CERQual components: data adequacy.

3When carrying out a CERQual assessment, we define adequacy of data as an overall determination of the degree of richness as well as the quantity of data supporting a review finding. The adequacy component in CERQual is analogous to the imprecision domain used in the GRADE approach for findings from systematic reviews of effectiveness.

4The aims of this paper, part of a series (Fig. 1), are to describe what we mean by adequacy of data in the context of a qualitative evidence synthesis and to give guidance on how to operationalise this component in the context of a review finding as part of the CERQual approach. This paper should be read in conjunction with the papers describing the other three CERQual components and the paper describing how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings table. Key definitions for the series are provided in Additional file 1.

5The initial stages of the process for developing CERQual, which started in 2010, are outlined elsewhere. Since then, we have further refined the current definitions of each component and the principles for application of the overall approach using a number of methods. When developing CERQual’s adequacy component, we undertook informal searches of the literature, including Google and Google Scholar, for definitions and discussion papers related to the concept of adequacy and to related concepts such as data quantity, sample size and data saturation. We carried out similar searches for the other three components. We presented an early version of the CERQual approach in 2015 to a group of methodologists, researchers and end users with experience in qualitative research, GRADE or guideline development. We further refined the approach through training workshops, seminars and presentations during which we actively sought, collated and shared feedback; by facilitating discussions of individual CERQual components within relevant organisations; through applying the approach within diverse qualitative evidence syntheses; and through supporting other teams in using CERQual. As far as possible, we used a consensus approach in these processes. We also gathered feedback from CERQual users through an online feedback form and through short individual discussions with members of the review teams. The methods used to develop CERQual are described in more detail in the first paper in this series.

6Qualitative research methods differ from quantitative research methods in many ways, including the role that numbers play and the manner in which data adequacy is conceptualised. Typically, quantitative researchers gather a relatively limited amount of information from a large number of people and use this information to make estimates, for instance about what people do. When answering these types of questions, quantity plays an important role, and larger numbers, for instance large study populations or large numbers of event rates, generally serve as one marker of data adequacy. Qualitative researchers, on the other hand, generally study fewer people, but attempt to delve more deeply into these people’s perceptions, experiences and settings in order to understand how they experience the world and why they do what they do. To answer these types of questions, rich and detailed data is often required, i.e. information that is detailed enough to allow the researcher or reader to interpret the meaning and context of what is being researched. For qualitative research, data richness therefore often serves as an important marker of data adequacy.

7This emphasis on data richness does not imply, however, that numbers do not matter in qualitative research. As Sandelowski has argued, sample sizes in qualitative research “may be too small to support claims of having achieved either informational redundancy or theoretical saturation, or too large to permit the deep, case oriented analysis that is the raison-d’etre of qualitative inquiry”. In other words, small numbers of study participants or observations can threaten our ability to make broad claims about a phenomenon while large numbers of participants or observations can threaten our ability to carry out a thorough qualitative analysis that would allow us to properly explore and explain a phenomenon. When determining sample size in qualitative studies, a trade-off needs to be made between conditions that require more versus fewer participants in a sample, including the aim of the study, the analytic approach and the extent to which it is based on existing knowledge and theory.

8When assessing data adequacy in the context of a finding from a qualitative evidence synthesis, the same principles apply as for other types of qualitative research, although our focus moves from data generated from individual participants and observations to data generated from individual studies that contribute to a specific finding.

9When carrying out a CERQual assessment, we define adequacy of data as an overall determination of the degree of richness as well as the quantity of data supporting a review finding.

10When we talk about the richness of the data, we are referring to the extent to which the information that the individual study authors have provided is detailed enough to allow the review author to interpret the meaning and context of what is being researched. When we refer to the quantity of data, we are thinking primarily of the number of studies and participants that this data comes from.

11There are no fixed rules regarding what constitutes sufficiently rich data or an adequate quantity of data. Instead, any assessment of data adequacy is always relative and is judged according to “what you are expecting the data to do”. In primary qualitative research, what we expect the data to do, and thereby the amount of data that we consider sufficient, will vary from study to study. While one in-depth interview could yield a rich and valid account of one person’s experiences and is sufficient to document, for instance, that these experiences are within the realms of possibility, three such interviews could be sufficient to prove that people may experience the same phenomenon very differently. Even higher numbers of in-depth interviews could show us how people’s experiences vary and can also help us identify the experiences that many people share. The same principles apply for qualitative evidence syntheses.

12When assessing data adequacy, our aim is not to judge whether data adequacy has been achieved, but to judge whether there are grounds for concern regarding data adequacy that are serious enough to lower our confidence in the review finding. We are likely to have concerns about data adequacy when we have concerns about the richness or the quantity of the data in relation to the claims made in the review finding.

13The concept of “data saturation” is related to, although not the same as the concept of data adequacy (Table 1).

14To carry out an assessment of data adequacy, you need to collect the following information for each review finding, and present it, for instance, in a table or matrix:An overview of the data upon which each review finding was basedAn overview of the number of studies from which this data originated, and where possible, the number of participants or observations. Information about the number of participants or observations supporting each finding may be difficult to gain from the individual studies. While most studies describe the number of participants they included in their study overall or give some indication of the extent of their observations, they may be less clear about how well represented participants are in different themes and categories. You can contact study authors for additional information, but they may not be able to readily provide this level of detail. In these cases, this lack of information should be noted, and your assessment of data adequacy will have to be made based on the information available.

15An overview of the data upon which each review finding was based

16An overview of the number of studies from which this data originated, and where possible, the number of participants or observations. Information about the number of participants or observations supporting each finding may be difficult to gain from the individual studies. While most studies describe the number of participants they included in their study overall or give some indication of the extent of their observations, they may be less clear about how well represented participants are in different themes and categories. You can contact study authors for additional information, but they may not be able to readily provide this level of detail. In these cases, this lack of information should be noted, and your assessment of data adequacy will have to be made based on the information available.

17If you are using CERQual on findings from your own review, it should be straightforward to collect the information described above as you may already have constructed this type of table or matrix as part of the review process. However, if you are using CERQual to make an assessment of findings from other people’s reviews, this is likely to be a time-consuming process. This is particularly the case when it comes to gaining an overview of the data upon which each review finding is based as review authors do not commonly report all of the data supporting each review finding. Unless you have access to their data extraction sheets, you will need to collect this data yourself by going back to the original references associated with each review finding. For more information on using CERQual on findings from somebody else’s review, see the second paper in this series.

18Once you have collected the necessary information, you can start to assess whether you have any concerns about the richness of the data supporting each review finding and the quantity of this data. (In addition to the guidance we offer here, Table 2 also presents examples of how data adequacy can be assessed for a selection of review findings. These examples illustrate how considerations of both data richness and quantity feed into the overall determination of data adequacy).

19You are likely to have concerns about the richness of the data if it does not provide you with sufficient details to gain an understanding of the phenomenon described in the review finding. This is a judgement call. However, as a rule of thumb, for review findings that are simple and primarily descriptive, relatively superficial data may be sufficient. But when a review finding is complex or explanatory, e.g. when it suggests associations between different factors, you are less likely to have confidence in that finding if it is based on data that is too superficial to allow a sufficient exploration of the phenomenon (see Fig. 3. See also Table 2 for examples). For a description of descriptive and explanatory review findings, see the paper in this series on coherence.

20Alongside your assessment of data richness, you also need to consider the number of studies, participants or observations upon which the data are based. As described above, it is often difficult to gain information about the number of participants or observations that contributed to each specific finding, and you may have to focus on the number of studies.

21While there is no fixed rule about what constitutes a sufficient number, you are likely to have less confidence in a review finding that is supported by data from only one or very few studies, participants or observations. This is because when only few studies or only small studies exist, or are sampled, we are less confident that studies undertaken in other settings or groups would have reported similar findings. As is the case for data richness, assessments should always be made in relation to the claims the review finding is making.

22Some review findings may make claims about a limited aspect of a phenomenon or a very specific group of people or type of settings, and may need fewer studies. However, when a review finding make claims about a broad phenomenon or a large variety of people we are less likely to have confidence in that finding if it is based on a small number of studies (Fig. 3).

23Other factors may increase your concerns about the richness or quantity of the data. Qualitative researchers aim not only to look for common attitudes and experiences, but are also interested in outliers and exceptions. However, for a review finding that makes claims about a relatively unexplored topic or a topic where people lack a widely shared language, detailed descriptions of context, intentions and meaning may be required. Similarly, a review finding that makes claims that are unexpected or that challenge common knowledge may require more data than review findings that represent a widely distributed experience or domain of knowledge and you may have less confidence in these findings if they are based on a small number of studies.

24Once you have assessed data adequacy for each finding, you should categorise any concerns that you have identified as either:No or very minor concernsMinor concernsModerate concernsSerious concerns

25No or very minor concerns

26Minor concerns

27Moderate concerns

28Serious concerns

29You should begin with the assumption that there are no concerns regarding adequacy. In practice, minor concerns will not lower your confidence in the review finding, while serious concerns will lower your confidence. Moderate concerns may lead you to consider lowering your confidence in your final assessment of all four CERQual components.

30Where you have concerns about adequacy, describe these concerns in the CERQual Evidence Profile in sufficient detail to allow users of the review findings to understand the reasons for the assessments made. The Evidence Profile presents each review finding along with the assessments for each CERQual component, the overall CERQual assessment for that finding and an explanation of this overall assessment. For more information, see the second paper in this series.

31Your assessment of data adequacy will be integrated into your overall assessment of confidence in each review finding. How to make this overall assessment of confidence is described in paper 2.

32Concerns about data adequacy may not only have implications for your confidence in a review finding, but can also point to ways of improving future research. First of all, these concerns suggest that more primary research is needed in relation to the phenomenon discussed in the review finding. The review team should also consider whether the review needs to be updated once that research becomes available.

33Secondly, concerns about the adequacy of the data may indicate that certain aspects of the review question were too narrow, and that you should consider widening the scope of future updates of the review. (Any changes that are made to the scope of the review are also likely to have an impact on your assessment of the other CERQual components).

34Concerns about data adequacy may influence your confidence in a review finding and are therefore part of the CERQual approach. We approach assessments of data adequacy in terms of the richness and the quantity of the data supporting each review finding, but do not offer fixed rules regarding what constitutes sufficiently rich data or an adequate quantity of data. Instead, we recommend that you make this assessment in relation to the nature of the finding. This paper aims to describe our thinking so far and help review authors and others assess data adequacy in findings from qualitative evidence syntheses. We expect the CERQual approach, and its individual components, to develop further as our experiences with the practical implementation of the approach increase.

35Peer review reports for this article are available in Additional file 2.