Cochrane's risk of bias tool for non-randomized studies (ROBINS-I) is frequently misapplied: A methodological systematic review

1In a sample of systematic reviews published across two months in 2020 (n = 124) that used the ROBINS-I tool to assess risk of bias, authors often modified or the tool or used it incorrectly. Poorly conducted reviews were more likely to rate risk of bias as low and moderate and less likely to rate it as critical, compared to well-conducted reviews.Risk of bias was rated as serious or critical in most studies, with the ‘confounding’ bias domain most often rated highly. This is the first study to investigate how ROBINS-I has been used in practice in the systematic review literature.Cochrane recommend that ROBINS-I should only be used by review teams with extensive methodological expertise, but our findings suggest that the tool is nonetheless frequently misapplied. Inadequate risk of bias assessment can substantively affect the findings of a systematic review, potentially leading to misleading recommendations and guidelines. Researchers and practitioners should be alert to this issue when reading systematic reviews, and authors should ensure that the conduct and reporting of risk of bias assessments are rigorous.

2In a sample of systematic reviews published across two months in 2020 (n = 124) that used the ROBINS-I tool to assess risk of bias, authors often modified or the tool or used it incorrectly. Poorly conducted reviews were more likely to rate risk of bias as low and moderate and less likely to rate it as critical, compared to well-conducted reviews.Risk of bias was rated as serious or critical in most studies, with the ‘confounding’ bias domain most often rated highly.

3In a sample of systematic reviews published across two months in 2020 (n = 124) that used the ROBINS-I tool to assess risk of bias, authors often modified or the tool or used it incorrectly. Poorly conducted reviews were more likely to rate risk of bias as low and moderate and less likely to rate it as critical, compared to well-conducted reviews.

4Risk of bias was rated as serious or critical in most studies, with the ‘confounding’ bias domain most often rated highly.

5This is the first study to investigate how ROBINS-I has been used in practice in the systematic review literature.Cochrane recommend that ROBINS-I should only be used by review teams with extensive methodological expertise, but our findings suggest that the tool is nonetheless frequently misapplied.

6This is the first study to investigate how ROBINS-I has been used in practice in the systematic review literature.

7Cochrane recommend that ROBINS-I should only be used by review teams with extensive methodological expertise, but our findings suggest that the tool is nonetheless frequently misapplied.

8Inadequate risk of bias assessment can substantively affect the findings of a systematic review, potentially leading to misleading recommendations and guidelines. Researchers and practitioners should be alert to this issue when reading systematic reviews, and authors should ensure that the conduct and reporting of risk of bias assessments are rigorous.

9Inadequate risk of bias assessment can substantively affect the findings of a systematic review, potentially leading to misleading recommendations and guidelines. Researchers and practitioners should be alert to this issue when reading systematic reviews, and authors should ensure that the conduct and reporting of risk of bias assessments are rigorous.

10Non-randomized studies of interventions (NRSIs) are an essential source of evidence in many fields where the effectiveness of interventions is of interest, but randomized controlled trials (RCTs) are not always feasible for practical or ethical reasons. NRSIs can achieve high precision when large data sets are used, but are susceptible to more complex and powerful biases than RCTs,. This presents challenges when evidence from NRSIs is included in systematic reviews and clinical guidelines.

11‘Risk of Bias In Non-randomized Studies – of Interventions’ (ROBINS-I) is a tool for assessing the risk of bias (RoB) in NRSIs, which was published in 2016 after 5 years of development and piloting,. Although ROBINS-I was designed to assess clinical intervention studies, modifications for other types of non-randomized study (NRS) are under development,. Unlike earlier appraisal tools, which tended to focus on identifying methodological flaws in specific study designs,, ROBINS-I integrates an understanding of causal inference based on counterfactual reasoning,,. Using this theoretical foundation, ROBINS-I assesses RoB on an absolute scale for the causal inferences of any NRSI regardless of study design, where ‘low’ RoB is comparable to a well-conducted RCT. ROBINS-I has been seen as a significant methodological innovation and a step change in how NRSIs can be utilized in evidence syntheses,,.

12ROBINS-I is the only tool recommended by the Cochrane Handbook for assessing RoB in NRSIs, and has been increasingly widely adopted in published and planned systematic reviews. However, there are concerns that the tool's conceptual complexity may have made it challenging to use, and that many review teams may lack the necessary expertise to apply it correctly,,. In areas of public health where uncertain evidence from NRSIs is common, there is also a concern that ROBINS-I may be of limited use due to the lack of discrimination among higher levels of RoB,.

13Previous methodological studies examining ROBINS-I have relied on small research teams applying the tool in a controlled setting,,,,,. However, no research has yet been undertaken on how ROBINS-I has been used in the large and growing body of reviews using the tool in practice. The aim of this study was to review how ROBINS-I has been used in recent systematic reviews, in order to identify ways that it may be misapplied and possible explanatory factors.

14A protocol for this study was developed prospectively and registered in PROSPERO (CRD42013006924; submitted 8 March, registered 11 June 2020). Searches were carried out in Scopus, Web of Science, MEDLINE, Embase, and the Cochrane Database of Systematic Reviews (CDSR). Title and abstract searches were conducted in all databases using the following search terms:“risk of bias in non randomi#ed studies of interventions” OR “risk of bias in nonrandomi#ed studies of interventions” OR “robins i”

15“risk of bias in non randomi#ed studies of interventions” OR “risk of bias in nonrandomi#ed studies of interventions” OR “robins i”

16Where possible (in all databases except Embase and CDSR), forward citation searches were also conducted to identify publications that cited the ROBINS-I tool.

17The searches were carried out on 2 March 2020, and were limited to publications from 1 January 2020 onwards. This time window of approximately 2 months was selected to balance a sufficient number of reviews against available time and resources, and to provide an understanding of contemporary research practice.

18Publications were included if they were presented by the authors as systematic reviews, reported that ROBINS-I was used to assess RoB, and were available in English. Search results were independently screened by two reviewers (E.I. and S.V.K.) and any disagreements resolved by consensus.

19The outcome of interest was the ROBINS-I RoB assessments reported within each review. Each outcome in an included study is assigned judgments in seven ‘bias domains’, as well as an overall RoB judgment. The possible judgments are ‘low’, ‘moderate’, ‘serious’, ‘critical’, and ‘no information’.

20Reported ROBINS-I assessments within each review were extracted using a Microsoft Excel spreadsheet. Where separate assessments were reported for multiple results or outcomes within a single study, assessments for each result were extracted separately. In addition to these assessments, the following information was extracted for each review: aim or research question as described by the authors, number of included studies (RCTs and NRSs), whether meta-analysis was performed, and any declared funding sources or competing interests. Additional information was also captured on how ROBINS-I was used and reported, including whether assessments were performed in duplicate, whether results from the pre-assessment stage were reported, in what form ROBINS-I assessments for individual studies were reported, and what categories were used for the assessments. Free-form textual comments were captured regarding non-standard uses of ROBINS-I, modifications to the tool, and any other noteworthy observations, including quotations from reviews where relevant.

21Data extraction was carried out by E.I. and review-level data were checked by a second reviewer (S.V.K., P.C., or M.C.).

22The methodological quality of the included reviews was assessed using AMSTAR 2 (‘A MeaSurement Tool to Assess systematic Reviews 2’). This tool consists of 16 items, each of which is rated as ‘yes’, ‘partial yes’, or ‘no’. Each item is regarded as either ‘critical’ or ‘non-critical’, and an overall confidence rating can be determined by the number of identified weaknesses (i.e., ‘no’ ratings) in critical and non-critical domains. Overall confidence in a review is rated ‘critically low’ if it has more than one critical flaw, ‘low’ if it has exactly one critical flaw, ‘moderate’ if it has no critical flaws but more than one non-critical weakness, and ‘high’ if it has no critical flaws and up to one non-critical weakness.

23Quality appraisals were conducted independently by two reviewers (E.I., M.C., P.C., and S.V.K.), and disagreements were resolved by consensus.

24Free-text comments on non-standard use of ROBINS-I were grouped into categories and summarized. For each review, we calculated the proportion of judgments in each RoB category and the range of categories used.

25For reviews that did not report overall RoB judgments, inferred overall judgments were calculated by taking the highest judgment in an individual domain, in accordance with the ROBINS-I guidance. For reviews that reported both domain-specific and overall judgments, the inferred overall judgment was compared to the reported judgment to identify instances where they differed.

26We considered the following explanatory variables as potential predictors of RoB judgments: methodological quality as assessed using AMSTAR 2, whether RoB assessment was performed in duplicate, whether the authors reported industry funding or competing interests, and whether the review included RCTs.

27Since the RoB assessments within the same review were expected to be similar, multilevel regression was used. The overall (or inferred overall) RoB judgments were treated as an ordinal outcome, and a separate generalized ordered logit model was fitted for each predictor with a review-level random intercept. Results were presented as odds ratios and population-average marginal predicted probabilities. Analyses were carried out in Stata MP/16.1 using the gllamm command.

28The literature search identified 124 systematic reviews satisfying the inclusion criteria (Fig. 1). The results covered a wide range of topics (Table 1), with most reviews (66%) studying individual-level clinical interventions. Methodological quality was rated as low or critically low in most reviews, with only 17% and 6% being classified as moderate and high quality, respectively. A complete list of included reviews and excluded full-text articles can be found in Supplementary Tables 1 and 2.

29Although most reviews (66%) reported both the overall RoB judgments and judgments for each individual domain, a considerable number omitted either or both. Most reviews (89%) reported only the judgments without any supporting comments or justifications: only six reviews (5%) reported explicit justifications for both overall and per-domain RoB judgments. The pre-assessment stage of ROBINS-I, where relevant confounding domains and co-interventions are identified, was not described at all in 94% of reviews; only eight reviews (6%) explicitly listed what confounding domains were considered.

30Modifications and non-standard uses of the tool were common, often without explanation or justification. These are described in Table 2, and in greater detail in Supplementary Table 3. Modifications of the rating scale were most common, occurring in 20% of reviews. Only one of these justified the use of a non-standard scale: Rhodes et al. used ‘low risk’, ‘some concerns’, and ‘high risk’, stating that ‘[these] terms were used to be concordant with the randomized trials’ (which were assessed using the Cochrane RoB 2 tool ). In the remaining reviews, no justification was provided.

31Most reviews used the seven bias domains listed in the tool. In five reviews, one domain was omitted or reported as ‘not applicable’ (domain 3, ‘classification of interventions’ or 4, ‘deviations from intended interventions’,,, ). One review used all seven domains, but added an eighth domain (‘vested interest bias’). One review appeared to use the seven ROBINS-I domains, but reported the results alongside results from the Cochrane RoB tool for RCTs, with domains renamed to fit both tools.

32Most reviews (69%) did not make references to RoB when synthesizing evidence (Table 2). Among the reviews that did, most referred to RoB in individual studies in narrative discussion or interpretation of results, and a smaller number incorporated RoB into meta-analysis (e.g., using subgroup analysis or sensitivity analysis) or excluded studies at high RoB. Contrary to guidance,. In 19% of reviews, studies rated at critical RoB were included in the narrative or quantitative synthesis; 7% included critical-RoB studies in a meta-analysis. Also contrary to guidance, 11% of reviews applied ROBINS-I to a non-interventional research question (e.g., estimating incidence or prevalence of a condition, or describing the features of a population,,,, ).

33ROBINS-I guidance states that the overall RoB judgment should normally be at least as high as the highest judgment in any bias domain. Out of the 90 reviews that reported both overall and domain-specific judgments, 20% did not adhere to this principle, and thus not all studies or results reported as low RoB overall were rated low in all domains. In the 68 reviews that reported judgments for all domains using the standard scale, 58% of the studies or results reported as low RoB overall had higher RoB in at least one domain, and 27% (all from a single review ) had critical RoB in at least one domain (Supplementary Table 4).

34Individual ROBINS-I assessments were extracted from 116 reviews, comprising a total of 1,565 assessments of 1,499 primary studies (Fig. 1). 221 assessments from 23 reviews used a non-standard rating scale and were excluded from further analysis. Of the assessments that remained, 171 only included an overall judgment, and 214 only included domain-specific judgments. For the latter, an inferred overall judgment was calculated by taking the highest RoB judgment in an individual domain. The final analyzed sample included 1,344 RoB assessments from 93 reviews.

35Among the 93 reviews reporting judgments using the standard scale, RoB was reported as low in 10% of studies on average, moderate in 36%, and critical in 15% (Table 3). However, these proportions varied by methodological quality as assessed using AMSTAR 2, with more well-conducted reviews containing fewer low- and moderate-RoB studies and more critical-RoB studies on average. Out of the reviews that included five or more NRSs, most made use of two (45%) or three (40%) adjacent categories; 13% of reviews assigned the same overall judgment to all included NRSs. Out of the seven bias domains, domain 1 (confounding) was more often rated at higher RoB than the remaining domains (Fig. 2), and was among the highest-rated domains in 74% of all the included NRSs. The pattern was similar across different AMSTAR 2 confidence ratings (Supplementary Figure 2).

36As Table 3 shows, reviews of higher methodological quality (as assessed using AMSTAR 2) appeared to have a lower proportion of low/moderate overall RoB judgments and a higher proportion of critical judgments, with an apparent dose-response relationship. The multilevel regression analyses confirmed this (Table 4), with the predicted probability of low RoB judgments ranging from 5% (95% CI: 2%–11%) in high/moderate-quality reviews to 13% (8%–20%) in critically low-quality reviews. Probability of moderate and lower RoB similarly ranged from 31% (19%–46%) in high/moderate-quality reviews to 57% (47%–67%) in critically low-quality reviews, and critical RoB was approximately twice as likely in moderate/high-quality reviews (30% [17%–45%]) as in low-quality (13% [7%–22%]) or critically low-quality reviews (15% [9%–23%]).

37There was less conclusive evidence of any association between RoB judgments and duplicate ROBINS-I assessment, reported industry funding/competing interests, or the inclusion of RCTs in the review. Although there were strong associations between duplicate RoB assessment and the proportion of critical RoB judgments, as well as between industry funding and low RoB judgments, the direction and strength of association were not consistent across levels of RoB.

38In a representative sample of 124 systematic reviews that used ROBINS-I to assess RoB in NRSs, methodological quality (assessed using AMSTAR 2) was generally low, unjustified modifications of ROBINS-I were common, and reporting of RoB assessments were often lacking in detail. Reported RoB was generally high, with confounding being by far the most common reason for high RoB judgemnts. However, low-quality reviews were more likely to report low or moderate RoB (where low RoB is interpreted as comparable to a well-conducted RCT ), and less likely to report critical RoB.

39This is the first study investigating the practical use of ROBINS-I in a sizeable sample of published reviews from a range of disciplines. Previous methodological studies of ROBINS-I have been conducted on a relatively small scale, with assessments conducted by the authors themselves, and including a limited number of studies,,,,. By using multilevel methods on a comparatively large sample of individual ROBINS-I assessments, we have achieved greater statistical power than would have been possible with review-level data alone. However, it is possible that the associations seen in the quantitative analyses may be confounded by review characteristics, such as topic area or discipline.

40In line with previous literature employing AMSTAR 2 to study review quality,,,, only a small proportion of included reviews were of high methodological quality, and only one was a Cochrane review. Hence, this study provides limited insight into how ROBINS-I has been employed in the most well-conducted reviews. We were also unable to include any reviews not available in English, although since this was checked at the screening stage, and no reviews were excluded for this reason, this did not affect the results. Due to resource limitations, we were also unable to conduct independent data extraction, although review selection and quality assessment were conducted in duplicate, and extracted data were checked by a second reviewer.

41The lack of detailed reporting on RoB assessments in most reviews was also a limiting factor. Since only a small number of reviews included explicit justifications for their ROBINS-I judgments, it was not possible to examine how far these were plausible or justified, as has been done in similar methodological reviews studying the Cochrane RoB tool for RCTs,. Assessments from a substantial number of reviews also had to be excluded from quantitative analyses because they used a rating scale that deviated too far from the official tool.

42The distribution of RoB judgments we observed is largely in line with previous studies. Losilla et al., Thomson et al., and Dhiman et al. reported overall RoB of at least moderate, serious, and serious respectively, used no more than two adjacent RoB categories, and reported that the confounding domain usually determined the overall judgment. Minozzi et al. reported statistics for inter-rater reliability, but no individual RoB judgments. Our results confirm that similar patterns can be seen in published reviews.

43Unexplained modifications of the tool and departures from guidance were common. This is not unexpected, given previous evidence that other RoB tools are often inadequately applied,. It is mostly unclear whether such departures were deliberate or in error, but some may clearly have impacted the results of a review in ways that could mislead a reader. For example, a majority of studies reported as having low RoB would have been given a more conservative judgment if the recommended process for determining an overall RoB judgment had been followed. Additionally, although ROBINS-I was explicitly designed to assess RoB in NRSIs, several reviews inappropriately applied the tool to NRSs with non-interventional research questions. This may reflect an unmet need for equally rigorous appraisal tools that can be applied to non-interventional study designs.

44We found that poorly conducted reviews were more likely to rate included studies as having low RoB. Assuming that poorly conducted reviews were more likely to have applied ROBINS-I incorrectly, this can be taken as evidence that incorrect application of ROBINS-I led authors to understate the true RoB on average. Although the association between review quality and RoB judgments could be confounded by review characteristics, we found no indication that low-quality reviews were systematically different from high-quality reviews with respect to topic or other features that might otherwise explain the differences in reported RoB. The reasons for any potential underestimation of RoB are unclear, given the lack of detailed reporting in the included reviews, but given that substantial expertise is needed to apply ROBINS-I correctly, it is plausible that less experienced or thorough review teams are likely to overlook subtle or complex sources of bias that a team with greater epidemiological expertise would identify.

45As has been emphasized in previous literature,, ROBINS-I is a complex tool, and should only be used by review teams with extensive methodological expertise and sufficient time to commit to learning how to apply the tool correctly. Our findings suggest that ROBINS-I is nonetheless often applied with insufficient rigour, and consequently RoB may be underestimated in many reviews that include NRSs.

46Our findings highlight that substantial expertise in epidemiology and systematic review methods is required when conducting reviews that include NRSs. The Cochrane Handbook emphasizes the importance of assessing RoB in seven key domains, with ROBINS-I being the only tool presented that explicitly does this, and this may have encouraged some review authors to use ROBINS-I despite lacking the necessary expertise. Authors should carefully consider the trade-off between rigour and ease-of-use in different tools: although more easily-applied tools are available (e.g., the Newcastle-Ottawa scale), these often fail to cover important bias domains, and hence potentially fail to identify sources of bias that a more rigorous tool could have identified,. At the same time, a more rigorous tool could give a false impression of certainty if it is not rigorously applied. Many alternative appraisal tools also fail to distinguish clearly between risk of bias and external validity.

47Since RoB ratings directly influence which studies are included in a meta-analysis or narrative synthesis, the accuracy of RoB assessments can have a direct impact on the results of a review. We thus recommend that readers carefully consider whether RoB assessments were conducted rigorously and correctly. Current appraisal tools, and reporting guidelines for systematic reviews make few explicit recommendations about how much detail is appropriate when reporting the results of RoB assessments. When using ROBINS-I, we recommend that authors report the output of the pre-assessment stage (including important confounding domains and co-interventions), both overall and domain-specific RoB judgments, and written justifications for at least the overall judgment.

48ROBINS-I appears to be frequently misused by systematic reviews of low methodological quality, which often modify the tool or deviate from its official guidance. Low-quality reviews are more likely to underestimate RoB, and may thus inappropriately include or give too much weight to uncertain evidence. Readers should be aware that such problems can lead to incorrect conclusions in reviews. Authors should identify the RoB assessment tool most suitable for their review, and be aware that correct application of the ROBINS-I tool requires expertise and sufficient resources.

49Erik Igelström: conceptualization, methodology, investigation, formal analysis, writing – original draft. Mhairi Campbell: validation (quality appraisal), writing – review & editing. Peter Craig: validation (quality appraisal), writing – review & editing. S. Vittal Katikireddi: conceptualization, supervision, validation (quality appraisal), writing – review & editing.