Instruments for Assessing Risk of Bias and Other Methodological Criteria of Published Animal Studies: A Systematic Review

1Results from animal toxicology studies are a critical—and often the only—input to evaluating potential harm from exposure to environmental chemicals or the safety of drugs before they proceed to human testing. However, there is significant debate about how to use animal studies in risk assessments and other regulatory decisions. An important part of this debate is how to evaluate the methodology and potential biases of the animal studies in order to establish how confident one can be in the data.

2For the evaluation of human clinical research, there is a distinction between assessing risk of bias and methodological quality. Risks of bias are methodological criteria of a study that can introduce a systematic error in the magnitude or direction of the results. In controlled human clinical trials testing the efficacy of drugs, studies with a high risk of bias—such as those lacking randomization, allocation concealment, or blinding of participants, personnel, and outcome assessors—produce larger treatment effect sizes, thus falsely inflating the efficacy of the drugs compared with studies that have these design features (Schulz et al. 1995;, ). Biased human studies assessing the harms of drugs are less likely to report statistically significant adverse effects. An assessment of a study’s methodology includes evaluation of additional study criteria related to how a study is conducted (e.g., in compliance with human subjects guidelines) or reported (e.g., study population described). Finally, risk of bias is not the same as imprecision. Whereas bias refers to systematic error, imprecision refers to random error. Although smaller studies are less precise, they may not be more biased.

3Although there is a well-developed and empirically based literature on how to evaluate the risk of bias of randomized controlled clinical trials, less is known about how to do this for animal studies. Some risks of bias in animal studies have been identified empirically. For example, analyses of animal studies examining interventions for stroke, multiple sclerosis, and emergency medicine have shown that lack of randomization, blinding, specification of inclusion/exclusion criteria, statistical power, and use of comorbid animals are associated with inflated effect estimates of pharmaceutical interventions. However, these studies used a variety of instruments to evaluate the methodology of animal studies and often mixed assessment of risks of bias, reporting, and other study criteria.

4Several guidelines and instruments for evaluating the risks of bias and other methodological criteria of animal research have been published, but there has been no attempt to compare the criteria that they include; to determine whether risk of bias, reporting, or other criteria are assessed; or to determine whether the criteria are based on empirical evidence of bias. The purpose of this review was 2-fold: a) to systematically identify and summarize existing instruments for assessing risks of bias and other methodological criteria of animal studies, and b) to highlight the criteria that have been empirically tested for an association with bias in either animal or clinical models.

5Inclusion/exclusion criteria. Articles that met the following inclusion criteria were included: a) The article was a published report focusing on the development of an instrument for assessing the methodology of animal studies, and b) the article was in English. Where multiple analyses using a single instrument were published separately, the earliest publication was used. Modifications or updates of previously published instruments were considered new instruments and included. We did not include applications of previously reported instruments that were used, for example, to assess a certain area of animal research.

6Search strategy. We searched Medline for articles published from January 1966 through November 2011 using a search term combination developed with input from expert librarians. Bibliographies from relevant articles were also screened to find any remaining articles that were not captured from the Medline search. Our search strategy contained the following MeSH terms, text words, and word variants:

7{(animal experimentation[mh]) AND (standards[sh] OR research design[mh] OR bias[tw] OR biases[tw] OR checklist*[tw] OR translational research/ethics)} OR {(animals, laboratory[majr] OR disease models, animal[mh] OR drug evaluation, preclinical[mh] OR chemical evaluation OR chemical toxicity OR chemical safety) AND (research[majr:noexp] OR translational research[majr] OR research design[majr] OR “quality criteria”) AND (guideline* OR bias[tw] OR biases[tiab] OR reporting[tw])} OR {(animal*[ti] OR preclinical[ti] OR pre-clinical[ti] OR toxicology OR toxicological OR ecotoxicology OR environmental toxicology) AND (methodological quality OR research reporting OR study quality OR “risk of bias” OR “weight of evidence”)} OR {(CAMARADES[tiab] OR “gold standard publication checklist” OR exclusion inclusion criteria animals bias) OR (peer review, research/standards AND Animals[Mesh:noexp])} OR {(models, biological[mh] OR drug evaluation, preclinical[mh] OR toxicology[mh] OR disease models, animal[majr]) AND (research design[mh] OR reproducibility of results[mh] OR “experimental design”) AND (quality control[mh] OR guidelines as topic[mh] OR bias[tw] OR “critical appraisal”) AND (Animals[Mesh:noexp])} AND eng[la].

8Article selection. Studies were screened in two stages. Initially, we reviewed abstracts and article titles, and only those articles meeting our inclusion criteria were further scrutinized by reading the full text. Any articles that did not clearly meet the criteria after review of the full text were discussed by two authors, who made the decision about inclusion. Exact article duplicates were removed using Endnote X2 software (Thomson Reuters, Carlsbad, CA).

9Data extraction. We extracted data on each criterion included in each instrument, as well as information on how the instrument was developed.

10Instrument development and characteristics. We recorded the method used to develop each instrument (i.e., whether the criteria in the instrument were selected based on consensus, previous animal instruments, and/or clinical instruments). We also recorded whether or not the criteria in the instrument were empirically tested to determine if they were associated with biased effect estimates. Empirical testing was rated as completed if at least one of the individual criterion was empirically tested.

11Numerical methodological “quality” scores have been shown to be invalid for assessing risk of bias in clinical research. The current standard in evaluating clinical research is to report each component of the assessment instrument separately and not calculate an overall numeric score. Although the use of quality scores is now considered inappropriate, it is still a common practice. Therefore, we also assessed whether and how each instrument calculated a “quality” score.

12We also noted whether the instrument had been tested for reliability and validity. Reliability in assessing risk of bias refers to the extent to which results are consistent between different coders or in trials or measurements that are repeated. Validity refers to whether the instrument measures what it was intended to measure, that is, methodological features that could affect research outcomes.

13Study design criteria to assess risk of bias and other methodological criteria. Based on published risk of bias assessment instruments for clinical research, we developed an a priori list of criteria and included additional criteria if they occurred in the review of the animal instruments.

14We collected risk of bias, methodological, and reporting criteria because these three types of assessment criteria were often mixed in the individual instruments. The final list of these criteria is as follows:

15Treatment allocation/randomization. Describes whether or not treatment was randomly allocated to animal subjects so that each subject has an equal likelihood of receiving the intervention.

16Concealment of allocation. Describes whether or not procedures were used to protect against selection bias by ensuring that the treatment to be allocated is not known by the investigator before the subject enters the study.

17Blinding. Relates to whether or not the investigator involved with performing the experiment, collecting data, and/or assessing the outcome of the experiment was unaware of which subjects received the treatment and which did not.

18Inclusion/exclusion criteria. Describes the process used for including or excluding subjects.

19Sample size calculation. Describes how the total number of animals used in the study was determined.

20Compliance with animal welfare requirements. Describes whether or not the research investigators complied with animal welfare regulations.

21Financial conflict of interest. Describes if the investigator(s) disclosed whether or not he/she has a financial conflict of interest.

22Statistical model explained. Describes whether the statistical methods used and the unit of analysis are stated and whether the statistical methods are appropriate to address the research question.

23Use of animals with comorbidity. Describes whether or not the animals used in the study have one or more preexisting conditions that place them at greater risk of developing the health outcome of interest or responding differently to the intervention relative to animals without that condition.

24Test animal descriptions. Describes the test animal characteristics including animal species, strain, substrain, genetic background, age, supplier, sex, and weight. At least one of these characteristics must be present for this criterion to be met.

25Dose–response model. Describes whether or not an appropriate dose–response model was used given the research question and disease being modeled.

26All animals accounted for. Describes whether or not the investigator accounts for attrition bias by providing details about when animals were removed from the study and for what reason they were removed.

27Optimal time window investigated. Describes whether or not the investigator allowed sufficient time to pass before assessing the outcome. The optimal time window used in animal research should reflect the time needed to see the outcome and depends on the hypothesis being tested. The optimal time window investigated should not be confused with the “therapeutic time window of treatment,” which is defined as the time interval after exposure or onset of disease during which an intervention can still be effectively administered.

28We extracted data on the study design criteria assessed by each instrument. We recorded the number of criteria assessed for each instrument, excluding criteria related only to journal reporting requirements (i.e., headers in an abstract).

29Analysis. Here we report the frequency of each criterion assessed, as well as the frequency of any additional criteria that were included in the instruments.

30As shown in Figure 1, we identified 3,731 potentially relevant articles. After screening the article titles and abstracts, we identified 88 citations for full text evaluation. After reviewing full text, 60 papers were excluded for at least one of three reasons: a) They did not meet inclusion criteria; b) the studies reviewed a preexisting instrument; and c) the article reported application of an instrument. After screening bibliographies, two additional instruments were found. Overall, 30 instruments were identified and included in the final analysis.

31Table 1 lists the criteria of each instrument. Of the 30 instruments, 13 were derived by modifying or updating previously developed animal research methodology assessment instruments or citing animal studies supporting the inclusion of specific criteria; 3 were derived from previously developed clinically based risk of bias assessment instruments or citing clinical studies supporting the inclusion of specific criteria; 5 were developed using evidence from clinical research and either through consensus or citing past instrument publications; 3 were developed through consensus and citing past publications; and 6 had no description of how they were developed.

32Six instruments contained at least one criterion that showed an association of the criterion with inflated drug efficacy in animal models.

33Seven instruments calculated a score for assessing methodological “quality.” Descriptions of how these scores were calculated are provided in Table 1. Sixteen of the instruments were designed for no specific disease model; the most commonly modeled disease was stroke (9 of 30 instruments).

34Only 1 instrument was tested for validity, and 1 instrument was tested for reliability. Overall, 18 instruments were designed specifically to evaluate preclinical drug studies, 8 instruments documented general animal research guidelines, and 4 instruments were designed to assess environmental toxicology research.

35The total number of risk of bias, methodological, and/or reporting criteria assessed by each instrument ranged from 2 to 25. Table 2 shows the study design criteria used to assess risk of bias for each of the 30 instruments. Although these criteria were included in at least some of the instruments, they were not all supported by empirical evidence of bias. Blinding and randomization were the two most common criteria found in existing instruments; 25 instruments included randomization and 23 instruments included blinding. The need to provide a sample size calculation was listed in 18 instruments. None of the instruments contained all 13 criteria from our initial list; 2 instruments contained 9 criteria, and 4 instruments contained only 1 or 2 of the criteria.

36Additional criteria assessed by each instrument are listed in Supplemental Material, Table S1. Some of these criteria related to reporting requirements for the abstract, introduction, methods, results, and conclusions, rather than risk of bias criteria. These reporting criteria were not included in the count for the number of risk of bias criteria assessed by an instrument. For example, stated that the ARRIVE Guidelines is a 20-criteria instrument. However, we consider the ARRIVE Guidelines as a 13-criteria instrument because 7 of the original criteria pertain to reporting requirements. Fourteen instruments contained criteria to describe animal housing, husbandry, or physiological conditions. Inclusion of these criteria is empirically supported by studies showing that changes in housing conditions affect physiological and behavioral parameters in rodents. Among instruments that did not specify the need to use randomization, 4 of 5 instruments stated that a control group should be used.

37In this systematic review we identified 30 instruments for assessing risk of bias and other methodological criteria of animal research. Identifying bias, the systematic error or deviation from the truth in actual results or inferences, in animal research is important because animal studies are often the major or only evidence that forms the basis for regulatory or further research decisions. Our review highlights the variability in the development and content of instruments that are currently used to assess bias in animal research.

38Most of the instruments were not tested for reliability or validity. One notable exception is the CAMARADES (Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies) instrument developed by; these authors combined criteria from four previous instruments and showed that the instrument appears to have validity. Similarly, tested the reliability of a modified version of the Australasian ecotoxicity database (AED) instrument and found an improvement in reliability compared with the original AED instrument. Furthermore, most of the instruments were not developed on the basis of empirical evidence showing an association between specific study design criteria and bias in research outcomes. Only six instruments included criteria that were supported by data showing an association between a particular methodological criterion and effect size in animal studies. Most of the instruments contain criteria based on expert judgment, and others extrapolate from evidence of risk of bias in human studies. In addition, seven instruments calculated a “quality score”; however, these scores are not considered a valid measure of risk of bias, and this practice should be discontinued.

39Types of bias that are known to influence the results of research include selection, performance, detection, and exclusion. These biases have been demonstrated in animal studies, and methodological criteria that can protect against the biases have been empirically tested.

40Selection bias, which introduces systematic differences between baseline characteristics in treatment and control groups, can be minimized by randomization and concealment of allocation. Lack of randomization or concealment of allocation in animal studies biases research outcomes by altering effect sizes. Performance bias is the systematic difference between treatment and control groups with regard to care or exposure other than the intervention. Detection bias refers to systematic differences between treatment and control groups with regard to how outcomes are assessed. Blinding of investigators can protect against performance bias, and there is substantial evidence that lack of blinding in a variety of types of animal studies is associated with exaggerated effect sizes. Blinding of outcome assessors is a primary way of reducing detection bias. There are many ways to achieve adequate blinding in animal studies, such as having coded data (blinding to treatment assignment) analyzed by a statistician who is independent of the rest of the research team. Exclusion bias refers to the systematic difference between treatment and control groups in the number of animals that were included in and completed the study. Accounting for all animals used in the study and using intention-to-treat analysis can reduce exclusion bias.

41Some criteria included in the animal research assessment instruments are not associated with bias. For example, a statement of compliance with animal welfare requirements is a reporting issue. Sample size calculations are often included as a criterion in animal research assessment instruments, but bias is not the same as imprecision. Whereas bias refers to systematic error, imprecision refers to random error, meaning that multiple replications of the same study will produce different effect estimates because of sampling variation. Although larger and more precise studies may give a more accurate estimate of an effect, they are not necessarily less biased. Furthermore, sample size calculations can be greatly affected by the underlying assumptions made for the calculation. Although a sample size calculation is not a risk of bias criterion, it is an important characteristic to consider in evaluating an overall body of evidence.

42Some of the criteria listed in the instruments are unique to animal studies. For example, in preclinical drug research, testing animals with comorbidities is necessary to identify whether or not candidate drugs retain efficacy in light of additional health complications and to more closely resemble the health status of humans. Empirical evidence supports the use of this criterion because studies that included healthy animals instead of animals with comorbidities overestimated the effect sizes of experimental stroke interventions by > 10%. For environmental chemicals, use of comorbid animals could result in the opposite influence on effect size (i.e., to decrease it), and considering this as a criterion is consistent with recommendations to evaluate the influence of biological factors that may influence risk. Timing of exposure also influences study outcome, and some effects may be observed only for exposures that occur during certain developmental periods. Sex, the nutritional status of experimental animals, and animal housing and husbandry conditions could also affect the response to an intervention or environmental chemical exposure, but these criteria should be studied to determine if they introduce a systematic bias in results. These unique criteria have not been sufficiently included in the study instruments; even if these criteria do not produce systematic bias, they should be clearly described and reported in animal studies to aid interpretation of the findings (Marshall et al. 2005).

43Although some risk of bias criteria have been investigated primarily in human studies, they warrant consideration for animal studies. Reviews of clinical studies have shown that study funding sources and financial ties of investigators (including university- or industry-affiliated investigators) are associated with favorable research outcomes for the sponsors. In that study, favorable research outcomes were defined as either increased effect sizes for drug efficacy studies, or decreased effect sizes for studies of drug harm. Selective reporting of outcomes and failure to publish entire studies is considered an important source of bias in clinical studies; however, little is known about the extent of this bias in animal research.

44Further research should consider potential interactions between criteria for assessing risk of bias. Existing instruments have tested the association of study design criteria on effect size using univariate models. Multiple regression models should be used to ascertain the relationship between a study design criterion and effect size when taking into account other criteria in the model. Covariance between methodological criteria should also be examined. For example, randomized studies may be less likely to omit blinding than nonrandomized studies. Knowing the relative importance of these criteria will provide additional support for inclusion of specific criteria in risk of bias assessment instruments.

45Most of the instruments identified for our study exclude some criteria that appear to be important for assessing bias in animal studies (e.g., allocation concealment). It is important to recognize that some authors purposely exclude certain criteria from their instruments to reduce complexity and unnecessary detail. The most complex instrument had 25 criteria. The detailed level of reporting needed to apply the gold standard publication checklist (GSPC), which has 17 criteria, was one of the main criticisms against it.

46Because many journals now allow online publication of supplemental data, risk of bias assessment should be less limited by a lack of space for reporting detailed methods. Reporting of clinical research has improved because risk of bias assessments for systematic reviews and other purposes have become more prevalent and standards for reporting have been implemented by journals. Recent calls for reporting criteria for animal studies recognize the need for improved reporting of animal research. As happened for clinical research, reporting of animal research is likely to improve if risk of bias assessments become more common.

47Many of the instruments identified in our review were derived to evaluate preclinical animal drug research, which could limit their potential application in environmental health research. Although selection, detection, and performance biases are relevant for all animal research, some of the preclinical instruments contain criteria specific for assessing the quality of stroke research, such as the “avoidance of anesthetics with marked intrinsic neuroprotective properties”. On the other hand, investigation of an optimal time window for outcome assessment, the timing of the exposure, and measurement of outcomes that are sensitive to the exposure at the appropriate time are particularly important for assessing animal studies of environmental exposures.

48Study limitations. A limitation of our study is that we may not have identified all published assessment instruments for animal research. Our inclusion criteria allowed only articles published in English; therefore, we may have missed some instruments published in other languages. Furthermore, because we limited our search to articles indexed in Medline, articles indexed exclusively in Embase or some other database would have been missed. However, both our consultation with a librarian and the large pool of studies identified through the electronic search suggest that it was comprehensive.

49In this review we identified a wide variety of instruments developed to evaluate animal studies. The individual criteria included in animal risk of bias assessment instruments should be empirically tested to determine their influence on research outcomes. Furthermore, these instruments need to be tested for validity and reliability. Finally, existing instruments (many of which were developed using stroke models) need to be tested on other animal models to ensure their relevance and generalizability to other systems.