Seriously misleading results using inverse of Freeman‐Tukey double arcsine transformation in meta‐analysis of single proportions

1A key application of meta‐analytical methods is the pooling of proportions, such as prevalence of a specific infection or disease.(1, 2, 3, 4) Classic fixed‐effect and random‐effects meta‐analysis methods5 are typically used to combine single proportions. In order to use these methods, proportions are generally transformed using either the log,6 logit,7 arcsine,8 or Freeman‐Tukey double arcsine9 transformations. These transformations are implemented for pure mathematical reasons, eg, variance stabilization (details on the transformations are given in Appendix A and summarized in Table A1). For pooling, the transformed proportions and corresponding standard errors are used in the generic inverse variance method.5 An alternative yet more elaborate approach based on the logit transformation are generalized linear mixed models (GLMMs),10 which account for the binomial structure of the data and thus avoid the generic inverse variance method. Irrespective of the meta‐analysis method and transformation, results are usually presented on the original probability scale after using the corresponding back‐transformation.

2Whereas the back‐transformation of meta‐analysis results is straightforward for the log, logit, and arcsine transformations, this is not the case for the Freeman‐Tukey double arcsine transformation, albeit possible.11 In order to calculate the inverse of the Freeman‐Tukey double arcsine transformation, a single sample size has to be specified. Accordingly, for a single study, a one to one relation exists between transformation and its inverse, however, in a meta‐analysis with different sample sizes the value of the back‐transformation depends on the specified sample size. Typically, the harmonic mean of sample sizes is used in the back‐transformation.11

3We report results of meta‐analyses with five studies estimating the prevalence of hepatitis C virus (HCV) infections in the general population of Nepal, which constitute a subset of an unpublished dataset with 28 studies.12 This unpublished dataset comprises testing for a total of 972 123 individuals among whom 3696 were HCV antibody positive. The prevalence across studies ranged from 0% to 18.4% with a median of 0.5%. We restrict ourselves to the five‐study subset for didactic reasons; the same issues encountered in this subset also exist in the full dataset.

4We conducted classic meta‐analyses using the arcsine, Freeman‐Tukey double arcsine, and logit transformations, respectively. Furthermore, we fitted GLMMs implicitly using the logit transformation. Details on the statistical methods are provided in Appendix A. We used R function metaprop from R package meta 13 (see Supporting Information). Results are summarized in Table 1.

5Under the fixed‐effect model, results depicted as transformed proportions (middle column in Table 1) are very similar for the two methods using the arcsine and logit transformations, respectively. Whereas the random‐effects estimates are also very similar with a slightly smaller confidence interval for the arcsine transformation, the results for the two logit methods are rather different due to a very different estimate for the between‐study variance.

6For easier interpretation, results are back‐transformed to the original scale. Due to the small prevalences, we express results as HCV infections per 1000 observations. In Table 1 (right column), the results using the inverse of the Freeman‐Tukey double arcsine transformation based on the harmonic mean of 85 are highly irregular with HCV prevalences and confidence limits exactly equal to zero. Under the fixed‐effect model, all of the other three methods show very similar results. Conversely, under the random‐effects model, results for the classic meta‐analysis method using the logit transformation are very different from the other results.

7Looking at Figure 1, we see that the meta‐analysis estimators are reasonable summaries of transformed prevalences. On the other hand, back‐transformed meta‐analysis results are clearly off the mark in Figure 2 with meta‐analysis estimators smaller than all individual study results. Note that the back‐transformation works as expected for individual study results, eg, the prevalence is 1/29 = 0.03448 for study 26, which corresponds to 34.48 HCV infections per 1000 observations.

8The harmonic mean of 85 is obviously the wrong choice in this meta‐analysis with sample sizes ranging from 29 to more than 200 000. Figure 3 shows the influence of sample size on meta‐analysis results (see also Table A2). For sample sizes between 10 and around 120, results are exactly zero for the back‐transformation of the Freeman‐Tukey double arcsine transformation. The number of HCV infections per 1000 observations then steeply increases up to a sample size of 500 when the effect of sample size starts to slowly level out.

9As noted earlier, the results of the random‐effects model are very different for the two logit methods due to different between‐study variance estimates. This discrepancy can be explained by looking at the confidence intervals of individual studies in the corresponding forest plots (Figures 4 and 5). Confidence intervals, based on the normal approximation, are much narrower for the two smallest studies in the classic random‐effects meta‐analysis (Figure 4) than the confidence intervals, based on the Clopper‐Pearson method taking the binomial distribution into account,(14, 15) in the GLMM meta‐analysis (Figure 5). Apparently, in these two small studies with only 1 HCV infection and less than 50 observations, the assumption of a normally distributed logit transformed proportion is not fulfilled. With increasing numbers of infections and sample sizes, approximate and Clopper‐Pearson confidence intervals get closer to each other. Obviously, the very narrow confidence intervals of the two smallest studies result in an inflated between‐study variance estimate leading to a larger estimate for the pooled mean HCV prevalence and a much wider confidence interval for the pooled mean HCV prevalence.

10Our case study shows that meta‐analysis results based on the back‐transformation of the Freeman‐Tukey double arcsine transformation11 can be very misleading and even smaller than all individual study results. We observe similar undesirable results in a meta‐analysis using the complete dataset with 28 studies. To our knowledge, this is the first publication reporting such an anomaly and erratic results.

11In our view, the main reason for this unexpected behaviour is the very extreme pattern of sample sizes that range from 29 to more than 200 000. The harmonic mean of 85 is much smaller than 3 of the 5 sample sizes. For such highly skewed sample sizes, the harmonic mean is by definition rather small, which may result in nonsensical back‐transformed probabilities.

12In order to prevent misleading conclusions for the Freeman‐Tukey double arcsine transformation, several sample sizes could be used to evaluate the sensitivity of meta‐analysis results; however, this may lead to diverging meta‐analysis estimates. In our example, using the arithmetic or geometric mean in the back‐transformation (see Table A2) would result in random‐effects estimates of 1.96 and 1.59 HCV infections per 1000 observations, respectively. Here, results for the harmonic mean are obviously wrong; however, it is rather unclear whether to rely on the results for the arithmetic or geometric mean. All other transformations (arcsine, logit, and log) do not have this intrinsic problem in the presentation of meta‐analysis results.

13Overall, the arcsine transformation appears to be the best classic method for the meta‐analysis of single proportions. However, as application of GLMMs for meta‐analysis is nowadays straightforward due to its implementation in common software, there is neither a real reason nor a clear advantage for using an approximate method. Accordingly, we support the viewpoint of previous works,(10, 16, 17, 18) recommending the use of GLMMs for the meta‐analysis of single proportions. From our perspective, the only disadvantage of a GLMM is that individual study weights are not available, which we consider as a minor drawback; analysts seeing this differently should use the arcsine transformation.

14Our recommendation is purportedly in contrast to advice by Barendregt et al1 promoting the use of the Freeman‐Tukey double arcsine transformation over the logit transformation. However, this publication only considered these transformations under the classic meta‐analysis model. We agree with Barendregt et al1 that the use of the logit transformation is problematic in inverse variance meta‐analyses with small event numbers or sample sizes; this is also visible in our example. These problems with the logit transformation under the classic meta‐analysis do not translate to GLMMs. The classic meta‐analysis model assumes that treatment estimates of individual studies follow a normal distribution that is obviously critical in studies with small numbers of events and observations. The arcsine and Freeman‐Tukey double arcsine transformation are less affected by this normality assumption than the logit transformation. However, GLMMs taking into account the binomial structure of the data are not affected by this problem at all.(10, 16)

15Our case study shows that the Freeman‐Tukey double arcsine transformation should only be used with special caution for the meta‐analysis of single proportions due to potential problems in the back‐transformation of meta‐analysis results. In our view, a sensitivity analysis using other sample sizes is mandatory for this transformation. GLMMs seem to be a promising alternative which is nowadays available in common meta‐analysis software.

16The author reported no conflict of interest.