A Review on Sample Size Determination for Cronbach’s Alpha Test: A Simple Guide for Researchers

1Cronbach’s alpha is a measure of the internal consistency or reliability between several items, measurements or ratings. In other words, it estimates how reliable are the responses of a questionnaire (or domain of a questionnaire), an instrumentation or rating evaluated by subjects which will indicate the stability of the tools. Alpha was developed by Cronbach, which was originally used to measure the reliability of a psychometric instrument. The value of Cronbach’s alpha ranges from zero to one with the higher values implying the items are measuring the same dimension. In contrary, if the Cronbach’s alpha value is low (near to 0), it means some or all of the items are not measuring the same dimension.

2For example, the Depression Anxiety and Stress Scale (DASS-21) that is used to measure the magnitude of respondents’ stress level condition for over the past week, has seven items measured by using a four-point Likert scale (ranging from zero to three). To ascertain whether the items are reliable in measuring the same dimension, a test for Cronbach’s alpha may be used. The Cronbach’s alpha test is usually applied to test the consistency and stability of the questionnaires which measure latent variables. Although Cronbach’s alpha test may be applied in situations other than questionnaire development or validation, there is limited literature of its application in such scenarios.

3Cronbach’s alpha has been applied in research to develop clinical tool. For instance, a study by Berg et al. developed an instrument to measure the balancing ability in elderly patients. The study used a scale consisting of 14 movements of patients such as sitting unsupported, change of position and etc. The movements were evaluated by five physical therapists and they were given a score from zero to four for each movement. The high degree of internal consistency in this study showed that the scale measures the same dimension

4One of the common issues in inferential studies is to determine the sufficient sample size. The lack of knowledge in sample size determination and unfamiliarity with sample size softwares are usually the challenges encountered by researchers especially among those who do not have adequate knowledge in statistics. Sufficient sample size is needed so that research conducted can provide reliable and reproducible evidence that can detect the desired consistency or stability of an instrument (questionnaire). It is an important aspect in research to avoid lack of power of the test due to underestimation of sample size and also prevent the waste of resources due to overestimation of sample size.

5The purpose of this present paper is to provide a simple guide for medical researchers to plan sample size estimation for their studies that involve Cronbach’s alpha test. The sample sizes are determined and presented in the form of tables to estimate the minimum number of samples needed to obtain the desired value of Cronbach’s alpha. Following that, the discussion of this paper emphasizes on the application of the formula and the sample size tables.

6Discussion on Cronbach’s alpha test in terms of concept, applications, statistical test and sample size determination have been widely discussed in the literatures. Some of the discussions related to sample size are summarised in Table 1. The present study emphasizes the review of sample size determination for Cronbach’s alpha and focuses on sample size determination introduced by Bonett.

7The null hypothesis is H0: CA0 = CA1 and the possible selection of alternative hypotheses are:

8Ha: CA0 ≠ CA1 (this option yields a two-tailed test)

9Ha: CA0 < CA1 (this option yields a one-tailed test)

10Ha: CA0 > CA1 (this option also yields a one-tailed test)

11CA0 refers to the value of Cronbach’s alpha in null hypothesis and CA1 refers to the value of Cronbach’s alpha in alternative hypothesis. Most commonly CA0 is assumed and set at 0 in which smaller sample sizes are derived since the gap between CA0 and CA1 is larger. Setting up the CA0 as not equal to 0 is uncommon in research but it is very useful in testing and comparing the hypothesis of two different alpha coefficients.

12The null hypothesis is H0: ρa = ρb and the possible selection of alternative hypotheses are:

13Ha: ρa ≠ ρb (this option yields a two-tailed test)

14Ha: ρa < ρb (this option yields a one-tailed test)

15Ha: ρa > ρb (this option also yields a one-tailed test)

16ρa refers to the value of Cronbach’s alpha in population “a” and ρb refers to the value of Cronbach’s alpha in population “b”.

17Sample size for Coefficient alpha or Cronbach’s alpha was calculated using Microsoft Excel. The formulation was from Bonett based on formula given;

18where

19Sample size was calculated based on power of 0.80 and 0.90 (Power = 1–β) while probability of type I error (α) was set at 0.05 at all time. There are options on whether to use two-tailed test or one-tailed test settings. In this paper, the more-commonly used two-tailed test was chosen with the aim to detect the difference between the two Cronbach’s alpha. There are three things that need to be considered when determining the sample size for Cronbach’s alpha test: the number of items or raters (k), the value of Cronbach’s alpha at null hypothesis (CA0) and the expected value of Cronbach’s alpha (CA1). The value for CA0 and CA1 could be any value ranges from −1 to 1, however CA1 should not be equal to CA0 (CA1 ≠ CA0).

20In order to illustrate the above formula with an example; there are 15 items in a questionnaire of which the reliability of its measurements need to be measured (CA0 and CA1 are identified at 0.0 and 0.7, respectively). Power is set at 90% and the value of alpha at 0.05. The minimum sample size required based on formula (i) and (ii) is as shown below:

21Calculations:

22Therefore, the minimum sample size required for this case study is approximately 18 samples.

23Sample size for comparing two Coefficient alpha or Cronbach’s alpha was also calculated using Microsoft Excel. The formulation was from Bonett based on formula given;

24where

25Sample size was calculated based on power of 0.80 and 0.90 (Power = 1 – β) while probability of type I error (α) was set at 0.05 at all time. There are options on whether to use two-tailed test or one-tailed test settings. In this paper, the more-commonly used two-tailed test was chosen with the aim to detect the difference between the two Cronbach’s alpha. There are four things that need to be considered when determining the sample size for Cronbach’s alpha test: the number of items or raters in group one (k1), the number of items or raters in group two (k2), the value of Cronbach’s alpha at null hypothesis or group one (ρa) and the expected or group two value of Cronbach’s alpha (ρb). The value of k1 and k2 are set to be similar and present as k. The value for ρa and ρb could be any value ranges from −1 to 1, however ρb should not be equal to ρa (ρb ≠ ρa). In order to illustrate the above formula with an example; there are ten items in a questionnaire in group 1 and 2 for which the reliability of its measurements need to be measured (ρa and ρb are identified at 0.3 and 0.7, respectively). Power is set at 90% and the value of alpha at 0.05. The minimum sample size required based on formula (iii) and (iv) is as shown below:

26Calculations:

27Therefore, the minimum sample size required for this case study is approximately 68 samples per group.

28Based on pre-specified alpha, power and effect size, the minimum sample size requirements are shown in Table 2 to Table 3. The pre-specified alpha and power are usually known (0.05 and 80.0% or 90.0%, respectively) and thus, the minimum sample size required is mainly determined by the value of k and the differences of the coefficients in null hypothesis and alternative hypothesis. The minimum requirement for sample size remains relatively unchanged when the value of k increases. Larger sample size was observed when the differences between the coefficients in the null and alternative hypotheses were smaller.

29For one coefficient of Cronbach’s alpha test, if the coefficient of Cronbach’s alpha equals to zero in the null hypothesis, it is possible to derive a smaller sample size of less than 30 to achieve a minimum desired effect size of 0.7 or more (Table 2). However, setting the coefficient of Cronbach’s alpha larger than zero (e.g., CA0 = 0.50) in the null hypothesis could be necessary in some cases and this will yield a larger sample size. (Table 2). For a comparison of two coefficients of Cronbach’s alpha in different groups, a larger sample size is needed to detect smaller effect size when the coefficients in the null hypothesis (ρa) and alternative hypothesis (ρb) are nearly about the same (Table 3).

30For a single coefficient of Cronbach’s alpha test, the application of the sample size tables can be used into two situations, which are sample size determination when CA0 can be assumed equal to zero and sample size determination when CA0 is assumed to be more than zero.

31In the questionnaire reliability studies, the coefficient of Cronbach’s alpha in the null hypothesis (CA0) is always assumed to be equaled to zero, implying that researchers assumed that there is no internal consistency of the whole items or for a particular domain of a questionnaire. Then, the coefficient of Cronbach’s alpha in the alternative hypothesis (CA1) will assume that there is a magnitude of internal consistency (either low, moderate or strong) of the questionnaire or a particular domain of a questionnaire. The minimum sample size requirement depends on the desired magnitude of internal consistency in the alternative hypothesis.

32Higher differences of Cronbach’s alpha coefficients in the null hypothesis and alternative hypothesis will yield the lower sample size. Based on Table 2, in order to achieve an acceptable Cronbach’s alpha of 0.7, when the values of alpha and power are fixed at 0.05 and 80.0%, respectively, the sample size ranges from four (k = 100 and CA0 = 0.0 versus CA1 = 0.95) to 52 (k = 3 and CA0 = 0.0 versus CA1 = 0.5), depending on the number of k (number of items). Hence, Cronbach’s alpha test is suitable for conducting pilot studies when the CA0 is usually assumed to be zero. In studies related to questionnaire development and validation, Cronbach’s alpha is a common indicator for internal consistency and has always been evaluated in the pilot studies.

33Based on the calculation, it is possible that the sample size can be as small as four with assumption of a very high internal consistency such as 0.95 or more (k = 100 and CA0 = 0.0 versus CA1 = 0.95). Although researchers may target a high coefficient of Cronbach’s alpha, however, it is impractical to do so as there is a possibility that it may not be achieved in an actual study. It is more reasonable to target a desired Cronbach’s alpha between 0.7 to 0.8 with an acceptable sample size.

34An example of a sample size statement is as followed: “The aim of the study is to evaluate the internal consistency of particular questionnaire “A”. Questionnaire “A” has 15 items with a five-point Likert scale for every item. The coefficient of Cronbach’s alpha in the null hypothesis and alternative hypothesis are assumed to be equal to 0.0 and 0.7, respectively. Based on alpha value fixed at 0.05, the minimum sample size requirement is 14 in order to achieve power of 80.0%. The calculation is based on the formula introduced by Bonett.” If the questionnaire has a different number of items, researchers will need to calculate the sample size based on the formula in Equation (i).

35Hypothesis testing when CA0 > 0 is rare in literature. However, the concept of testing the null hypothesis with CA0 > 0 may be useful in cases when researchers set a high target of internal consistency of an instrument. Some statistical soft-wares do not provide statistical test to assess internal consistency when CA0 is assumed to be larger than zero. However, with the sample size calculation, researchers can calculate and determine the sufficient sample size to assess the internal consistency when CA0 is assumed to be larger than zero.

36For example, when k = 3 and CA0 = 0.50, based on alpha < 0.05 and power of at least 80.0%, a minimum sample size of 31 is sufficient to detect CA1 at 0.80. A larger minimum sample size is needed (n = 93) to detect CA1 at 0.70 (Table 2). An example of a sample size statement is as followed: “The aim of the study is to determine whether a particular questionnaire “B” has high magnitude of internal inconsistency. Questionnaire “B” has 10 items with a 5-point Likert scale for every item. The null hypothesis has set the coefficient of Cronbach’s alpha is at 0.50 while in the alternative hypothesis, the coefficient of Cronbach’s alpha is set at 0.80. Based on alpha of 0.05, the minimum sample size requirement is 23 to be able to detect at least 80.0% power of the test. The calculation is based on the formula introduced by Bonett.”

37The present article recommends testing hypothesis with CA0 = 0.5 is necessary to test whether an instrument has an excellent internal consistency. However, it is a choice by the researchers to set the value of CA0, including setting CA0 as equals to zero. But, solely targeting a large difference of Cronbach’s alphas between CA0 and CA1 to get a lower sample size is not recommended. Therefore, justification is needed whether testing such effect size is scientifically necessary.

38Hypothesis testing to compare two Cronbach’s alphas in two different groups is also rare in literature. However, this hypothesis testing is useful if the researcher aims to compare two coefficients of Cronbach’s alphas in two different groups. Although some statistical softwares might not provide statistical test to assess such hypothesis, researchers can calculate and determine sufficient sample size for the planned study to compare two Cronbach’s alphas in two different groups.

39For example, when k = 3 and ρa = 0.50, based on alpha = 0.05 and power = 80.0%, a minimum sample size of 183 is sufficient to detect ρb at 0.70. A lower minimum sample size is needed (n = 59) to detect ρb at 0.80 (Table 2). An example of a sample size statement is as followed: “The aim of the study is to determine whether the internal consistency of particular questionnaire “B” in group “two” is higher than in group “one”. Questionnaire “B” has 15 items with a 5-point Likert scale for every item. The coefficient of Cronbach’s alpha in group “one” was reported at 0.70 while the coefficient of Cronbach’s alpha in group “two” is hypothesised to be at least 0.80. Based on alpha of 0.05, the minimum sample size requirement is 207 to be able to detect at least 80.0% power of the test. The calculation is based on the formula introduced by Bonett.”

40This article provides sample size tables and simple guides for sample size estimation for Cronbach’s alpha. This guideline is useful for researchers especially in medical field where the recommended tables will ease them in estimating sample size for reliability studies. For one coefficient of Cronbach’s alpha test, the assumption of “the CA0 equals to zero” will usually yield a smaller sample size, and that is appropriate for pilot studies. However, to ensure an instrument has an excellent internal consistency, testing the hypothesis with CA0 larger than zero (CA0 > 0), such as CA0 = 0.5 is recommended.

41Besides that, the calculation of sample size based on comparing two Cronbach’s alpha values in two different groups is also presented. Justifications are needed to decide whether the testing for extremely low or extremely large effect sizes are scientifically necessary. Hopefully, this guideline could help the researchers in sample size estimation for their researches to assess the reliability of an instrument.