Higher number of items associated with significantly lower response rates in COS Delphi surveys

1Multilevel linear regression was carried out to investigate what design characteristics were associated with response rates in studies that had included a Delphi study for core outcome set (COS) development. Two characteristics were significantly associated with a lower response rate. COS studies with larger panel sizes had significantly lower response rates, and studies that included a higher number of items had significantly lower response rates.Other characteristics, all previously identified as potentially influencing response rates, including item order, length of time between rounds, length of time each round is open, format of feedback, and details of reminders, were not reported in the Delphi study reports.The Delphi studies included in this study were predominantly e-Delphi studies. We were therefore unable to compare methods of delivery to investigate any impact this might have on the response rate.Previously, only one study has investigated the impact of design features on response rates, specifically the impact of item order within a Delphi survey. It showed that health professionals were less motivated to respond when clinical items appeared first. This is the first study to explore design characteristics beyond this.COS developers need to pay particular attention to panel size and number of items when designing a COS development study.Studies within studies to answer research questions identified in this study should be carried out, and will be an efficient and timely way to address the research uncertainties identified in this study.

2Multilevel linear regression was carried out to investigate what design characteristics were associated with response rates in studies that had included a Delphi study for core outcome set (COS) development. Two characteristics were significantly associated with a lower response rate. COS studies with larger panel sizes had significantly lower response rates, and studies that included a higher number of items had significantly lower response rates.Other characteristics, all previously identified as potentially influencing response rates, including item order, length of time between rounds, length of time each round is open, format of feedback, and details of reminders, were not reported in the Delphi study reports.The Delphi studies included in this study were predominantly e-Delphi studies. We were therefore unable to compare methods of delivery to investigate any impact this might have on the response rate.Previously, only one study has investigated the impact of design features on response rates, specifically the impact of item order within a Delphi survey. It showed that health professionals were less motivated to respond when clinical items appeared first. This is the first study to explore design characteristics beyond this.COS developers need to pay particular attention to panel size and number of items when designing a COS development study.Studies within studies to answer research questions identified in this study should be carried out, and will be an efficient and timely way to address the research uncertainties identified in this study.

3Multilevel linear regression was carried out to investigate what design characteristics were associated with response rates in studies that had included a Delphi study for core outcome set (COS) development. Two characteristics were significantly associated with a lower response rate. COS studies with larger panel sizes had significantly lower response rates, and studies that included a higher number of items had significantly lower response rates.

4Other characteristics, all previously identified as potentially influencing response rates, including item order, length of time between rounds, length of time each round is open, format of feedback, and details of reminders, were not reported in the Delphi study reports.

5The Delphi studies included in this study were predominantly e-Delphi studies. We were therefore unable to compare methods of delivery to investigate any impact this might have on the response rate.

6Previously, only one study has investigated the impact of design features on response rates, specifically the impact of item order within a Delphi survey. It showed that health professionals were less motivated to respond when clinical items appeared first. This is the first study to explore design characteristics beyond this.

7COS developers need to pay particular attention to panel size and number of items when designing a COS development study.

8Studies within studies to answer research questions identified in this study should be carried out, and will be an efficient and timely way to address the research uncertainties identified in this study.

9Problems with outcomes measured in trials and wider health research are well documented. Problems include outcome reporting bias,, inconsistency in measuring and reporting of outcomes, and relevance to patients. These problems can lead to the use of ineffective or even harmful interventions and to the waste of health care resources that are already limited. These problems are being addressed by the development and application of core outcome set (COS), the minimum agreed set of outcomes that should be measured and reported in all trials for a specific clinical area. COS are also applicable in other settings such as for use in systematic reviews and routine care or audit. The focus of this article is COS developed for research studies. When developing a COS, it is typical to first gain agreement about “what” to measure, with decisions about “how and when” to measure these outcomes later in the process. This article examines studies that have gained agreement about “what” should be measured.

10Systematic reviews of COS demonstrate the growing number of COS developed for research,,, and an increase in the use of Delphi in their development. This suggests that developers are increasingly adopting a more structured approach to COS development, which in turn, has prompted the publication of the Core Outcome Measures in Effectiveness Trials Handbook Version 1.0, to bring together accumulating methodological work in this area and offer recommendations for COS development.

11A systematic review of studies that used the Delphi technique to determine which outcomes to measure in clinical trials concluded that there was variability in both methodology and reporting. This resulted in recommendations to improve the quality of studies that use the Delphi process for determining outcomes to use in clinical trials, including the recommendation that patients and clinicians be involved, researchers and facilitators avoid imposing their views on participants, and attrition of participants be minimized. Attrition of participants could mean that people with minority opinions drop out of the Delphi study, leading to an overestimation of consensus. The validity of the results will ultimately be affected by response rates. It is therefore important to try to maximize response rates to a Delphi study, minimizing attrition rates and therefore any potential attrition bias. Furthermore, qualitative interviews with COS developers highlighted response rates in Delphi studies for COS development as a priority area for further research and guidance.

12In the context of COS development, we are aware of only one empirical research study investigating the impact of design features on response rates. Brookes et al. used a parallel randomized controlled trial design nested within a Delphi survey for COS development, to explore the impact of item order within a Delphi on the response rates. The impact of item order within a Delphi survey showed that item order mattered, and health professionals appeared to have been less motivated to respond when clinical items appeared first. Response rates in Delphi studies for COS development have not previously been explored beyond this, and so the design characteristics that impact response rates in a Delphi study used for COS development are unknown.

13The objective of this study was to explore the potential impact of different design characteristics on response rates in Delphi surveys within COS development projects. The following hypotheses were considered:The number of rounds will affect the response rate, with the expectation that response will decrease as the number of rounds in a Delphi study increases.The number of items included in a round of voting will affect the response rate, with the expectation that Delphi response rates will be lower in studies that have included a higher number of items.The size of the panel will affect the response rate, with smaller panel sizes having a higher response rate.

14The number of rounds will affect the response rate, with the expectation that response will decrease as the number of rounds in a Delphi study increases.

15The number of items included in a round of voting will affect the response rate, with the expectation that Delphi response rates will be lower in studies that have included a higher number of items.

16The size of the panel will affect the response rate, with smaller panel sizes having a higher response rate.

17Other characteristics, including single vs. multidisciplinary panels; international participation; format of feedback; length of time between rounds; length of time of each round; whether reminders were sent between rounds; mode of delivery; and acute vs. chronic health conditions, were also considered.

18A previous systematic review, with subsequent updates, of COS have identified 259 published studies up to and including December 2016,,,. The methods of those reviews are reported in accordance with PRISMA guidelines in the original publication. Methodology reporting of older studies was poor; this study was therefore limited to more recently published studies in the hope that they would report relevant information and data. Studies from the three update reviews,, that included Delphi in their methods to develop a COS were eligible for inclusion in this study.

19Ongoing studies that have used DelphiManager in the development of a COS, and that had completed the second round of voting in the study, were also eligible for inclusion in this study. These were identified via the DelphiManager software developer (R.C.). DelphiManager is a web-based system designed to facilitate the building and management of e-Delphi surveys, and includes functionality allowing the researcher to email participants regarding missing responses, and view the accumulating response rates for rounds two and beyond. Permission was sought from each of the ongoing COS developers to include their data in this study.

20Delphi studies that did not report response data, or that did not report both a numerator and denominator for the response rate (e.g., only reporting total percentage response rates), were excluded. Studies with only one round of voting were also excluded from this study. Studies that had not yet completed the second round of voting were excluded from this study.

21First round response rates were described using varying denominators, including the number of participants invited and the number of participants who agreed to participate, making it difficult to combine, compare, and contrast. We therefore analyzed the response rates in the second rounds, where the denominator is the number of participants invited to the second round of voting. This was not necessarily the round named “round 2” in the reports. Furthermore, some studies included an open first round for generating the list of outcomes therefore including a nonvoting round. This was therefore not counted as a first round for the purposes of this study. Hereafter, when using the term “response rate,” we are therefore referring to response rate in the second round. All ongoing study data included in the analysis and write-up of this study has been pseudoanonymized, with the removal of identifiable features, such as place and person names. Details necessary for analysis (e.g., clinical area) were retained.

22The following information was extracted for each included Delphi study:Study details includingSurname of first authorYear of publicationDisease area/nameDesignMode of delivery usedRecruitment methodParticipant contact methodWhether a reminder was sent between rounds (and method of reminder)Number of roundsNumber of items included in each roundNumber of panelsFormat of feedbackLength of time between each roundParticipantsStakeholder groupsCountriesResponse rateNumber of participants overall who completed each roundNumber of participants overall invited to complete each roundNumber of participants by panel who completed each roundNumber of participants by panel invited to complete each round

23Study details includingSurname of first authorYear of publicationDisease area/name

24Surname of first author

25Year of publication

26Disease area/name

27DesignMode of delivery usedRecruitment methodParticipant contact methodWhether a reminder was sent between rounds (and method of reminder)Number of roundsNumber of items included in each roundNumber of panelsFormat of feedbackLength of time between each round

28Mode of delivery used

29Recruitment method

30Participant contact method

31Whether a reminder was sent between rounds (and method of reminder)

32Number of rounds

33Number of items included in each round

34Number of panels

35Format of feedback

36Length of time between each round

37ParticipantsStakeholder groupsCountries

38Stakeholder groups

39Countries

40Response rateNumber of participants overall who completed each roundNumber of participants overall invited to complete each roundNumber of participants by panel who completed each roundNumber of participants by panel invited to complete each round

41Number of participants overall who completed each round

42Number of participants overall invited to complete each round

43Number of participants by panel who completed each round

44Number of participants by panel invited to complete each round

45Authors were contacted by email and asked to provide any missing data where possible.

46Multilevel linear regression was carried out to investigate whether Delphi design characteristics were associated with second round response rate. The models were fitted to reflect the panel structure within studies. The response variable in the model was the percentage response rate in the second round for each panel. Independent variables included for each panel were panel composition (mixed discipline/single discipline), size of panel invited to the second round, and whether the panel was international or based in a single country. Independent variables included for each study were number of rounds in the study and number of items included in the second round. An initial model was fitted with random intercept. The effect of adding random slopes for each independent variable was explored by comparing the model fit using the Akaike Information Criterion. Models were fitted using PROC MIXED in SAS software version 9.4.

47Third and fourth round response rate data were not analyzed due to the small sample size included in this study.

48Thirty studies from the systematic review updates,, used Delphi methodology. Six studies did not report response rates, did not report response rate data that were useable, or only included one round of voting, and were consequently excluded from this study.

49Eight ongoing COS studies (as of February 2018) had completed the second round of voting using DelphiManager. Seven gave permission for their study data to be included and were therefore included in this study. The declined study author cited complex governance procedures, combined with personal circumstances, as the reason for not contributing to this study.

50These results therefore pertain to 31 studies, 24 published and seven ongoing studies that had used Delphi in the process of developing a COS.

51A descriptive summary of key characteristics is provided in Table 1.

52Overall sample sizes ranged from 9 to 678 (median 110). Response rates for the second round of voting ranged from 45% to 100%. These data, per study, are provided in Table 2. The overall response rate is typically 80% or higher, with only four studies where the overall response rate in the second round of voting is below this at 45%, 50%, 52%, and 64%. Email reminders were sent between first and second rounds of voting in nineteen studies to encourage participation and increase response rates.

53Thirteen studies included two rounds, sixteen studies included three rounds, and two studies included four rounds (Table 2). Studies that included two rounds had a range of response rates between 69% and 100%. Studies that included three rounds reported response rates between 45% and 93%. The two studies with four rounds reported response rates of 80% and 86%, respectively. Number of rounds was not significantly associated with response rate (P-value 0.634) (Table 3).

54Three studies did not report the number of items included per round, but this was provided for two studies when authors were contacted by email. This analysis therefore relates to 30 studies. The number of items included in the second round of voting ranged from 8 to 148, and the number of items per study is reported in Table 2.

55The multilevel linear regression analysis demonstrated a significant association between number of items and response rate in the second round (P-value 0.017) (Table 3), where studies with more items included in the second round had significantly lower response rates. The coefficient for number of items is −0.14, so for every 10 additional items included in the round, the estimated response rate drops by 1.4 percentage points. The association between number of items and response rate is displayed graphically in Fig. 1.

56The number of stakeholder panels per study ranged between one and eight. A summary is provided in Table 1, and by study in Table 2. In the study that had eight panels, one of the panels was not invited to participate beyond round 1 (R1), and one was combined with another group after the first round of voting. These two panels are therefore excluded from further analysis, and this study is included as having six panels for the purpose of this study. Two panels were excluded from DelphiManager studies; one of the panels was not invited to participate beyond R1 and so is therefore excluded from further analysis, and another panel was excluded from the final analysis because it had a single member that did not respond to the second round, giving a 0% response rate. The analysis by panel therefore relates to 72 panels (summarized in Table 4, described in more detail in Appendix 1).

57Studies that included only one panel had a response rate range of between 50% and 100%. Studies with two panels had panel response rates between 75% and 94%. Studies with three panels had panel response rates between 53% and 94%, and finally, studies with four or more panels had panel response rates between 24% and 100%.

58In all but one of the published studies with multiple panels, feedback in the second round of voting was provided separately for each panel (i.e., the different stakeholder groups). In the one exception, feedback was provided at the start of the second round of voting as the mean scores and standard deviations for each outcome; they then provided scores by each stakeholder group at the start of the third round of voting. In all but one of the Delphi Manager studies, feedback in the second round of voting was provided separately for each panel. In the remaining study, it was provided as combined percentage distribution across all panels.

59The only independent variable where allowing random slopes improved model fit was the size of the panel. Random slopes for this variable were added to the final model. The multilevel linear regression analysis demonstrated a significant association between the size of the panel and response rate (P-value 0.035) (Table 3), where larger panels had significantly lower response rates. The coefficient for the panel size is −0.08; that is, that the estimated response rate drops by 0.08 of a percentage point on average for every additional member. Therefore, as an example, for an additional 10 members, estimated response rate dropped by approximately 0.8 of a percentage point, and for an additional 50 members, it dropped by four percentage points. Fig. 2 shows the association between size of panel and response rate.

60The range of response rates in the second round for different stakeholder group panels was similar for each panel composition. If panels are considered as mixed (including those described here as multidisciplinary clinical experts and mixed) compared to those of a single discipline (including here the descriptor clinical experts' single discipline, other, and patient and public representatives), then the response rates are also similar. Panel composition (mixed vs. single) was not significantly associated with response rate (P-value 0.598) (Table 3).

6125/72 panels (35%) only included participants from one country (national panels). National panels had response rates of 53%–100%. The remaining 47 (65%) panels included participants from more than one country (international panels), between 3 and 25 countries in the 17 that reported this information (one from correspondence with author). International panels had response rates of between 24% and 100%. Participant countries (national vs. international) were not significantly associated with the response rate (P-value 0.882) (Table 3).

62Insufficient data were reported on the format of feedback provided, the length of time that each round was open, and the length of time between rounds. We were therefore unable to consider these characteristics in the analysis. The Delphi studies included in this study were predominantly e-Delphi studies, that is, delivered electronically, and therefore, we were unable to compare against other modes of delivery such as postal. Finally, the COS in these Delphi studies were developed principally for chronic conditions, again making it impossible to make a comparison against COS for acute conditions.

63Multilevel linear regression was carried out to investigate whether design characteristics were associated with second round response rates in studies that had included a Delphi survey as part of COS development. Two characteristics were significantly associated with a lower response rate: larger size of panels and studies with more items included in the second round.

64Studies that included a higher number of items had significantly lower response rates in the second round. It has previously been shown that odds of response increase for shorter surveys. Use of a shorter list might minimize nonresponse, but this would need to be traded-off against the need for the list of outcomes to be comprehensive. While it might be regarded as more comprehensive to retain outcomes through the rounds, retaining all outcomes through all rounds on a large list may be burdensome to participants and increase attrition between rounds. If the decision is made to reduce the number of items between rounds to lessen the burden on participants, this should be clearly stated a priori and the criteria for doing so defined in advance to avoid bias.

65COS studies with smaller panel sizes had significantly better response rates in the second round. The method of recruitment and participant contact at the start of the study might explain why smaller panels have higher response rates. Smaller panels were contacted directly by the study team (this information is only available for published studies, see Fig. 3) compared to studies with larger panels that utilized an indirect approach of contact such as a website call through a charity or professional organization or the method was unclear. Adopting a personalized approach has been suggested to increase odds of response for surveys more generally, and it would seem that this is also true for Delphi surveys in COS development.

66A Delphi study must consist of a minimum of two rounds (at least one round of feedback) to be considered a Delphi survey. We hypothesized that the number of rounds would have an inverse association with response, as COS developers have previously expressed concern around retaining participants over the course of a study. The number of rounds was not associated with second round response rates in this study. Furthermore, because the first round of voting was reported with varying denominators, we needed to analyze the second round of voting in this study. Participants in these studies were already committed to the study and so are likely to explain the high levels of response rates seen in most COS Delphi studies included in this analysis.

67Panel composition was not associated with response rate when comparing panels of heterogeneous participants with panels of experts from one particular stakeholder group. We hypothesized that single discipline panels would have higher response rates because the outcomes rated by a homogenous group might be more or less relevant to that particular group, and so individual participants might identify more with the outcomes, and therefore more likely to respond. However, with the exception of one study that had some differences in the outcomes being rated between panels, all panels within studies were presented with the same list of outcomes, which could explain why no difference was detected. Interviews are currently underway to explore participant perspectives on how relevant the list of outcomes was for them and whether this affected their behavior and decision on whether to complete the Delphi or not (personal communication).

68We hypothesized that response rates would decrease with international participation but did not find an association between international panels and lower response rates. The number of countries included in most international studies was not reported, so we were not able to explore this. Although this study did not find a significant association, the challenges of working internationally have been highlighted previously, including the resources required for international participation,.

69Sending a reminder in-between rounds has previously reported increasing odds of survey response by more than a quarter. However, there are many underlying complexities in sending reminders, including whether it was an a priori or ad hoc decision, timing of reminder, and length of time the round remains open following the reminder. COS study reports do not include such in-depth information about reminders, which meant we were unable to include this in the analysis in this study. Furthermore, the inclusion of whether a reminder was sent or not was not deemed reliable enough to include as a simple variable as reminder sent or not, as it was plausible that this might not have been reported for all studies where response rates were already regarded as high. A recent COS study asked participants about their experiences of participating in a COS Delphi, and it was concluded that participants did not find it bothersome to receive reminders to encourage timely voting. Furthermore, another study has recently highlighted methods that worked well in relation to recruitment and retention, including the sending of reminders. Reminders should continue to be sent in an attempt to maximize response.

70We identified other potential characteristics that might influence response rates, including the mode of delivery of the Delphi survey. Research has previously shown that internet-based questionnaires are associated with lower response rates than postal. Almost all the studies included in this study were administered online, so we were unable to explore any potential differences between postal and e-Delphi surveys. Other factors, all previously identified as potentially influencing response rates,, including item order, length of time between rounds, length of time each round is open, format of feedback, and the aforementioned reminder complexities, were not reported in the Delphi study reports. These details should be included in future reports to allow comparisons to be made. Furthermore, studies within studies to answer research questions should be carried out and will be an efficient and timely way to address the research uncertainties identified in this study. The final characteristic that we considered in relation to response rates in Delphi studies for COS development was whether the COS was developed for an acute or chronic disease population. Research in clinical trial recruitment has shown that in practice, recruitment and retention rates vary depending on this. We were unable to analyze this for the Delphi COS studies included in this study because they were predominantly developed for chronic conditions.

71This is the first study to investigate the association of a range of characteristics to response rates in Delphi studies for COS development. By including ongoing COS as well as published, the conclusions drawn are current and likely to remain relevant to inform COS development for the foreseeable future. Ongoing studies were identified through the use of DelphiManager only; we therefore do not know the extent to which our findings might be relevant to ongoing COS Delphi studies using other software, although it is not expected that these studies would have any differing characteristics. The use of multilevel linear regression allows multiple panels within a single study to be included in the analysis, without the assumption that individual panels within a study are independent observations. It should be acknowledged that this was an exploratory analysis with a relatively small sample size, and that, these results should be interpreted as indications of potential associations and not definitive causal relationships. A larger study is needed to confirm these findings.

72In summary, this analysis showed that larger panels, and studies with more items included in the round, had significantly lower response rates. COS developers should pay particular attention to these characteristics when designing a COS development study. Suitable early planning is essential to optimize response rates in the Delphi process.