Incentive and Reminder Strategies to Improve Response Rate for Internet-Based Physician Surveys: A Randomized Experiment

1Surveys remain a widely used, cost-effective means of assessing the attitudes, beliefs, and practices of physicians. However, conducting physician surveys is challenging. Response rate, a benchmark commonly used in appraising survey quality, is typically 10% lower in physician surveys than in nonphysician surveys and appears to be decreasing for many physician subgroups. Although a high response rate may not be the best predictor of overall survey quality, low response rates raise concerns about study precision, nonresponse bias, and the overall inferential value of study findings.

2Recent reviews of the literature on physician survey response behavior have identified the method of initial contact, mode of administration, and approach to nonrespondent follow-up as important determinants of response rate. Electronic survey methods, and Internet-based surveys in particular, are becoming increasingly popular owing to their potential for reduced time spent in collecting data, immediate availability of data once collected, improved data quality, and overall cost savings. However, electronic modalities often have lower response rates than paper mailed surveys.

3Although uncertainties remain regarding how to enhance physician response rate in Internet-based surveys, incentives constitute one possible support. Incentives increase the perception of value, trust, reciprocity, and appreciation on the part of the respondent. Research indicates that incentives improve response rates and that monetary incentives are more effective than nonmonetary incentives, yet most studies investigating the impact of incentives on physician survey response have done so in the context of mailed, paper-based forms. There remains a paucity of evidence concerning how incentives work in the context of electronic surveys of physicians.

4Physician survey response can also be improved through multiple follow-up contacts, although the optimum number of contact attempts remains unclear. In the context of an Internet-based survey, sending follow-up reminders via email to invitees who fail to complete an Internet-based survey after the initial invitation costs virtually nothing. However, repeated email reminders may backfire, hardening prospective respondents’ resistance to future requests. Thus, email reminders may be less desirable than reminders via other methods (eg, mail or telephone). In one study, even after sending the nonresponding physicians 4 emailed reminders, the overall response rate improved substantially after investigators sent nonresponding physicians a paper letter with a printed URL that the physicians used to access the Internet-based survey. Little consensus exists regarding what reminder approaches are most efficient and effective in increasing response rates among physicians, regardless of data collection mode. This investigation is a response to specific calls for further testing in this area.

5Despite the importance of physician surveys and ongoing concerns over participation, few randomized trials have examined potential strategies to address nonresponse. Moreover, even when physician surveys are optimally executed (and enjoy high levels of participation) the investigators infrequently describe the challenges they encountered. As such, significant gaps remain in our understanding of best practices in physician survey research. We sought to address these knowledge gaps.

6We hypothesized that a low-cost nonmonetary incentive and a paper mail reminder would improve response rates in an Internet-based survey of physicians. We also hypothesized that a personalized formal letter would have only a slightly greater benefit than a personalized postcard. We tested these hypotheses using a factorial-design randomized experiment in the context of a nationally representative Internet-based physician survey. To our knowledge, no one has applied such a design to study the process of conducting physician surveys.

7From September to December 2015, we conducted a self-administered Internet survey among a random sample of licensed physicians in the United States. The survey sought physician opinions regarding 2 topics that we believed they would perceive as personally important, namely, maintenance of certification and continuing medical education. Maintenance of certification is presently a matter of controversy and debate, and both this and continuing medical education directly affect the professional lives of nearly all physicians. Details of survey development and results will be published separately.

8The survey was administered using Qualtrics (Qualtrics LLC, Provo, UT, USA). All invitees received an email (Textbox 1) inviting them to complete the survey, accessible via an embedded URL link unique to each invitee. Consistent with the protocol used by Braithwaite and colleagues in which it took 5 reminder emails to achieve a 52% response rate, we sent 6 reminder emails on days 11, 20, 31, 36, 48, and 58. A subset of participants also received 1 paper reminder as outlined below.

9[Email subject line: CME and MOC – Survey of physicians' needs]

10Dear Dr. [last name],

11Continuous professional development (CPD), including maintenance of certification (MOC), affects all physicians. Our group is trying to promote changes to make CPD easier and more effective.

12We invite you to participate in a nation-wide survey, and make your voice heard on these important issues! Those who complete the survey can request a free copy of the “Mayo Clinic Handbook for Happiness: A Four-Step Plan for Resilient Living.”

13This survey asks physicians' opinions about MOC, continuing medical education, and online learning. We want to understand what you do to maintain your professional knowledge and skills, what challenges you face, and—most importantly—what needs to change.

14We anticipate this will take about 7 minutes to complete. Your responses will be anonymous. We would be grateful to receive responses by October 21, 2015.

15Click here to start the survey.

16We thank you in advance for your participation. Feel free to contact any of us if you have questions.

17Sincerely,

18[study investigators]

19This study has been approved by the Mayo Clinic Institutional Review Board. All responses will be anonymous (unless otherwise explicitly noted) and strictly confidential.

20-------------------------

21Editorial note: The italicized text was deleted in the “no book offer” group for the initial email and all subsequent reminders. The email text varied slightly for each follow-up email, but the incentive text remained unchanged. CME = continuing medical education.

22This factorial-design study tested 2 interventions (Figure 1). First, we systematically altered the emails inviting physicians to participate, with one version offering a book (the Mayo Clinic Handbook for Happiness: A Four-Step Plan for Resilient Living nominal cost about US $10) upon completion of the survey and the other version making no mention of any tangible incentive (Textbox 1). Although the wording of the reminder emails varied slightly, the presence or absence of the book offer remained constant. We randomly assigned 75% of invitees to receive emails with the book offer. All respondents were offered the book upon completion of the survey; the intervention in this case was the up-front notification regarding the book.

23Second, 7 days after the first email we sent a reminder via paper mail to a subset of invitees using two formats. We timed this mailing to arrive at approximately the same time as the first email reminder. The invitees in one group received a personalized letter (Multimedia Appendix 1, Box 1), printed on institution letterhead bonded paper and sealed in an envelope, asking them to complete the survey using the link they had received via email, or to contact the study investigators if they had not received or had deleted the email. A second group received a similar message via a personalized postcard (Multimedia Appendix 1, Box 2) that included the institution logo on both sides and was signed by one of the investigators. A third group received no paper reminder. We randomly assigned 40% of the initial sample to the letter group, 40% to the postcard group, and 20% to the no paper reminder group. Only 1 paper reminder was sent.

24We obtained a sample of 4648 names and email addresses selected at random from the LexisNexis Provider Data Management and Services database of all licensed US physicians (LexisNexis Risk Solutions, Alpharetta, GA). This study was approved by the Mayo Clinic Institutional Review Board. Survey response was tracked but responses were deidentified before analysis. Randomization for both interventions was done at the same time before the survey began by a research assistant using a random number generator (Microsoft Excel).

25The primary outcome of this study was the survey response calculated using the American Association for Public Opinion Research formula RR2 (Response Rate 2), that is, the number answering at least one survey question divided by the number of surveys sent less those returned as undeliverable. Secondary outcomes included the number of respondents who completed the entire survey, those who actually claimed a book, and time to response (defined as the number of days between the initial survey mailing and the response date). We obtained demographic information on physician sex, age, degree, specialty, and practice location from the LexisNexis database. For analysis purposes we dichotomized physician specialty as generalist (family medicine, general internal medicine, or general pediatrics) and nongeneralist.

26We compared response rates between interventions and across physician subgroups using the chi-squared test and report odds ratios with 95% confidence intervals for dichotomous variables. We used logistic regression to explore potential interactions between the interventions and physician demographic subgroups. We graphed time to response to visually explore its association with reminder emails. We performed statistical analyses using SAS version 9.4 (SAS Institute, Inc, Cary, NC, USA), using two-sided tests and P ≤.05 as the threshold of statistical significance. To achieve 80% power to detect an absolute improvement of 5%, we estimated we would need to invite 4000 physicians (providing 81% power assuming baseline response rate of 10%, 87% power assuming baseline rate of 25%). All participants were analyzed in the groups to which they were originally assigned (“intent to treat”).

27We obtained the email addresses of 4648 licensed physicians, randomly assigned these to intervention arms (Figure 1), and sent them an email invitation. We received notification that 682 emails were undeliverable, leaving 3966 potential respondents (Table 1). Of these 3966 physicians, 451 (11.4%) responded to at least one survey question during the study period and 336 (8.5%) completed the entire survey. Twenty-six physicians clicked the survey link but did not answer any questions, and 84 opted out of the survey. We received no response during the study period for the remaining 3405 emails.

28Of the undeliverable emails, for 673 we received notice that the email address was invalid and for 9 we received automated messages indicating that the physician would only accept emails from approved senders. One representative message read, “To control spam, I now allow incoming messages only from senders I have approved beforehand. If you would like to be added to my list of approved senders, please fill out the short request form (see link below).” We did not fill out such forms.

29Of 2973 physicians who were offered a book in the email invitation, 345 (11.6%) responded to at least one survey question compared with 106 of 993 (10.7%) physicians who were not offered a book (odds ratio 1.10, 95% CI 0.87-1.38, P=.42); see Figure 2. Converting this odds ratio to absolute rates relative to the baseline rate for those not offered a book, the 95% CI for response among those offered a book ranged from 9.5% to 14.2%.

30Among the 336 respondents who completed the entire survey, 224 (66.7%) requested a book. Those who had been offered a book ultimately requested one slightly more often (177/257, 68.9%) than those who had not been offered a book (47/79, 59.5%; odds ratio 1.51, 95% CI 0.89-2.54, P=.12).

31Sixty-six physicians responded before receiving the mailed paper reminders. Among the remaining 3900 physicians, the mailed paper reminders had no significant impact on response rates, with 168/1572 (10.7%) letter recipients, 148/1561 (9.5%) postcard recipients, and 69/767 (9.0%) email-only recipients responding to at least one survey question (P=.35). When dichotomized as any paper reminder (316/3133, 10.1%) versus none, the results were again not statistically significant (odds ratio 1.13, 95% CI 0.86-1.49, P=.36). Converting this odds ratio to absolute rates relative to the baseline rate for those not receiving a paper reminder, the 95% CI for response among those receiving any paper reminder ranged from 7.9% to 12.8%. Comparison of the response rate for those receiving letters or postcards was likewise not statistically significant (odds ratio 1.14, 95% CI 0.91-1.44, P=.26).

32Results for both the book incentive and paper reminder were similar when only those who completed the entire survey were counted as respondents (data not shown).

33We explored associations between response rate and several demographic features. Response rate varied significantly by age (P<.001), with both younger (response rate 8.8% for age < 45 years) and older (9.0% for age ≥65 years) physicians responding less frequently than others (12.8% for age group 45-54 years, 14.9% for age group 55-64 years). We found no significant difference in response rates across specialties as classified in Table 1 (P=.06); however, generalist (family medicine, internal medicine, and pediatric) physicians responded less often (9.4%) than nongeneralist physicians (12.3%; P=.007). Physicians with a doctor of medicine (MD) degree were twice as likely (442/3791, 11.7%) to respond as those with a doctor of osteopathy (DO) degree (9/174, 5.2%, P=.008). We found no significant difference by sex (P=.30) or geographic region (P=.90).

34In multivariate analysis simultaneously accounting for all 5 demographic variables, namely, age, specialty (generalist vs nongeneralist), degree, sex, and region, statistically significant predictors of response were age (P<.001), specialty (P=.003), and sex (161/1329, 12.1% female, 290/2637, 11.0% male; P=.02).

35We explored interactions between the study interventions and demographic features as well. Independent adjustment for age, specialty (generalist or nongeneralist), sex, and geographic region did not change the finding of no effect from the book incentive or the reminder. We found no significant interaction between any intervention and any of these demographic features (Pinteraction ≥.09).

36Figure 3 shows a timeline of cumulative responses and illustrates the continued favorable impact of repeated reminder emails.

37Optimizing the response rate of Internet-based surveys is a matter of great importance to survey researchers. In this randomized experiment, we found that offering a modest incentive had no impact on response rate in an Internet-based survey. We also found that paper reminders had no impact on response rate, whether in the form of a formal letter or a postcard. Generalist (family medicine, internal medicine, and pediatric) physicians responded less often than nongeneralist physicians, and middle-aged physicians responded more often than younger or older physicians, but we found no significant interaction between the study interventions and specialty or age. We observed a meaningful number of additional responses after each of the 6 reminder emails.

38The greatest limitation of this study is the overall low response rate. A recent meta-analysis estimated the average survey response rate among health professionals at 53%; however, a review focused on Internet surveys of physicians found that response rates <20% are not uncommon. Although the low response rate does not affect the internal validity of the study findings, it could affect the generalizability of these results to other survey contexts. We note that despite the low response rate, the confidence intervals exclude what would be considered by most researchers to be a meaningful difference in response rate.

39One possible explanation for the overall low response rate is that email addresses were invalid or not monitored, but we were not notified. If true, this would not affect our conclusions unless there were a systematic bias in the distribution of invalid emails (which seems unlikely). Another possibility is that physicians perceived relatively low value in this incentive—perhaps they simply were not interested in this particular book. Research suggests that an unappealing incentive in some situations may be worse than no incentive at all. For example, one study found that offering free continuing medical education credit to Internet survey respondents actually resulted in lower response rates compared with no incentive. Alternatively, the promise that the book would be delivered at a later date might have been less motivating than an immediately present, tangible incentive. The logistical difficulty of providing an immediate meaningful incentive reflects a limitation of Internet-based surveys generally and among physicians specifically. A third possibility is that the invitees did not find the survey topic interesting. Like other professionals, physicians will not complete a survey if the topical importance is unclear or perceived as low. Although maintenance of certification and continuing medical education appear prominently in current physician discussion venues, it is possible that the invited physicians perceived low personal salience in these topics.

40Strengths of our study include the randomized design and adequate power to detect even small intervention effects, if present.

41Prior investigations of methods for physician surveys, conducted primarily in the context of paper surveys, indicate that incentives increase participation. Our findings suggest that in Internet-based surveys the incentive may play a smaller role, perhaps because the promise of a future reward prompts less motivation than would a tangible gift. This suggestion is supported by prior research indicating that noncontingent prepaid monetary incentives (eg, including US $5 with an initial paper survey invitation) yield higher response rates than contingent postcompletion offers, such as that used in our investigation.

42Research also suggests that sending paper reminders after an initial email invitation or offering the opportunity to complete the survey via the Internet or on paper increases physician survey response rates. We hoped that a single paper reminder might have a similar effect, but unfortunately, such was not the case.

43The field of electronic surveys continues to evolve. Tools to conduct such surveys have become more ubiquitous, more powerful, and more user-friendly in recent years. While these improvements make it easier for health service researchers to conduct Internet-based survey research, it also makes it easier for everyone else to do the same. Thus, physicians (along with many other people) often feel overwhelmed by survey invitations (“surveyed to death”) and other unsolicited emails (“spam”) and may indiscriminately ignore, delete, or opt out of all survey requests. In addition, the increasing prevalence and awareness of electronic threats (eg, computer viruses, phishing, and identity theft) lead people to protect themselves from emails arising from unknown senders. A small percentage of the physicians invited to participate in our survey blocked all emails from unknown senders, a solution that may become more popular in coming years. The evolving landscape of Internet-based surveys will continue to challenge those using this mode of administration.

44Our study has several implications for best practices in survey research. First, the book incentive did not improve response rates. Because this contradicts prior work on incentives in paper surveys, it remains to be seen whether this finding was simply related to the particular incentive or survey topic or whether it indicates a general lack of benefit from contingent nonmonetary incentives in the context of Internet-based physician surveys.

45Second, the paper reminder also did not help, whether in the form of a postcard or a formal letter. Assuming this finding replicates in future work, it suggests that researchers using electronic surveys will need to find other ways to encourage response. Mixed-modality surveys using both paper and Internet will likely achieve the best results.

46Third, repeated email reminders continued to work, at least through 6 reminders spaced at approximately 10-day intervals. The long-term impact of such reminders (eg, potential negative impact on physicians' response to other surveys) could not be evaluated in this study.

47The methods of Internet-based survey research continue to evolve. What was once a powerful and easily accessible tool may no longer continue to be such, given the current state of electronic threats and excessive surveys from marketing, customer satisfaction, and political pollsters as well as health care service researchers. Extra effort may be required to follow other approaches known to improve survey response, such as highlighting the salience of the topic, having someone known to the invitee champion the survey, broadening the options for survey completion to include additional modes (eg, fax, telephone, interactive voice response telephone, or face-to-face), and decreasing the burden of survey completion by utilizing the briefest forms possible.