Evaluating the “return on patient engagement initiatives” in medicines research and development: A literature review

1There is increasing consensus among stakeholders that patient engagement in research and development (R&D) of medicines provides benefits for patients, researchers, industry, regulatory bodies, payers and policy makers.1, 2, 3 The case for patient engagement is often presented in ethical and political terms referring to fairness, transparency and accountability.4, 5 Methodological arguments consider the experiential knowledge of patients—acquired by their personal experience of a condition—as valuable to improving the quality and relevance of the research.6, 7, 8 The inclusion of patients in decision making about the development of new innovative medicines is a substantial change, requiring time and (financial) commitments from researchers, industry and patients.2, 4 Despite efforts to promote and support patient engagement in research, the prevalence of patient engagement in medicines research and development remains low.9, 10 Patient engagement has not been fully embedded in the health research system, partly because it is not yet clear to all involved what the added value is.11 To address this need, an increasing number of studies aim to evaluate the impact of patient engagement, underscoring the growing interest in the “return on engagement,” or why it makes sense for patients, society and industry.2, 12

2The perceived value of patient engagement practices can vary for different stakeholder groups, and the metrics of interest will therefore differ accordingly.13 For example, for researchers and industry partners it might be about evidence that patient engagement improves the quality and efficiency of research and the uptake of findings, whilst for patients it might be more about influencing the R&D agenda to develop medicines for unmet needs. Some argue that evidence is needed to justify the ‘business case’ for engagement. This could also help to establish a financial model to support engagement.2, 14, 15 Evaluation could also define the genuine value of patients’ contributions, contributing to valued rather than tokenistic inclusion for appearances’ sake.16 There is also some resistance; people are concerned about assessing impact too simplistically. Some question whether it is fair to evaluate the value of patient input in isolation, and not that of others such as key scientific leaders,12 not least because it may be the synergy of working in partnership that produces benefit.17 As mentioned by Staniszewska, it is important to recognize that “any form of measurement sits within a political or ideological context that cannot be ignored.”13 Nonetheless, there is a desire to assess the impact of patient engagement, to demonstrate better decision making, avoidance of previous errors and a contribution to continuous efficiency and quality improvement.15, 16, 18

3Despite this perceived importance of assessing the return on patient engagement, little is known about “what” to evaluate, and even less about “how.”19, 20, 21 A number of researchers have tried to assess how patient engagement makes a difference.3, 5, 8, 12, 22, 23, 24, 25, 26, 27 Although there is no standardized way to assess the impact of patient engagement, very similar benefits, costs and challenges are reported in literature reviews.4, 17, 19, 20, 28, 29, 30, 31, 32 The current assessment of patient engagement is considered weak, partly because much of the evidence is mainly anecdotal17 and because methods used have not captured the complexity, context or mechanisms of change.17, 33 Previous studies have identified a number of gaps in the literature and identified challenges such as the delayed nature of impact, inconsistent terminology, absence of accepted criteria for judging the success or quality of research, no agreed evaluation methods or framework and few reliable measurement tools. The absence of a control group—identical research carried out without patient engagement—is problematic too, particularly in an area of science where direct comparison to an existing standard is routinely demanded.8, 16, 34, 35 It is argued that to build an evidence base, some level of consensus on measurable impacts is needed, whilst others state that the outcomes of engagement cannot easily be quantified.13, 30, 36 In sum, it remains unclear what methods can be best used to evaluate patient engagement.

4To address the need for means of determining the “return on engagement,” the aim of this paper was to scope, review and summarize the literature on monitoring and evaluation of patient engagement. Many publications present useful guidance for conducting patient engagement and assessing the quality.37, 38, 39 Evaluation studies focus mainly on qualitative methods and only occasionally link to specific outcomes.12, 33, 40, 41 Therefore, this paper is focused on ways to evaluate patient engagement with both qualitative and explicitly quantitative methods.

5This work is part of the PARADIGM project, a public‐private partnership that is developing ways to ensure that patients are always meaningfully involved in the development of medicines. The impact of patient engagement may differ at different points in the development of a medicine. Accordingly, PARADIGM focuses on three decision‐making points during R&D at which point integration of the patient perspective is considered likely to be valuable, specifically as part of research priority setting, design of clinical trials and at early dialogues with regulators and health technology assessment bodies. Each of these represents a point at which engagement can influence effective planning and implementation, and demonstrate impact on the final product.

6We undertook a scoping review of published academic and grey literature as recommended by Arksey and O’Malley, also drawing on Mays et al and Peters et al42, 43, 44 Scoping reviews are similar to systematic reviews in that they follow a structured search process; however, they are performed for different reasons.45 Our aim was not to answer a precise question addressing the effectiveness of a certain practice, as in a meta‐analysis, but to provide an overview of the breadth of the available literature about evaluating patient engagement.

7Whilst the review is concerned with patient engagement at the three key decision‐making points, we used broader search limits to ensure capture of related publications in other areas of health research. One of the challenges was the variety of terminology. For example, the words “measure,” “metric” and “indicator” are often used interchangeably and their definitions may vary, if they are stated at all. Furthermore, the terms used for “patient engagement” differ globally. In this paper, we use the term patient engagement; in our search, we included terms such as public involvement, patient participation, community engagement and user involvement. In Table 1, we provide definitions of terms developed by the authors and as we used them in this review.

8Prior to the database search, we did a search to identify a tentative sample set of relevant studies for a snowballing exercise. Using broad key words, we searched Google Scholar for published articles and Google for grey literature. We also searched the Patient‐Centered Outcomes Research Institute (PCORI) database46 and the INVOLVE evidence library.47 A snowballing exercise using references and citations from these articles provided a starting set of publications that informed the protocol for the main review. This is recommended for the clarification of concepts and search terms when interrogating large, diverse fields of literature.48

9Accordingly, with the assistance of a specialist librarian, we searched CINAHL, Embase, Medline, PsychINFO and PubMed databases for peer‐reviewed published literature. The following key words were used “patient engagement” combined with « AND» “research” « AND» “outcomes,” including a variation of terms combined with « OR». An overview of all search terms can be found in Table 2.

10Grey literature was searched using the same terms; items recommended by consortium partners and external stakeholders were added manually, and reference lists of items included were searched for additional publications. All searches were conducted between 1 May 2018 and 31 July 2018. The search was limited to publications in English. We excluded articles that did not provide information on possible evaluation approaches or outcomes associated with patient or public engagement. No date limits were applied other than a cut‐off of publications after 31 July 2018. Following completion of the search, duplicated items were removed.

11Two researchers (TF, LV) independently screened all items’ title and abstract. To ensure inter‐rater reliability, items were marked for inclusion or exclusion with each researcher's initials, discrepancies were discussed and consensus reached. Both researchers read all the selected items in full and followed up references for final inclusion. At this stage, further exclusions were made of items that did not include methods for evaluating outcomes and/or impact of patient engagement practices in health research or health technology assessment—discrepancies were discussed, and consensus agreed for final inclusion in the data extraction and analysis. Figure 1 demonstrates the number of articles identified, screened, selected and reviewed.

12TF and LV developed a data extraction sheet to record relevant information from each item, including the publication year and focus, country of origin, methodology, patients involved as partners, use of a framework or model, definitions included, outcome and/or impact on research, benefits and costs per stakeholder group, measurement or evaluation methods suggested or applied. When available, context and process criteria were also included. Both researchers extracted data independently from 30% of the scientific articles and then compared their findings to agree the approach to data extraction. Thereafter, all the peer‐reviewed published papers’ data were extracted by LV and the grey literature by TF and LV.

13Data were thematically analysed following Braun and Clarke's approach.49 To achieve the summary, we coded data using the review question, aim and objective as included on the data extraction sheet. Codes used were benefits (B), costs and challenges (C), outcome or impact on research (O or I) and types of benefits or costs and challenges. Codes were then clustered into themes, which were agreed by TF and LV; themes were identified deductively and include benefits, costs and challenges for each stakeholder group, benefits, costs and challenges for research per decision‐making point, indicators, methods or tools. LV and NG clustered the indicators, methods and tools into qualitative and quantitative types. Benefits and costs were mapped to suggested or applied indicators and tools or methods. The decision‐making point focus of articles was interpreted by the researchers if not defined in the article. Benefits and costs that could not easily be linked to one particular decision‐making point were analysed separately. LV and TF agreed on the data analysis strategy, and sections of the analysis were cross‐checked by comparing interpretations of results; inconsistencies were discussed and agreed.

14The preliminary results of the review were presented and discussed during a PARADIGM meeting held in London, 18 July 2018. This session provided valuable input on how best to present and categorize the results. Participants in the meeting included representatives from patient organizations, pharmaceutical companies and academia with an interest and considerable expertise in patient engagement. Based on the discussions during the meeting, it was agreed to structure the results per key decision‐making point and per stakeholder, including benefits and costs. These members of the PARADIGM consortium were involved in writing this article; their interpretation of results informed the discussion and conclusion. Furthermore, their contributions to the entire research process informed the direction of research, the terminology and definitions used in this article.

15A total of 91 documents met the eligibility criteria (academic literature n = 77 and grey literature n = 14). Included documents were published between 2000 and 2018 and focused mainly on the health research field. We found limited documents in the field of regulation and health technology assessment. Most documents were published in the United Kingdom in an academic setting. We found largely qualitative studies and literature reviews. Sixteen studies reported that patients were involved in the study as partners. Table 3 provides an overview of the characteristics of included documents.

16In this section, we present the findings of our review, first considering the three decision‐making points, which were relevant to our search. Not all reported benefits and costs could easily be linked to one decision‐making point. These referred to overall benefits and costs for stakeholders and general costs or challenges for research and development. Therefore, we report them separately.Additionally, reported benefits and costs were omitted where they related to other phases of the research process (such as interpretation of research findings or dissemination of results).

17A total of 18 benefits and five costs of patient engagement at three R&D decision‐making points were identified. These were grouped into 11 domains and mapped with 28 possible indicators for their evaluation. Tables 4 and 5 provides an overview of indicators per domain. Please refer to Appendices S3–S5 for more detailed indicators, evaluation methods and tools.

18Literature suggests that patient engagement in research priority setting has several benefits. We identified four unique benefits and nine possible indicators. We clustered the benefits into three domains: usability benefits, societal benefits and funding benefits. Usability benefits refer to impact on the topic generation and prioritization process, for example more relevant topics and priorities based on patients’ needs3, 4, 15, 17, 20, 22, 23, 29, 30, 50, 51, 52, 53, 54, 55 and the relevance of studies, for example more relevant research questions and medical interventions or technologies.30 Societal benefits refer to the way public and private resources are allocated, for example more appropriate resource allocation based on patients’ needs.30 Funding benefits refer to new funding and funding opportunities, for example success in gaining funding due to enhanced credibility of research proposals.25, 29, 30, 31, 32, 56, 57, 58

19In the literature, quantitative methods are used to collect information about the perceived importance of studies by patients, the perceived influence of stakeholders in research priority setting,23, 59 or to compare academic and lay scores assigned to research proposal evaluation.60 For example, studies suggest rating the importance or influence of partners in developing the research topics.23, 59 Qualitative methods are used to explore the relevance of research topics and how patients’ experiential knowledge helped shape the research question.30 The Patient‐Centered Outcome Institute (PCORI) uses mixed methods (survey, focus groups, database review) to explore the perceptions incorporated into the topic selection process and the kinds of research gaps documented as important to patients and other stakeholders that were not previously identified.61 Quantitative methods could also be used for comparison of academic and lay scores assigned to research proposals.60 Qualitative methods are suggested for exploring similarities and differences in research priorities.15 For example, Brown et al invited patients with diabetes to focus groups to identify research priorities. Results were analysed using the constant comparative method and compared with current expert‐led research priorities in diabetes.62 Additionally, documentary analyses (eg review of minutes, grant applications, reports) may be conducted to compare patient input and responsiveness to patients’ ideas.54, 61, 63

20We identified ten unique benefits of patient engagement for the design of clinical trials, including 13 possible indicators. We clustered the benefits into three domains: ethical benefits, methodological benefits and study quality benefits. Several studies described ethical benefits such as a more appropriate, inclusive and sensitive research design.8, 17, 29, 30, 52, 55, 58 Furthermore, studies described methodological benefits such as more appropriate wording and timing of research instruments and interventions,17, 20, 22, 24, 25, 27, 29, 31, 55, 56, 64, 65, 66, 67, 68 and improved consent forms and accessible recruitment materials.4, 20, 24, 25, 29, 31, 40, 55, 56, 67, 68 Study quality benefits are also reported, for example improved trial recruitment and retention.23, 24, 29, 40, 69

21The literature suggests several indicators and methods for the evaluation of patient engagement in the design of clinical trials. For example, Guarino et al measured participants’ understanding of the study consent form, using the Informed Consent Questionnaire‐4 questionnaire. The reading levels of the consent forms were assessed using Flesch‐Kincaid reading level scores.70 Rating the impact of patient engagement on study volunteer attitudes about aspects of the participation process (eg ease of understanding the informed consent form; convenience of study visits and procedures) is also suggested.59 Other studies suggest collecting data on the number of studies that gain research ethics committee approval,12 the number of protocol amendments59 and the number and type of patient‐reported outcomes.61 Furthermore, several studies have assessed study quality benefits, for example recruitment rates, using different quantitative methods.40, 69, 70 Iliffe, McGrath and Mitchell40 compared recruitment levels before and after the involvement of the public. Guarino et al70 also conducted a comparison; they assessed the effect of two different consent documents on recruitment levels using one consent form developed by a consumer focus group compared with another developed by the study investigators. Ennis and Wykes conducted a quantitative analysis of successful recruitment by studies where patient engagement was undertaken. A change in patient engagement over time was assessed by correlating study entry order (studies were ordered by the date identified) with the level of patient engagement. Additionally, suggested indicators include recruitment and retention rates among hard‐to‐reach populations,61 the number of dropouts for reasons other than adverse reactions, the total number of changes made to clinical trial communications as a result of patient feedback,59 and the number of studies completed within a particular time frame.3, 61 Validated questionnaires such as the Client Satisfaction Questionnaire‐8 measure overall satisfaction of study participants.70 Qualitative methods are mostly suggested for gathering information about participants’ experiences of taking part in a clinical trial.15

22We identified five unique benefits of patient engagement in regulatory processes and HTA, including four possible indicators. The benefits can be categorized into three dimensions: instrumental benefits, study uptake benefits and developmental benefits. Instrumental benefits are related to improving the relevance of assessment to making better quality decisions, for example higher accuracy in measuring needs and preferences of patients and better quality of assessment and relevance of reports to the local context.71, 72 Study uptake benefits refer to the usefulness of assessments for decision makers and the uptake of evidence by decision makers, for example gaining regulatory approval.2, 73 Developmental benefits include, for example, increasing the public's understanding of HTA and openness of decision processes.72

23Literature suggests a few methods to evaluate the benefits of patient engagement in regulatory processes and HTA. Quantitative methods are suggested to assess study uptake benefits such as the time to response/approval of regulators and a change in the proportion of drugs recommended for reimbursement.3, 74 Furthermore, quantitative methods could be used to assess the perceived impact. For example, the European Medicines Agency has used a survey to assess the perceived added value of patient input in scientific advice processes and feedback.75 Qualitative methods can also be used to explore measures of change or uptake of patients’ input. For example, Abelson et al76 assessed how patients’ input informed the HTA process through document analysis, interviews and observations. Dipankui et al77 used semi‐structured interviews and document analysis (eg HTA reports, minutes) to evaluate how patient engagement changed the HTA report and its recommendations.

24Limited studies have published costs and challenges. Of those which have, most studies reported increased time and costs for researchers and research institutions due to the practical aspects of planning and managing patient engagement. For example, there are increased time and financial costs from building relationships with the relevant community, setting up user groups, organizing and providing training and education for users and researchers, and the additional time needed for users to read and comment on documentation.20 Only two studies suggest that patient engagement could potentially result in a more homogenous sample or biases in recruitment.24, 67 For example, Blackburn et al24 reported that a more homogenous study sample may have been recruited, since the young contributors encouraged their friends to participate in a study on reproductive health in young people. Furthermore, Brett et al found that studies indicated that patient engagement led to scientific and ethical conflict in protocol design. Also, patient engagement may lead to tokenistic engagement and can lead to power struggles between researchers and patient partners.20 Furthermore, stakeholders have raised concerns that engaged patients may want to see their clinical trials succeed, and as a result, these patients may bias the study findings.59 It was also reported that a number of clinical research professionals fear that patient centricity is pushing them to discard traditional practices, including the use of blinded, randomized controlled clinical trials.59

25Methods to assess costs include qualitative methods to gather insights into the perceived effort of engagement as well as a quantitative method to gather insights into financial costs. For example, the costs and consequences framework developed by Blackburn et al includes questions about costs for researchers such as total costs associated with recruiting patients involved, the total costs associated with training patients involved, the total costs associated with supporting patients, financial payment/rewards, total costs of expenses reimbursed to all patients for their involvement and other costs (including parking permits, room booking, audio‐visual, equipment). A separate questionnaire developed for patients includes questions about the hours spent on engagement, the costs they incurred (eg travel, child care, food and drinks, accommodation) and any costs related to arrangement and planning (for instance changed shifts at work or arranged care for a relative).24 Log sheets are also used to gather insights into time and costs.27 Open questions are used to gather insights into (non‐financial) negative impacts.24

26Studies that assessed patient engagement for individuals and organizations mostly highlighted benefits, costs and challenges for patients engaged, with comparatively less published on the benefits and costs for other groups. Based on our review, suggested dimensions to measure the benefits, costs and challenges for the individuals and organizations involved relate to personal development, skills and knowledge, emotions and meaningful relationships, financial, performance and strategic value, transparency and awareness, trust and mutual respect. A summary of reported benefits and costs for stakeholders can be found in Table 6. Please refer to Appendices S1 and S2 for more detailed information on benefits, costs and challenges for patients and other stakeholders.

27Multiple tools have been developed to assess the benefits and costs for stakeholders. The Evaluation Toolkit is a resource designed for practitioners of the health sector, produced after the completion of a rigorous systematic review of patient and public engagement evaluation tools.78 Boivin et al reviewed the tools and concluded that most tools were designed to collect information from patients and the public; very few instruments measure the perspectives of other stakeholder groups. The authors of the review reported that the outcomes of patient engagement were least often evaluated (55.6% of the tools), in contrast to the engagement process and context. The most common focus of tools that measure outcomes was on perceived, self‐reported impacts. Methods are qualitative (eg interviews, focus groups) and quantitative for perceived self‐reported benefits (eg surveys using Likert scales). Self‐administered questionnaires and surveys were the most common types of tools identified.21

28To address the need for means of determining the “return on engagement,” the aim of this paper was to review the literature on monitoring and evaluation of patient engagement. This review identified a range of benefits, costs and challenges that patient engagement can have on R&D and describes several indicators associated with their monitoring and evaluation. In addition, we summarized the overall reported benefits, costs and challenges for stakeholders involved in patient engagement initiatives. In this section, we reflect on the indicators and methods found in this review and consider the review's methodological strengths and limitations.

29A total of 18 benefits and five costs of patient engagement at the three decision‐making points were identified in this review. These were grouped into 11 domains and mapped with 28 possible indicators for their evaluation. Little is known about the validity and performance of these indicators as most were suggested rather than applied, or used in single studies. Those studies mostly considered a single indicator (eg recruitment rate) for trying to answer a single question (eg Does patient engagement in research lead to better recruitment?). Measuring this may be feasible but may not be useful in predicting impact for other studies, as the factors influencing impact may differ. This has been noted by other authors.36 We argue that currently available indicators are of some use in measuring benefits, but are not sufficient to understand the pathway to impact, or whether the interaction between researchers and patients involved could lead to change in the external environment (eg research culture, structure and practice). We argue that the impacts of patient engagement can best be determined not by applying a single indicator, but a coherent set of measures. Given the importance of context and the complexity of evaluating patient engagement that this review illustrates, we are developing a monitoring and evaluation framework that considers various indicators for patient engagement practices in medicines research and development. This framework is informed by other frameworks and being tested in practice. We will publish our findings of working with a more coherent evaluation approach in medicines research and development shortly.

30This review also looked at methods for evaluation. We identified several quantitative methods to measure the benefits of patient engagement; these mostly assess the benefits on study quality, study uptake and self‐reported benefits. Qualitative methods are mostly suggested for gathering information about experiences, attitudes and perceptions. We agree with others that there is a need for new evaluation methods and tools that focus on observable impact on the research process and benefits for those involved.4, 21, 79, 80 Some argue for broadly applicable, quantitative methods whilst others contend that more subjective, qualitative methods are necessary to capture the nuances of outcomes and impacts of patient engagement.13, 17, 59, 76 Universally applicable evaluation criteria that capture all aspects of engagement are supported for reasons of consistency, reliability and comparison across different projects.16 To build an evidence base, conceptual and practical guidance and some level of consensus on measurable impacts are needed. This has also been suggested by other authors.13, 30 However, whilst a standardized approach may be appealing to health research and development communities, it is problematic in the complex and contextually dependent arenas of patient engagement.81 It might inhibit capacity‐building in projects and makes changes difficult; arguably, this undermines the original rationale for patient engagement. The tension between obtaining comparable data on patient engagement by using metrics (standardized or agreed techniques for gathering information) and tailored participatory evaluative approaches should not be overlooked. By implication, it should be recognized that measures can be valued and applied differently in different contexts; therefore, we recommend discussing relevant and feasible indicators and methods per setting.

31To our knowledge, this is the first literature review that attempts to capture the existing publications about the evaluation of patient engagement practice as it relates to medicines development. It both maps outcomes and impacts of patient engagement with suggested measures for each decision‐making point in R&D.

32Very few publications refer to costs or negative impact of engagement, compared with positive findings. This may be because people tend not to report negative outcomes and impacts despite their being just as important. There were very few studies that considered patient engagement in the HTA process, and only, three publications were authored by (and for) the pharmaceutical industry. Furthermore, of the papers included in our review, very few reported that they had involved patients; therefore, the conclusions derived from the studies may be based on the perspectives of researchers. For this review, a meeting was held to discuss preliminary findings with a broad range of stakeholders in our project and the co‐authors of this paper work for patient representative groups and industry. We therefore feel that our findings may be considered relevant to a broader audience than a predominantly academic one.

33Our focus on the measurement of impact of patient engagement in the development of medicines has resulted in several limitations to our review. Because this is a scoping review rather than a systematic review, we may have missed relevant articles. Our search focused on titles and abstracts of publications and three decision‐making points, which means that some articles (eg related to other time points) have been excluded. We specifically searched for outcomes and impact of patient engagement in the R&D of medicines; therefore, our paper does not include context or process indicators, or the indicators per stakeholder group. Furthermore, we cannot draw hard conclusions about the relationship between input, outcomes and impact with respect to the benefits and costs for the people and organizations involved in patient engagement. Finally, we had to exclude articles not published in English. Whilst we are aware that most publications on this topic are written in English originating from the UK and North America, we acknowledge that we may have missed relevant publications in other languages.

34For patient engagement in the development of medicines to become standard practice at the key decision‐making points of priority setting, clinical trial design and regulatory and HTA processes, benefits need to be demonstrable to all stakeholders. This literature review has mapped benefits, costs and challenges with indicators in current literature. Discrete tools and methods for evaluation are less apparent, as is evidence of their application. The approaches to evaluation we found are largely qualitative, and our review suggests that there are few quantitative tools and no standardized approaches to assessing the outcomes and impact. The reported costs, challenges and benefits are largely congruent, with agreement that there is a need for consensus‐based monitoring and evaluation frameworks that include metrics.

35We suggest that the development of a coherent set of measures warrants further investigation and that the benefits, costs and challenges of patient engagement for all stakeholders should be given more consideration (rather than the current focus on benefits for research). To this end, we will co‐develop and test an evaluation framework with stakeholders using a reflexive monitoring approach in real‐life cases of patient engagement in medicines research and development.

36None declared.