Patient and public engagement in research and health system decision making: A systematic review of evaluation tools

1In the past decade, there have been rapid developments in patient and public engagement1, 2 as illustrated by a tenfold increase in the number of articles published annually on the subject.3 As the commitment to patient and public engagement has grown, so too has the call for robust evaluations.4, 5 Without adequate evaluation tools in place, it is difficult to ensure the integrity of engagement principles and practices, assess the outcomes of engagement, learn from current practices and demonstrate accountability for public investments.

2Evaluative efforts on patient and public engagement have expanded recently, with the development of evaluation principles and frameworks4, 6, 7, 8, 9, 10 and an increasing number of published evaluations.6, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 The development of structured evaluation tools has been slower, and mostly performed through unpublished, project‐specific instruments, thus limiting the potential for comparison and mutual learning across engagement projects. Systematic reviews of evaluation instruments have been conducted for patient engagement in individual health‐care decision making,21, 22, 23 but need to be expanded to other domains of patient engagement, including research and health system decision making.

3The aim of this study was to systematically review and appraise existing evaluation instruments for patient and public engagement in research and health system decision making.

4This work was conducted as part of the Canadian Strategy for Patient‐Oriented Research (SPOR), which promotes patient engagement in research and health system transformation, hence our focus on these 2 engagement domains.24 As part of the SPOR strategy, methodological SUPPORT Units (Support for People and Patient‐Oriented Research and Trials) in each province are mandated with strengthening patient and public engagement. We also conducted a systematic assessment of identified evaluation tool to characterize their main strengths and weaknesses, guide engagement practitioners’ choice of specific instruments and orient the future development of such tools.

5This systematic review was based on CIHR's definition of “patient engagement” as a meaningful and active collaboration in the governance, priority setting, and conduct of research, as well as in knowledge translation.24 The “public” includes any “people who bring the collective voice of specific, affected communities.”24 “Evaluation” refers to “the systematic acquisition and assessment of information to provide useful feedback about some object.”25 Within the context of this systematic review, evaluations could focus on the context, process or impacts of patient and public engagement.6 “Evaluation tool” refers to any instrument that can help to systematically acquire and assess information about patient and public engagement activities. This may include questionnaires, scales, interview guides or observation grids for use by engagement participants, organizational sponsors or external evaluators.

6We chose a critical interpretive synthesis method to guide our review. Critical interpretive synthesis allows for the conceptual translation of quantitative and qualitative studies, as well as non‐empirical papers.26 This approach is particularly well suited to the synthesis of diverse types of literature, such as quantitative and qualitative, published and grey, and health and social sciences, for which the phenomena of interest, populations, interventions and outcomes vary and may not be well‐defined.

7We searched for literature published between January 1980 and February 2016 in the Ovid MEDLINE, Embase, Cochrane database of systematic reviews, CINAHL and PsycINFO databases, without any language restrictions. An information specialist was consulted to help develop, update and execute specific search strategies and bibliographic queries for each database (Table 1).

8Five additional strategies were used to complement the electronic database search: (i) contacting authors of published evaluation reports; (ii) hand‐searching bibliographies from retrieved articles; (iii) searching for unpublished evaluation tools in Google, Google Scholar and the websites of select Canadian and international organizations (eg, the Patient Engagement Resource Hub of the Canadian Foundation for Healthcare Improvement and the INVOLVE library in the United Kingdom); (iv) soliciting recommendations from Canadian and international experts and networks; and (v) soliciting the input of a larger audience through social media.

9We included original and review articles on evaluations of patient and public engagement in research and health system decision making that included evaluation tools, and background papers offering critical discussions of key evaluation tools that pertain to patient and public engagement in research and health system decision making.

10Documents were excluded if they focused strictly on patient and public communication; patient engagement in individual health‐care decisions; mechanisms to engage patients and the public as research subjects; theoretical and/or conceptual frameworks that were not linked to an evaluation tool; and evaluation tools not used in the context of research and health system decision making.

11All search results were transferred to a reference database and duplicates were removed. The titles and abstracts were screened by 2 members of the research team before retrieving the full‐text versions of the references included. Discrepancies between reviewers were resolved by consensus. In total, 4 reviewers were involved in the inclusion process, and 1 reviewed all citations, abstracts and articles.

12Two research team members independently reviewed and summarized the peer‐reviewed articles and grey literature using a standardized extraction sheet, compiling the following information: authors; year of publication; main stated purpose; and the name, dimensions and domains, development procedure, theoretical/conceptual foundation and psychometric properties of the evaluation tool. Tool developers who did not publish the tools but only the process or use of their tool were contacted.

13Assessment criteria were developed in collaboration with the evaluation tools’ intended users. Each evaluation tool was assessed based on 4 criteria:

14Scientific rigour: Was the development of the evaluation tool scientifically rigourous and based on existing evidence on patient and public engagement (eg, based on a literature review in at least 2 databases, assessed for reliability and validity)?Patient and public perspective: Does the evaluation tool take into account the views of patients and the public (both in its development, use and reporting)?Comprehensiveness: Is the tool comprehensive in evaluating the context, process, outcomes and impacts of patient and public engagement?Usability: Is the evaluation tool easy to use (eg, availability and readability level)?

15Scientific rigour: Was the development of the evaluation tool scientifically rigourous and based on existing evidence on patient and public engagement (eg, based on a literature review in at least 2 databases, assessed for reliability and validity)?

16Patient and public perspective: Does the evaluation tool take into account the views of patients and the public (both in its development, use and reporting)?

17Comprehensiveness: Is the tool comprehensive in evaluating the context, process, outcomes and impacts of patient and public engagement?

18Usability: Is the evaluation tool easy to use (eg, availability and readability level)?

19Two members of the research team independently appraised the evaluation tools using the assessment grid and its 5‐point rating system. (See Data S1 for the complete assessment grid.) The evaluation tools’ readability level was assessed using the Flesch Reading Ease test (score of 70 or more, or 7th‐grade level).

20Evaluation tool users, including patients and members of the public with engagement experience, were involved in all stages of the project, using an integrated knowledge translation approach.27 One patient partner was involved in the research team and participated in study design, governance, interpretation and knowledge translation. A steering committee composed of representatives from all sponsoring SUPPORT Units (including patient and public engagement practitioners) met 6 times during the project to review the design, assessment criteria, preliminary findings and knowledge translation strategy. A 1‐day face‐to‐face consensus meeting was held mid‐project to review preliminary findings, discuss assessment criteria and develop the knowledge translation strategy. Thirty‐one people participated in the consensus meeting, half of whom were patients and members of the public with engagement experience, along with researchers, patient engagement practitioners and representatives from national research and health‐care organizations (see Acknowledgement section). An open‐access online evaluation toolkit (http://www.ceppp.ca/en/our-projects/evaluation-toolkit/) including a description of all included evaluation tools was published upon project completion to facilitate dissemination and uptake. End users’ engagement modified the evaluation criteria (eg, adding a focus on usability) and dissemination strategy (eg, open‐access online dissemination).

21After removing duplicates, 10 663 unique references were identified and 648 full‐text articles were reviewed (Figure 1). Forty potential evaluation tools were identified. Thirteen of those were excluded because they were not specific to patient and public engagement (n = 9), did not relate to research and health system decision making (n = 1) or did not actually include an evaluation tool (n = 3). Few evaluation tools’ developers responded to requests for additional information and unpublished tools.

22Twenty‐seven patient and public engagement evaluation tools met the inclusion criteria (full references for those included are found in Table S1). Of the tools included, 14 are specific to research, 11 focus on health system decision making and 2 can be applied to both domains. All of the tools were published in English and the majority of them (85%) were developed after 2005. Most were developed in the United Kingdom (n = 10), Canada (n = 8) and the United States (n = 7). Self‐administered questionnaires and surveys were the most common type of tool identified (n = 15). T he stated purpose behind most of the tools was the improvement of patient and public engagement activities. Table 2 describes the included evaluation tools.

23Table 3 provides a summary of the evaluation tools’ scores for each of the 4 assessment domains. Assessment scores for each individual assessment criteria are included in Supplementary Document 3.

24On average, the evaluation tools scored lowest in the scientific rigour domain. The tools were mostly developed based on stakeholder expertise and experience. Only a small number of tools (11.1%) were informed by a review of literature on patient and public engagement in at least 2 databases and 63% were grounded in a specific theoretical or conceptual framework. Reliability testing was rare (7.4% of tools).

25Patients and members of the public were involved in the tools’ development more than half the time (59%), but mostly in the piloting stage. Most tools (74.1%) were designed to collect information from patients and the public. However, very few instruments measured the perspectives of patients or members of the public in relation to those of their engagement partners (eg, researchers, clinicians and managers). Only 18.5% of tools were explicitly designed with the objective of reporting back evaluation results to patients and the public.

26Five of the tools covered all assessment criteria for comprehensiveness: 2 covering health system decision making and 3 covering research. The outcomes of patient and public engagement were least often evaluated (55.6% of tools), in contrast with the engagement process (74.1%) and context (81.5%). For those tools seeking to evaluate the outcomes of patient and public engagement, the most common focus was on perceived, self‐reported impacts, as opposed to observed impacts by external evaluators.

27Two of the tools covered all assessment criteria for usability. All tools described the purpose of the instrument, and the majority (74.1%) were available free of charge. The most important usability issue identified related to readability, with only 11.1% of tools requiring a 7th‐grade or lower reading level.

28This systematic review documents a recent growth in patient and public engagement evaluation tools development. We identified 27 evaluation tools for engagement in research and health system decision making, most of which were developed in the last decade. These findings suggest that engagement evaluation activity is increasing around the globe and provide an important basis for future evaluation work.

29One of this review major contributions is that it not only identifies existing evaluation tools for patient and public engagement, but it also systematically assesses their main strengths, weaknesses and characteristics using predetermined criteria codeveloped with key stakeholders, thus complementing related work in the area.28 The goal of our assessment grid was not to provide an overall quality score, but aimed to guide user's selection of tools to fit their own evaluation needs. As such, our assessment grid cannot be used to “rank” evaluation tools or to identify the “best” tool, but helps identify the strengths and weaknesses of each.

30A number of potential weaknesses were identified regarding evaluation tools’ development process. First, scientifically rigorous methods must be used to develop evaluation tools, including more frequent psychometric testing and validation studies.29 The fact that only a small number (11%) of instruments are informed by a literature review in at least 2 databases is disconcerting, pointing to a potential duplication of effort (new tools being created because existing ones are unknown) and misalignment with key dimensions of engagement documented in the scientific literature. Secondly, efforts must be made to address the lack of an explicit conceptual framework in most tools, which is significant given the importance of linking empirical evaluation with an explicit theoretical foundation.6, 30, 31 Lastly, the high level of literacy required to understand most instruments should be addressed, particularly because patients and members of the public are the target users of most evaluation tools and because engagement with vulnerable populations is a frequent concern.32

31In line with the ethos of participation, key stakeholders are often engaged in the development of evaluation tools for engagement. However, the involvement of patients and the public has largely been limited to the data collection stage and rarely extended to the design of evaluation instruments or the reporting of evaluation results.

32The predominance of context and process evaluation instruments is surprising, given the frequent calls for greater evaluation of the impact of engagement outcomes.33 Evaluating the context and process of engagement is consistent with the objective of developmental and formative evaluation as a means of improving engagement practices.7, 12, 34, 35 The number of evaluation tools measuring “perceived self‐reported impacts” as the main measure for outcomes suggests a need for new evaluation tools based on observable impacts.36

33The main strengths of this review are its comprehensive search strategy and rigorous appraisal of existing tools based on predetermined criteria codeveloped with a broad group of evaluation users. To limit the possibility of omitting relevant material, the search strategy was adapted for each type of database with an information specialist.

34Two specific issues may have limited our ability to identify all relevant tools. First, the poor indexing of evaluation tools in some search engines may have contributed to the relatively low number of tools identified. In a further update of this review, additional search engines could be considered to identify additional material. Second, the fact that practitioners rarely publish their evaluation tools—often because of space limitations and journal editors’ publication policies—may also have limited the identification of relevant tools. Many tools have been developed for a single project or for internal use by organizations without being made publicly available. These limitations could be addressed through broader use of publication reporting guidelines for patient and public engagement research.14, 37 Evaluators and journal editors should pay special attention to publishing evaluation instruments, and properly index them to facilitate the development and use of common instruments across studies.

35Furthermore, a number of assessment criteria could not be fully assessed due to a lack of published information. It is possible that certain assessment criteria were applied but not reported by tools’ developers (eg, literature review was performed but not reported), thus reducing the assessment score of the tool. The readability level of about a third of the tools could not be assessed because the complete tool was unavailable.

36Patient and public engagement practitioners have access to a broad set of evaluation tools, most of which have been designed to help improve the quality of engagement activities. When developing new instruments or refining existing evaluation tools, particular attention should be given to literature review; alignment with engagement frameworks and theories; readability; impact assessment; psychometric testing; and involvement of patients and the public in evaluation design and reporting.

37Most tools identified are targeted instruments evaluating a precise dimension of patient and public engagement, usually concentrating on context, process and perceived self‐reported impact. Future developments could complement existing instruments with more comprehensive evaluation tools that can be used across multiple projects, with evaluation tools’ using a dyadic approach to data collection (eg, assessing engagement from the perspective of patients and their research partners), as well as instruments measuring observable engagement impacts by external evaluators.

38The tools identified in this review were assembled and disseminated by means of an open‐access repository of evaluation instruments (http://www.ceppp.ca/en/our-projects/evaluation-toolkit/), thus strengthening engagement practitioners’ capacity for evaluation and reducing duplication of efforts when appropriate instruments already exist. Given the rapid growth of the engagement evaluation field, it is possible that relevant evaluation tools are under development or that existing tools are being refined: an update of this review is recommended in a few years.

39A growing set of tools is available for the evaluation of patient and public engagement in research and health system decision making. Knowledge of the tools’ specific strengths and weaknesses can guide practitioners in choosing an appropriate instrument for their evaluation needs. Practitioners developing new tools should place greater emphasis on scientific rigour, the involvement of patients and the public in evaluation design and reporting, and the readability of evaluation instruments. The identification, appraisal and dissemination of existing evaluation tools in an open‐access online repository constitute an important contribution of this review in strengthening collective capacity for evaluating patient and public engagement.