Better Measurement for Performance Improvement in Low‐ and Middle‐Income Countries: The Primary Health Care Performance Initiative (PHCPI) Experience of Conceptual Framework Development and Indicator Selection

1The international community has embarked on the agenda of the Sustainable Development Goals, with primary health care (PHC) a critical platform for achieving universal health coverage (UHC) efficiently, effectively, and equitably.1 Despite global political commitment to investing in and improving PHC, many countries have not prioritized strengthening PHC. Furthermore, the interventions driving PHC system performance improvement over time in low‐ and middle‐income countries (LMICs) are still poorly understood.2 Thus, even when countries decide to prioritize PHC, they often lack the necessary information to pinpoint weaknesses, identify strengths, and improve their PHC systems.3, 4 Better measurement of PHC system performance is a critical first step to identifying areas for reforms that will strengthen performance, as well as evaluating the impacts of these reforms to guide continued performance improvement.

2Filling the performance measurement gap in PHC is one of the primary objectives of the Primary Health Care Performance Initiative (PHCPI), a collaboration of the Bill and Melinda Gates Foundation, the World Bank, and the World Health Organization, in partnership with Ariadne Labs and Results for Development.4 Launched in September 2015, the PHCPI's aim is to catalyze improvements in PHC systems in LMICs and accelerate progress toward achieving UHC through better, more comprehensive, and actionable measurement of quality PHC and better insights into interventions that effectively address numerous performance gaps.4, 5 The initiative is supported by a grant from the Gates Foundation over the period 2016‐2020.

3Through a series of novel conceptual and methodological developments that distill new lessons from existing literature and provide new tools for performance assessment, the PHCPI is improving measurement and advancing understanding of the determinants of PHC system performance in LMICs. The first tool, which was introduced and described by Bitton and colleagues, is a new conceptual framework of PHC system performance that emphasizes service delivery, improving on the existing frameworks’ traditional focus on inputs and outputs.4 The second tool is the compilation of 2 sets of indicators, “Vital Signs” and “Diagnostic Indicators,” to measure performance. These indicators reflect the principal dimensions of the conceptual framework, identify PHC systems’ specific strengths and weaknesses, and are highly relevant at global, national, subnational, and, in some cases, facility levels. The third tool is the creation of a set of composite indices that provide more holistic measures of PHC systems to assess relative performance across countries and broad trajectories of performance over time, which can guide strategic and investment priorities of national and international financers of PHC. The last tool is national PHC performance dashboards that provide comprehensive and multidimensional snapshots of each country's PHC system performance, which will likely be used to inform policymakers, advocates, and diverse audiences about the strengths and weaknesses of PHC systems. Companion tools are also being developed, such as an assessment guide that outlines the steps to comprehensively evaluate PHC system performance with participation from diverse stakeholders as well as a solution strategies model that identifies typologies of PHC systems’ strengths and weaknesses to help in selecting reforms to improve performance. With input from a broad range of global experts and policy practitioners and based on testing in and engagement with partner countries, each of these instruments is continually being refined.

4In this article, we review the PHCPI experience with the conceptual framework development and the selection of core performance indicators, addressing the question of how to measure and understand key aspects of PHC systems’ performance in LMICs. We also describe ongoing work on composite indices and performance dashboards and articulate an agenda for future research to develop better indicators of quality PHC in LMICs. We conclude with a discussion of possible strategies to improve data availability, increase capacity to measure quality PHC in LMICs, and operationalize the measurement agenda to improve performance in LMICs.

5A substantial literature has debated the precise definitions of and distinctions between primary care and primary health care.6 According to Muldoon and colleagues, primary care “describes a narrower concept of ‘family doctor‐type’ services delivered to individuals,” whereas primary health care is broader, encompassing “both services delivered to individuals (primary care services) and population‐level ‘public health‐type’ functions.”7 The PHCPI's conception of primary health care follows this broader definition, incorporating prevention and health promotion at the societal, community, and individual levels, in addition to the provision of primary care services. The PHCPI endorses a vision of a well‐functioning PHC system as one that can manage and fulfill each of these roles, and it is this conception of PHC that guides its work.

6The development of each of the PHCPI instruments reflects substantial input from a group of approximately 30 international experts, representing a broad range of expertise in measurement, health systems research, and primary health care reform implementation in low‐, middle‐, and high‐income countries, including leaders from various LMICs’ ministries of health. Experts were identified through the authors and references of seminal publications in the field of health systems performance assessment and leading institutions in the fields of PHC and health systems strengthening, and were selected to provide a diversity of perspectives from a range of professional backgrounds with global representation, including policymaking at the national level. In future iterations of its advisory committee, the PHCPI seeks to further improve this range of perspectives by including more experts from currently underrepresented world regions and by incorporating more experts with a specialized knowledge of primary health care delivery, such as family medicine. To date, the PHCPI has convened 3 major international advisory meetings. More than 50 country‐level, NGO, World Bank, and academic experts attended the initial global input meeting in July 2014. Two subsequent meetings of the PHCPI international expert advisory committee were attended by 13 experts in July 2015 and by 16 experts in November 2016. In addition, we conducted online surveys to help refine the selection of the indicators before the meetings. Appendix 1 lists the experts who participated in each of the 2 expert advisory committee meetings and responded to each of the online surveys.

7Working in LMICs is fundamental to the PHCPI's objectives, and country engagement will progressively expand as the PHCPI assessment toolkit is further developed and finalized. To date, the PHCPI team has tested early versions of its various tools in a select group of countries, chosen opportunistically to combine this work with ongoing related activities by one or more partner organizations. The PHCPI's conceptual framework and indicators were first applied in Cameroon, helping elucidate subnational disparities in PHC performance across regions. Teams in Mexico used the PHCPI's indicators to systematically identify available data across each of the conceptual framework's domains, and the PHCPI conceptual framework was used in Rwanda to guide the development of a performance management scorecard for utilization by the Ministry of Health. These early experiences provided valuable feedback on the relevance of the conceptual framework to different PHC systems and the feasibility of compiling sufficient data to estimate the PHCPI's indicators and generate meaningful conclusions about PHC performance. As a next phase for country engagement contributing to tool development, the PHCPI is identifying countries and making plans to test its assessment guide. As this work expands to a more diverse set of countries, this will also help identify any issues arising from the generalizability of the tools to different epidemiological and health systems contexts.

8Underlying the development of the PHCPI's conceptual framework is an understanding that the organization and management of service delivery for better population health management as well as the quality of health care services are paramount aspects of performance. This emphasis adds a level of analysis to existing models and frameworks predominantly focused on system inputs and outputs. Historically, the focus of routine health systems measurement, as well as the majority of facility surveys, in LMICs has been on inputs, such as the total number of health workers, medicines, and supplies available, and outputs and outcomes, such as intervention coverage and mortality rates.8, 9 Policymakers, however, often lack data on the service delivery processes by which these inputs produce the desired outputs and outcomes. Furthermore, the experience of patients who receive care, constraints faced by health workers who provide care, and barriers encountered by people who may not interact with the system are rarely measured. For example, systematic and comparable data on how often health workers are present at health centers and the accuracy of their diagnoses are collected in very few LMICs.10, 11 In other words, many LMICs lack measures of quality service delivery.12

9In the first step of developing a conceptual framework, the PHCPI team reviewed the literature on key characteristics and determinants of high‐performing PHC systems. This included a search of peer‐reviewed databases, as well as reports and other publications from multilateral organizations. In the second step, the team reviewed approximately 40 different frameworks and measurement platforms on PHC, health systems, and health scorecards to identify their strengths, limitations, and common features (Appendix 2). During the initial convening meetings with country consultants at the World Bank in July 2014, the PHCPI collaborators gathered feedback from experts on the emerging conceptual framework.

10The PHCPI team then used this background research to develop a draft framework that was reviewed and discussed at the initial meeting, in July 2015, of the PHCPI's international expert advisory committee. Based on this discussion, the team revised the draft framework and used the first version as the context for performance reporting in the initial release of the PHCPI's website.4 During the following year, additional inputs from a range of experts and partners and the experience of the PHCPI team in working with the framework informed a set of proposed updates. In particular, initial PHC performance assessments with the teams in Cameroon and Mexico allowed us to explore the applicability of the theoretical framework to real‐world settings. We presented and discussed these tentative changes at the second advisory committee meeting, in November 2016, and incorporated them in the second version of the framework, currently in use and supporting measurement activities.

11The first set of PHCPI performance indicators, the “Vital Signs,” consists of internationally comparable indicators for diagnosing the level and trend of PHC system performance. The second set, the “Diagnostic Indicators,” is an expanded set of indicators to understand the underlying drivers of core performance. In developing these two indicator sets, our objective was to measure strengths and weaknesses across all dimensions of the conceptual framework and to measure the provision of key PHC functions through the lens of the patient, the provider, and the health system.

12Differing criteria guided the selection of Vital Signs and Diagnostic Indicators, which were chosen through coinciding processes of identifying candidate indicators, consulting experts, and selecting the final sets. Vital Signs were selected for their ability to assess how an overall PHC system is performing. In order to facilitate cross‐country comparisons and trend analysis, indicators included as Vital Signs had to be validated, reliable, and available from as many LMICs as possible. Diagnostic Indicators were chosen to provide insight into why performance and outcomes vary between countries and how these can be improved. They were designed to be used in concert with the Vital Signs as part of assessing performance at national, subnational, and facility levels. While validity and reliability were also priorities for Diagnostic Indicators, indicators with less internationally available data (eg, treatment accuracy, high blood pressure diagnosed and receiving treatment) were permitted in this set, as the goal of the Diagnostic Indicators was focused less on international comparability than on local applicability.

13Figure 1 summarizes the process for selecting the Vital Signs and Diagnostic Indicators. First, we identified through a literature review those indicators relevant to all domains of the conceptual framework. We searched specifically for indicators in the following relatively new areas of measurement, which reflect performance dimensions not traditionally included in health systems assessments in LMICs: first‐contact access, coordination, comprehensiveness, continuity, safety, and organization and management. We then cross‐referenced these indicators against existing indicators available globally, taken from the following international sources: Health facility surveys: Service Delivery Indicators (SDI), Service Availability and Readiness Assessment (SARA), Service Provision Assessment (SPA). Household surveys: Demographic and Health Survey (DHS), Multiple Indicator Cluster Survey (MICS), World Health Survey (WHS), Performance Monitoring and Accountability 2020 (PMA2020). Global databases: Global Health Observatory (GHO), UNICEF Data: Monitoring the Situation of Children and Women, World Bank World Development Indicators (WDI).National Health Accounts (NHA), System of Health Accounts 2011 (SHA 2011).

14Health facility surveys: Service Delivery Indicators (SDI), Service Availability and Readiness Assessment (SARA), Service Provision Assessment (SPA).

15Household surveys: Demographic and Health Survey (DHS), Multiple Indicator Cluster Survey (MICS), World Health Survey (WHS), Performance Monitoring and Accountability 2020 (PMA2020).

16Global databases: Global Health Observatory (GHO), UNICEF Data: Monitoring the Situation of Children and Women, World Bank World Development Indicators (WDI).

17National Health Accounts (NHA), System of Health Accounts 2011 (SHA 2011).

18We found more than 600 indicators, which we narrowed down to a priority list of approximately 100 indicators by subject‐based experts from the PHCPI's partner organizations, based on the criteria of importance and relevance, and mapped to the PHCPI's conceptual framework.4 For this set of potential indicators, we compiled definitions of numerators and denominators; exclusion criteria; information on validity and reliability; existing data collection mechanisms; and references to literature supporting the use of the indicator.

19After receiving feedback from 10 independent measurement experts, we chose the initial set of 25 Vital Signs from this list of indicators. We did not use a more formal expert review process for the initial selection of Vital Signs because these indicators were selected from established internationally valid and comparable indicators. In 2016, the Vital Signs were updated using a modified Delphi approach to review existing and proposed new indicators.

20Our selection of the Diagnostic Indicators was guided by a modified Delphi approach,13 consisting of an online survey and an in‐person meeting, both drawing on the input of the experts advisory committee. Experts (n = 33) were approached via email to determine their interest and willingness to participate in the survey. All those consenting were given a web link to the survey and provided with supplementary materials, including the PHCPI's conceptual framework, a glossary of key terms, and reference sheets for every proposed indicator listing the source, definition, rationale for inclusion, and limitations. The survey was open for 3 weeks, and the participants received up to 3 weekly reminder emails if they had not yet responded. The survey was completed by 23 experts (response rate of 70%).

21In the e‐Delphi survey, respondents were requested to complete 3 activities. First, they were asked to rate each indicator on a scale from 1 (not at all) to 5 (very strong) on the 5 dimensions of relevance, validity, actionability, reliability, and feasibility. Second, they were asked whether each indicator should be “included,” “included with adaptation,” or “excluded” from the Diagnostic Indicators. Third, we asked for recommendations for additional indicators. The results from the online survey were summarized for the in‐person meeting, during which experts were asked to provide additional feedback on the indicators. The aim of the modified Delphi approach was not to generate a final indicator list with complete group consensus but to gather feedback on the identified indicators to inform the PHCPI's measurement strategy and research and development agenda. The literature review, survey results, and expert input were then consolidated to generate a final set of 56 Diagnostic Indicators.

22In 2016, after approximately 1 year of use, including application of the Vital Signs for PHC assessment in Cameroon and Rwanda, we reviewed and expanded the 25 original Vital Signs to better cover measurement gaps in the performance dimensions of the revised conceptual framework. The PHCPI team selected an additional 15 indicators for possible inclusion as Vital Signs from a range of sources. We then compiled PHC‐relevant indicators from the indicators lists for Universal Health Coverage,14 the Sustainable Development Goals,15 the World Bank Global Financing Facility in support of Every Woman, Every Child,16 the WHO 100 Core Indicators17 from a WHO Eastern Mediterranean Regional Office working group on quality of care in PHC,18 and the World Bank Service Delivery Indicators.19 We selected those indicators meeting the selection criteria earlier and available for more than 10 countries as potential new Vital Signs. In addition, the definitions of 4 existing Vital Signs indicators were modified. The 15 proposed new indicators, along with the 25 original or slightly amended Vital Signs, were then scored and considered through a modified Delphi approach, again entailing an online survey followed by an in‐person meeting.

23The online survey was sent to 23 experts, of whom 17 provided complete responses (response rate of 74%). The survey methods and questions closely resembled the preceding Diagnostic Indicators survey, with minor changes in the phrasing of questions and response options. Respondents were requested to rate each indicator on a 7‐point Likert scale against 4 criteria—relevance and importance, reliability, validity, and actionability (Table 1)—and to indicate whether they believed an indicator “must be included” or “should not be included” as a Vital Sign. Experts were also invited to suggest additional indicators for consideration.

24We summarized the survey results and shared them with participants prior to the second expert advisory committee meeting, to inform the in‐person discussion. To obtain further input, in January 2017 we also shared the list of 40 proposed indicators with participants at a meeting of the Joint Learning Network PHC Measurement for Improvement Collaborative,20 an international community of leaders from LMIC health systems, focused on knowledge sharing and collaborative tool development for PHC performance improvement. Finally, we used correlation analysis to explore suspected collinearity between selected indicators that capture closely related information. After considering the survey results, input from the various discussions, and results of the correlation analysis, we proposed a revised set of 36 Vital Signs to the PHCPI's steering committee (composed of the principals from the 5 organizations), which they adopted. They also agreed that a process would be carried out to revise the Vital Signs indicators on an annual basis.

25From the literature review of the characteristics and determinants of high‐performing PHC systems, we identified several key elements of strong PHC. More than a package of services, quality PHC has 4 core functions: comprehensiveness of promotive, preventive, curative, and palliative care services; continuity across the life cycle; coordination across service providers and levels of the health care system; and a point of first contact access for the majority of patients’ health needs.21 These core functions, first articulated by Starfield, have been broadly accepted by and included in all PHC frameworks. The influential work on quality of care by Donabedian, which emphasizes interactions among patients, providers, and communities as fundamental to quality,22 is also reflected in many models of health system performance. Recent work in LMICs to assess the quality of primary care is also particularly informative; for example, a study in Brazil included an assessment of management, organization, and working conditions.23

26In existing frameworks, domains such as efficiency are often depicted as cross‐cutting attributes, as opposed to goals of a strong system.24, 25 Dimensions reflecting “hardware” inputs (funds, human resources, supplies, facilities, and information systems) are prevalent,8 and those incorporating “software” inputs (financing mechanisms, provider payment incentives, regulations, and market structures) are less prevalent. LMICs disproportionately measure hardware inputs, compared with HICs, providing an opportunity for a new focus on holistic system functioning in LMICs, rather than only on the availability of inputs. In addition to the overreliance on hardware inputs, very few frameworks emphasize integrating PHC with other health care settings. Moreover, most frameworks pay little attention to the perspective of the people involved in the system, that is, the providers, families, communities, and individual patients that interact with the PHC system.

27Of the existing frameworks for assessing PHC performance, the Primacy Care Assessment Tool (PCAT)26 and the Primary Health Care Activity Monitor for Europe (PHAMEU)27 are arguably the most closely aligned with the measurement approach advocated by the PHCPI, although they were developed for the context of higher‐income health systems. Previous applications of the PCAT tools in LMICs suggest that these are adequate but require some adaptation.28, 29 The PHCPI's framework retains the principal aspects of the PCAT and the PHAMEU (eg, first contact access, coordination, and comprehensiveness) but diverges in its unique applicability to low‐ and middle‐income settings. For example, the PHCPI indicators include the availability of basic equipment and access to treatment for infectious diseases that are not highly relevant to high‐income settings. Addressing the aforementioned gaps in existing approaches to performance assessment also drives WHO's emphasis on integrated people‐centered health services (IPCHS),30 the new guiding framework for service delivery reforms to expand access to health services and achieve UHC. While IPCHS strategies span the entire health system, IPCHS requires prioritization of high‐quality PHC as its cornerstone.

28Figure 2 shows the PHCPI's revised conceptual framework (the first version is provided in Appendix 3). It has 5 domains: System, Inputs, Service Delivery, Outputs, and Outcomes, and each has 5 or more subdomains and sub‐subdomains. This conceptual framework draws on several important earlier system frameworks, such as the logic model,31 Control Knobs Framework,32 Health Systems Performance Assessment,9 economic models of supply and demand, and predefined key characteristics of high‐performing PHC systems.21, 33, 34 The PHCPI's framework describes the key inputs, functionalities, and desired goals of an effective PHC system and focuses on the intersection between service delivery and the core functions of PHC (first contact access, continuity, coordination, comprehensiveness, and person‐centeredness) as the main drivers of performance variation. It also highlights people‐ and community‐centered care, supply and demand functions, system responsiveness, resilience, and integrated service delivery through effective organization and management.4

29The framework structure is similar to the commonly used input‐process‐output‐outcome logic model, indicating logical relationships between constructs. We placed a System‐Level Determinants domain before the Inputs domain to indicate the importance of the modifiable PHC system structure emphasized in the Control Knobs Framework.32 In addition, we more clearly defined process as the various critical subdomains of Service Delivery. The framework exhibits an overall directionality of influence, where the System domain influences the Inputs domain, which affects the complex interplay within the Service Delivery domain, contributing to effective Outputs, which subsequently affect Outcomes. This framework also incorporates the health system goals for the Outcomes domain—health status, responsiveness, equity, efficiency, and resilience—as articulated by numerous health systems performance assessment frameworks. Finally, in accordance with a biopsychosocial, as opposed to a strictly biomedical, model of disease causation,35 we acknowledge that PHC performance lies within a larger health system, which itself lies within wider political, cultural, demographic, and socioeconomic contexts.

30The revised version of the conceptual framework, finalized after the second experts advisory committee meeting, did not alter the 5 performance domains of System, Inputs, Service Delivery, Outputs, and Outcomes arranged in a logic model; however, the content of the System, Service Delivery, and Outcomes domains were altered. We made minor changes to the Outcomes domain to remove the components of “Reduced Mortality” and “Reduced Morbidity” from the subdomains of Health Status to better match the level of specificity provided in the other subdomains of Outcomes. Additionally, the name of the subdomain of “Health System Responsiveness” was changed to “Responsiveness to People” to better reflect the span of responsibility of PHC systems, and the System domain was revised to move the concept of “Community Engagement” from the subdomain of “Social Accountability and Community Engagement” to the Service Delivery domain. The concept of “Payment Systems” had been missing from the subdomain of “Health Financing” in the first version and was added.

31In the revised framework, we made the greatest number of changes in the Service Delivery domain. Because we recognized that the “Organization and Management” subdomain was 2 different types of management, population health management and facility organization and management, we separated them and clarified the lower‐level components of each subdomain. We reframed the “People‐Centered Care” subdomain as “High‐Quality Primary Health Care,” reflecting conceptual shifts toward thinking of “person‐centeredness” as a core function of high‐quality PHC of equal importance to the 4 functions first laid out by Starfield21 and others at WHO.34 Alterations to “Availability of Effective PHC Services” included moving “Safety” to this subdomain; adding “Patient‐Provider Respect and Trust” to explicitly recognize this concept; and changing “Provider Absence Rate” to “Provider Availability.” Finally, we reorganized the 5 subdomains in Service Delivery to better reflect the logical flow of service delivery, in keeping with the overall flow of the conceptual framework and recognizing interdependencies among the subdomains.

32The original set of 25 Vital Signs (provided in Appendix 4) covered all 5 domains of the conceptual framework, but many subdomains were still not reflected in this indicator set. The Vital Signs revision in 2016 resulted in 36 Vital Signs, 24 from the original 25 and 11 new indicators. Table 2 lists the 36 updated Vital Signs and the results of the experts’ evaluation of each indicator in the e‐Delphi survey. The average total rating reflects the mean of the experts’ summed scores (from 0 to 6) on each of the 4 criteria (relevance and importance, reliability, validity, and actionability). The number reflecting the qualitative recommendations was constructed by counting each response to definitely exclude an indicator as −1 and each response to definitely include an indicator as 1, and taking the mean across all responses. Providing this type of feedback was optional for each indicator, so values closer to 0 may indicate polarization in expert opinions or a lack of strong opinions in either direction. The last column in the table also identifies the number of LMICs with available data for each indicator; the numbers in parentheses reflect the number of countries expected to have data available soon as a result of data collection efforts currently under way.

33The modified Delphi approach we used in July 2015 resulted in the selection of 62 Diagnostic Indicators, which cover most of the original conceptual framework's subdomains; we did not find appropriate indicators for 7 subdomains. Table 3 gives the full list of Diagnostic Indicators and the average score for each indicator from the e‐Delphi survey. In the update to the Vital Signs, 1 Diagnostic Indicator was reclassified as a Vital Sign (cervical cancer screening rate); there currently are 56 Diagnostic Indicators. Many of the Diagnostics Indicators, especially those representing the Service Delivery domain, were validated in high‐income settings but were recognized as requiring further work in order to be adapted to LMICs. Similarly, the PHCPI team recognized that for some areas, existing indicators poorly reflect the attribute of interest (eg, provider caseload as a measure of provider motivation) and thus are areas where further research and development are needed.

34Our in‐person discussions about the Diagnostic Indicators validated the results of the scoping exercise and online survey. The participants debated the merits of some of the proposed indicators but had few recommendations for including additional indicators not identified in the initial review. They also discussed the appropriateness of many of the proposed Diagnostic Indicators for low‐income countries. Even though the relevant core domains of PHC were consistent across settings, there was consensus among participants that many of the existing indicators do not adequately capture these domains in low‐income settings (eg, percentage of births with a skilled birth attendant as a measure of effective service coverage). The experts also agreed that the Diagnostic Indicators should be captured and reported at the subnational level in order to understand variation in performance across countries and to support local improvement efforts.

35The results of the 2 modified Delphi processes confirmed that the indicators selected as Vital Signs and Diagnostic Indicators are strong existing measures that are validated, that are relevant to quality PHC, and for which many LMICs have data. The experts also agreed that these sets of indicators, though likely composed of the best available indicators, are not fully adequate for measuring the core components of quality PHC service delivery in LMICs and that substantial future work is needed to improve the indicators and increase data availability for these improved indicators. Currently, several subdomains of the conceptual framework (A1, A2a, A3a, A3c, B3, C1a‐c, C2a, C2c, D1g, E2, and E5) are not reflected in the Vital Signs and Diagnostic Indicators, owing to the limitations of validated and globally used measures. Furthermore, we need to consider issues salient to measuring quality PHC, such as governance, trust in the system, resilience, patient experience, communication and autonomy, pandemic preparedness, and mental health services. More scoping is needed to determine whether we have adequate indicators for these domains or whether further research and development of new indicators are needed.

36The expert meeting participants agreed as well that the measurement and data collection strategies for all the proposed indicators should focus on strengthening national and subnational data systems in order to enable local leadership, governance, ownership, and use of data in a sustainable manner. Experts also agreed that the goal of any new measurement and data collection effort should be to generate not more but better measures that reduce the use of irrelevant or not actionable indicators to improve performance.

37One priority area for ongoing methodological development is the creation of composite indices. The PHCPI initially developed 1 composite index as part of the original 25 Vital Signs, and we are now working to develop more comprehensive indices for each of 4 of the 5 domains of the conceptual framework: Inputs, Service Delivery, Outputs, and Outcomes. For the System domain, we will likely develop several composites instead of a single composite, because the diversity of subdomain topics (governance and leadership, health financing, adjustment to population health needs) makes aggregation into a single metric problematic.

38The original composite Vital Sign was the service coverage index, which combined data from 3 Vital Signs in the Outputs domain: antenatal care coverage (4 or more visits), percentage of children under age 5 with diarrhea receiving oral rehydration therapy and continued feeding, and DTP3 (diphtheria‐tetanus‐pertussis) vaccine coverage. We constructed this as the mean scaled residual of the 3 indicators and rescaled to range from 0 to 1; we made no imputation or adjustment for missing data.

39Our initial work on the new Outputs domain composite index seeks to incorporate additional primary health care services for a broader range of conditions relevant to a wider population, including family planning, pneumonia, tuberculosis, HIV, and malaria. The objective is also to closely align the indicators and methodology with the WHO index on the coverage of essential health services, which will be used to monitor progress toward UHC. Our current methodological challenges include choosing an approach to fill missing data (using the most recent year of available data for a country, the regional average, or data for a highly correlated indicator) and whether to construct the index as the mean of all services or as a compilation of subgroup means (eg, reproductive, maternal, newborn, and child health, infectious disease care).

40The PHCPI team is also currently developing national‐level performance dashboards of PHC systems. The objective is to produce a uniform set of visualizations for as many LMICs as possible that effectively communicate summary performance information about each country's PHC system. Our first step in designing the dashboard template was to review existing scorecards to generate ideas for data visualizations and layouts. Using ideas from this review and initial decisions about what information to convey, we developed a prototype through an iterative process.

41The prototype was initially developed for Uganda, which had the most complete Vital Signs data availability of any country. We created various layouts and graphics to present contextual information (eg, population, gross domestic product per capita), each of the Vital Signs, and selected data over time to convey trends. Through several iterations, we generated different variations of the dashboard, which we shared with the PHCPI team, experts, and partners for feedback, which was integrated into subsequent versions. We are currently amending the prototype to reflect recent updates to the Vital Signs and methodological developments in the production of the composite indices.

42The PHCPI's work to date has important limitations, primarily arising from the lack of available data for many performance indicators from most LMICs, in addition to the absence of validated and reliable indicators for measuring key aspects of PHC, even when data are available. While the PHCPI has articulated a comprehensive ideal of the components of PHC performance assessment in the form of the conceptual framework, substantial advancements in developing and validating indicators and collecting data are required to populate the full scope of this framework with indicators, mainly for the multiple subdomains that currently have no indicators. In other domains, the optimal means of collecting accurate data for relatively new indicators (eg, diagnostic accuracy, caseload per provider) are still being explored, leading to variability in the types of data available across countries. While challenging, these limitations are the primary factors informing and motivating planned activities to improve data availability and guide future research, which will address these gaps over time. There are additional limitations arising from the methods that the PHCPI has used to develop its conceptual framework and indicator sets. These tools reflect findings from the literature review and the opinions of the PHCPI team and experts committee. Although we have attempted to incorporate a diversity of sources and perspectives, additional or alternative input would likely lead to slight differences in the framework and indicators. The PHCPI welcomes continued feedback on and refinement of its tools and seeks to improve them over time.

43Even for the current set of Vital Signs and Diagnostic Indicators, data availability is highly variable: ranging from a low of 7 to 9 countries (indicators on provider competence, absenteeism, caseload) to a high of 100 to 130 countries (coverage indicators, mortality rates). The reason is that some aspects of performance are inherently difficult to measure (notably, measuring the effectiveness and quality of care is difficult in any setting), but also that many LMICs lack reliable data‐collection systems and tools.

44In discussion with the international experts’ advisory committee, the PHCPI team identified 4 major strategies for filling these information gaps: Influencing internationally comparable sources of information in global initiatives such as the Health Data Collaborative and other efforts to standardize health information systems and measurement approaches (eg, System of Health Accounts, SHA 2011)Working with and helping to strengthen countries’ routine health information systems to generate comparable performance informationIdentifying similar (acceptable substitute) indicators from routine information systems to feed into country‐level PHC dashboardsImputing data when information is missing

45Influencing internationally comparable sources of information in global initiatives such as the Health Data Collaborative and other efforts to standardize health information systems and measurement approaches (eg, System of Health Accounts, SHA 2011)

46Working with and helping to strengthen countries’ routine health information systems to generate comparable performance information

47Identifying similar (acceptable substitute) indicators from routine information systems to feed into country‐level PHC dashboards

48Imputing data when information is missing

49Other organizations and initiatives provide useful lessons for maximizing the benefits of a combination of these strategies. For example, the Organisation for Economic Co‐operation and Development (OECD) has worked extensively with member countries to develop comparable indicators, by working with national statistical institutes to adopt standardized measures. This process combines 2 approaches: bottom‐up when more than 20% of countries already have an indicator for a domain and top‐down when fewer than 20% of countries already have defined indicators. WHO's regional offices have also used a bottom‐up process, similar to the OECD's, to develop and validate indicators on quality of care with countries. This requires time and commitment to encourage countries to develop ways of measuring and to build capacity and systems to collect data.

50Some experts expressed concern that countries might see the PHCPI's efforts as measurement for judgment, rather than measurement for improvement. For this reason, a bottom‐up approach for developing new indicators that emphasizes the data a country needs to improve performance rather than the data a country needs to be globally comparable may be preferable for the PHCPI. Because the PHCPI works with countries, trust also is critical, particularly when considering ways to handle missing data and observing the limitations of using imputation methods in order to understand performance. Preserving the integrity and credibility of the indicators published and the conclusions based on indicators remains the initiative's top priority. The PHCPI is committed to involving countries in the production of indicators and composite indices before making any public release.

51As noted earlier, there was a general consensus within the expert advisory committee that key aspects of quality PHC currently lack validated, broadly used indicators and that existing measures for coordination of care, first‐contact access, continuity of care, and comprehensiveness of care must be refined so as to be more applicable to LMIC settings. We also need to increase the measurement of some validated indicators that are not currently widely collected (eg, provider competence). Developing new indicators and adapting existing indicators used in high‐income settings to fill these gaps are the priorities of the PHCPI's research and development agenda. Using facility and household surveys, research and development activities are currently under way in Ghana for indicators of organizational and managerial performance, as well as for experiential quality of care.

52Additional topics requiring research and development of new indicators include Patient trust in the system and providerProvider motivationTimeliness of health information systemsAvoidable hospitalizationsAvoidable mortality amenable to primary health care interventionsPalliative care coverageOut‐of‐pocket expenses and financial protectionUser‐reported access barriersOrganization and management, including referral systems, information systems, and performance incentive systemsMental health care coverageInformal health workforceGovernance at the facility levelCommunity engagement

53Patient trust in the system and provider

54Provider motivation

55Timeliness of health information systems

56Avoidable hospitalizations

57Avoidable mortality amenable to primary health care interventions

58Palliative care coverage

59Out‐of‐pocket expenses and financial protection

60User‐reported access barriers

61Organization and management, including referral systems, information systems, and performance incentive systems

62Mental health care coverage

63Informal health workforce

64Governance at the facility level

65Community engagement

66To develop new and improved indicators for aspects of quality PHC that have been neglected by existing measures, the PHCPI is partnering with academic researchers, survey programs, and other initiatives. WHO is currently selecting indicators for monitoring integrated people‐centered health services (IPCHS), which will cover several of the topics mentioned earlier, for example, patient experience, referral systems, and community engagement. The PHCPI and IPCHS indicators will be aligned when appropriate. The PHCPI is also a member of the Health Data Collaborative, which strives to improve the coordination of data collection activities across global health agencies and donors, such as through the creation of harmonized survey instruments for household and facility surveys. Through these partnerships, the PHCPI and others are developing and testing new indicators in household and facility survey tools with the goal of identifying improved indicators of PHC performance and integrating these indicators into the global measurement agenda.

67Designing and testing survey questions and then evaluating the validity and reliability of indicators generated from survey data, particularly for LMICs’ many health systems, takes a long time. As a result, the multiyear PHCPI research and development agenda must be harmonized with efforts to increase the availability of data for established PHC performance indicators and to enable health information systems to incorporate data collection for any additional measures.

68The applicability of the PHCPI's approach to assessing performance is most meaningfully tested and refined by engaging with countries working to improve their PHC systems. The PHCPI is engaging with these trailblazer countries to help them apply and strengthen performance measurement and improvement instruments by linking them with other initiatives in the PHCPI's partner institutions, including the Global Financing Facility (GFF) in support of Every Woman, Every Child; the Joint Learning Network (JLN); the World Bank Service Delivery Indicators; and countries’ results‐based financing initiatives. An example of country engagement was using the PHCPI's conceptual framework and Vital Signs to diagnose PHC system performance bottlenecks in Cameroon, as part of the development of the GFF investment case. This engagement resulted in a better understanding of service delivery bottlenecks and changes in the prioritization of interventions as proposed by the government of Cameroon and its partners. Through alignment with these and other ongoing activities in countries, PHC performance indicators can provide valuable insights for partners and simultaneously generate new data and feedback on tools for the PHCPI. The World Bank will seek to engage with a minimum of 14 countries through its investment operations from 2017 to 2019.

69The PHCPI has identified 5 key mechanisms for country engagement. First, in partnership with other members of the initiative, the PHCPI can support the development of new partner investments and engagements, including World Bank operations and GFF investment cases. Through its national PHC performance dashboards and other diagnostic instruments, countries can assess their PHC systems’ strengths and weaknesses in order to set policy and investment priorities. Second, the PHCPI is useful for supporting the implementation of new investments. The assessment activities help to identify implementation bottlenecks and improve capacity for strategically using performance information. Third, data gaps identified by the PHCPI can guide investments in information systems. Key indicators for which countries have no data or no recent data highlight areas where new or improved measurement is most needed. Fourth, through its global network, the PHCPI facilitates knowledge sharing and mutual learning on innovative service delivery models and the use of performance measurement for improvement. The PHCPI is already supporting these activities through the JLN PHC Measurement for Improvement Collaborative and establishing a community of practice in the World Bank. Finally, through engagement over time, the PHCPI will develop expertise to guide innovative financing approaches for strengthening PHC systems. As the PHCPI's work in countries expands, the range of experiences can inspire and inform new ways of financing and incentivizing PHC performance improvement.

70Over the past 2 years, the PHCPI has been developing a suite of performance assessment tools to provide in‐depth and comprehensive understanding of PHC systems in low‐ and middle‐income countries. The foundation of the PHCPI's approach to performance assessment is the conceptual framework, which is unique among health system assessment frameworks for its emphasis on service delivery and quality of care. Two sets of performance indicators, selected with substantial input from a diverse and global group of experts, provide the backbone for measuring system performance in alignment with the conceptual framework. Additional instruments are still being developed, such as country‐level performance dashboards, a national and subnational performance assessment tool, and performance improvement pathways. These will improve countries’ abilities to design and implement an assessment of PHC system performance; communicate national and international findings with diverse audiences; evaluate changes in performance over time; and identify and select appropriate policy reforms for performance improvement.

71The PHCPI's main challenges are the lack of available data for many indicators, particularly for measuring the PHC's core functions, and a lack of validated indicators for other critical dimensions of quality PHC. The PHCPI is working with partner organizations and other initiatives to develop and test new indicators and improve data collection systems to generate better PHC performance information.

72Through substantial country engagement and continual expert oversight, the PHCPI will continue to improve each of its performance assessment tools; develop new and improved indicators; and increase the availability of data, particularly for aspects of PHC that are not well understood in LMICs. Each of these activities catalyzes the dissemination of new knowledge and best practices for measuring and improving PHC system performance, supporting the achievement of UHC, and improving quality of care globally.