An Extension of RE-AIM to Enhance Sustainability: Addressing Dynamic Context and Promoting Health Equity Over Time

1RE-AIM is a robust framework that has been widely applied over the past 20 years across a range of public health, clinical, community, and behavioral settings. RE-AIM was created to help address the well-documented research-to-practice gap that hinders the reduction of health inequities and widespread population health impact. It is one of the most commonly applied frameworks in public health, health behavior, and implementation science (IS). RE-AIM can facilitate transparent reporting and enhance planning for successful dissemination and implementation of evidence-based interventions, programs, practices, and policies (“EBIs”). Recent years have seen expansion of RE-AIM to address contextual factors (e.g., RE-AIM/PRISM); and integrate qualitative methods.

2As a framework, RE-AIM has both individual-level and staff/setting-level dimensions, including Reach and Effectiveness (individual-level), Adoption and Implementation (staff and setting levels), and Maintenance (both individual and staff/setting levels). Recognizing sustained delivery and impact of EBIs as central challenges across settings, RE-AIM has historically been one of the few IS frameworks that explicitly built in measurement and consideration of “maintenance.” “Maintenance” in RE-AIM has been operationalized at the individual level (e.g., long-term effectiveness or impact of EBI) and the setting level (e.g., sustainability of EBI program components after original implementation). The “maintenance” dimension of RE-AIM has typically been assessed at relatively short-term intervals (e.g., 6 months after EBI delivered or initially implemented) and its evaluation has focused on the extent to which a program/policy becomes institutionalized (e.g., made part of routine organizational practices and policies).

3Within IS, there is growing recognition of the importance of understanding and addressing longer-term sustainability of EBIs. Achieving sustained impact and delivery of EBIs over time has been identified as one of the most important yet understudied challenges across settings, populations, and health issues. There is growing consensus on conceptualizations and definitions of sustainability; e.g., Moore et al. described sustainability as “after a defined period of time, the program, clinical intervention, and/or implementation strategies continue to be delivered and/or individual behavior change (i.e., clinician, patient) is maintained; the program and individual behavior change may evolve or adapt while continuing to produce benefits for individuals/systems.” Of note, we recognize that the terms sustainment and sustainability are both used in reference to the outcome of an intervention being delivered over time, as well as the characteristics of the intervention that make it more likely to be delivered over time. For this paper, we used the term “sustainability” to refer to both the desired outcome and the characteristics or processes by which it is more likely maintained.

4There has been an important shift away from “static” conceptualizations of sustainability, with awareness that this may impede adoption of more effective practices as the environment changes or new evidence emerges. Investigators also increasingly recognize the need for a dynamic conceptualization of sustainability, in light of complex “real-world” contexts in which EBIs are delivered that require responsiveness, capacity building, and adaptation of EBIs. This is consistent with the Dynamic Sustainability Framework (DSF), which focuses on continued learning and evaluation, problem-solving, improvement and ongoing adaptation of EBIs to enhance fit with contexts and populations. Just as a balance between fidelity and adaptation is needed to achieve “fit” in the context of pre-implementation and implementation efforts, there is a similar balance between sustainability of original EBIs and ongoing “evolvability” to achieve ongoing fit and sustained population health impact within broader communities or health systems. Evolvability relates to the adaptation of EBIs and implementation strategies in response to changing contexts and resources over time, as well as emerging needs and evidence across the life cycle of an EBI. This includes both the systematic, planned adaptation of EBIs and strategies, as well as ongoing refinement of EBIs and strategies organically within specific community or clinical settings. Over an EBI's life cycle, this evolution within a changing system or organization may ultimately involve “de-implementation,” or the removal or replacement of EBIs that no longer fit or are ineffective.

5As explicated below, equity and costs are foundational driving forces across RE-AIM dimensions that shape sustained impact, and warrant the need for initial and ongoing adaptation. EBIs can only succeed at the population health level if they are affordable across most settings and are delivered routinely and equitably over time across diverse settings and populations. As we consider the life cycle of an intervention, it may be less useful to think about “sustainability” of the original EBIs as an “end goal”, and instead consider “evolvability” across the dynamic life cycle of the EBI within a broader context or system, with the goal of sustainable and equitable health impact.

6Important gaps persist in existing frameworks' ability to provide guidance in concretely conceptualizing, measuring/operationalizing, and planning for longer-term sustainability within a dynamic context. For example, RE-AIM does not capture such dynamic conceptualizations of sustainability, and has often been applied as a “one-time” evaluation and planning tool. Given the numerous conceptual frameworks and models in IS, we did not seek to create a new framework. Instead, we propose an expansion of RE-AIM to enhance sustainability by focusing on key issues across RE-AIM dimensions, with the goal of increasing health impact and health equity over time.

7The purposes of this article are to: (1) discuss the extension of RE-AIM to address dynamic conceptualization of sustainability over time, including iterative application of RE-AIM to guide adaptation and evolvability of EBIs and implementation strategies; (2) provide concrete guidance on issues pertinent to understanding, measuring, and planning for sustainability in changing context, including explicit consideration of costs and equity; and (3) propose testable hypotheses and detailed research questions to guide future research that applies RE-AIM for sustainability.

8The following sections discuss and provide recommendations to guide planning, adaptation, and measurement when applying RE-AIM to facilitate sustainability, reflecting dynamic sustainability with a focus on context and equity. Each section concludes with example hypotheses to guide research. Five key issues are discussed below, and summarized in Table 1 and Figure 1.

9Given growing consensus of sustainability as dynamic in nature, it is important that indicators of sustainability reflect this longer-term conceptualization. While 6 months, as originally proposed in RE-AIM, is useful in providing an indicator of early maintenance, a more comprehensive approach to also capturing sustainability over time includes measurement at least 1 year post initial implementation and over time (e.g., quarterly to annually).

10Consistent with recent conceptualizations, we recommend that operational indicators of maintenance/sustainability include (see Table 1 for details): (1) extent to which the core components/functions of EBIs and implementation strategies continue to be delivered over time with fidelity (e.g., continuation of active ingredients and essential functions/related activities), and the “evolvability” of the EBI and implementation strategies needed to support continued EBI delivery over time, including adaptations (planned and organic) and why they occur; (2) extent to which the EBI has continued impact on health behaviors/outcomes, when feasible, including patterns in health inequities over time (e.g., who continues to experience health benefits and who does not); and (3) extent to which community and organizational capacity and infrastructure to deliver the EBI are maintained, including partnerships, networks, and coalitions. It is critical to actively engage with stakeholders (e.g., community members, implementers, organizational leaders) to prioritize which maintenance/sustainability outcomes will be measured and when (e.g., which are meaningful and pragmatic to assess and how often). We also recognize the challenges of including ongoing measurement of the effectiveness of EBIs; while in some cases, existing resources may provide data to monitor frequent and continued impact on health behaviors/outcomes, we realize this may not feasible across all settings.

11Example Hypothesis: Informed by a broadened, longer-term conceptualization of sustainability, the dose and nature of implementation strategies needed to initially implement an EBI will differ from the strategies needed to sustain an EBI over time (e.g. implementation strategies focused on sustainability may relate to providing proactive planning and ongoing evaluation/monitoring to manage likely changes in the implementation setting, including turnover, EHR upgrades, treatment guideline updates, changes in patient population).

12In many cases it is neither feasible nor optimal to continue to deliver the same EBI “protocol” with high fidelity, as context changes over time and across settings. There is often a need in early program stages to make planned “fidelity-consistent” adaptations that reflect diverse settings, cultures, and populations in which they are delivered. Failing to make planned cultural or contextual adaptations may have adverse impact on effectiveness, and ultimately, perpetuate health inequities. EBIs and implementation strategies that are not aligned with and do not reflect changing community needs, culture, and context are unlikely to be sustained or have sustained impact over time.

13It is also likely that there will be evolving evidence (e.g., guidelines change, new populations are exposed to EBI with varied results), setting changes (e.g., staff turnover/attrition; resources change), and shifting population health needs over time that require ongoing adaptations or refinements over time. We recommend proactively planning for adaptations, and documenting why adaptations are needed, and the extent to which EBIs and implementation strategies evolve over the life cycle of a program. Iterative application and measurement of RE-AIM dimensions over time enables documentation of effective adaptations to retain fit with ever evolving context. De-implementation (e.g., the removal or replacement of low-value, harmful, costly or non-evidence-based care/EBIs, including the need to make a program and its delivery less expensive), may also be necessary and should be tracked to inform changes in implementation.

14Example Hypothesis: Settings that maintain core functions of EBIs but include proactive, planned, iterative adaptations to intervention components and implementation strategies in response to changing context and needs will be sustained longer than those that do not, and will have greater impact on reducing health inequities.

15Assessment of RE-AIM dimensions can help guide settings on how to proactively monitor or adapt and may identify early indicators of sustainability challenges, including the need to “change course” to promote the sustainability of EBIs over time. Results on RE-AIM dimensions should not be assumed to be static. Thus, as explicated in Table 1, RE-AIM indicators (e.g., Reach, Effectiveness) should be measured repeatedly and iteratively when possible to provide insight into how to achieve sustained health impacts, monitor progress, and shed light on where and when both equity and sustainability issues arise (e.g., over time, which populations and settings is the intervention reaching, and why or why not?).

16These findings may impact the nature and timing of actionable solutions across RE-AIM dimensions and program life cycle —e.g., adapt the recruitment or implementation strategies. RE-AIM qualitative probes can also be used to (a) help ensure that the perspectives of key stakeholders and community members are being assessed regularly; and that (b) stakeholders are being actively engaged in planning for sustainability in ways consistent with their values (e.g., “What would it take for you/your organization/your community to sustain the EBI over the long term?”). Aligned with existing taxonomies of implementation strategies (e.g., evaluation/iterative strategies), RE-AIM can be used as a tool to complement existing quality improvement (QI) and performance management resources (e.g., PDSA cycles). As such, iterative application of RE-AIM can provide guidance and a conceptually-based, standardized evaluation approach to understand what is working or not; this information can be used to inform QI activities (e.g., who participates and why; where in the system is implementation of highest/lowest quality), with implications for long-term sustainability and impact.

17Example Hypothesis: Programs that iteratively assess and address RE-AIM dimensions over time to guide their sustainability planning and adaptations will have stronger sustainability outcomes (e.g. higher levels of continued delivery of EBI; higher levels of sustained behavior change across population groups) than those that do not.

18While several frameworks provide consideration of multi-level contextual factors that influence sustainability, many have been most explicitly applied in the context of implementation e.g., PRISM; and Consolidated Framework for Implementation Research [CFIR;]. There may be value in also considering frameworks that have focused specifically on sustainability, including the Program Sustainability Assessment Tool [PSAT;] and the Integrated Sustainability Framework [ISF;], which provide a strong foundation for understanding multi-level contextual determinants of sustainability, but less guidance in measuring sustainability outcomes, or thinking explicitly about dynamic sustainability. These multi-level determinant frameworks may call attention to constructs that are particularly important to sustainability (e.g., sustainability planning, funding stability, staff retention over time), and can be integrated with RE-AIM to inform questions, measurements and actions related to contextual determinants of sustainability. For example, the ISF could be used to understand and assess multi-level aspects that may influence sustainability- e.g., “How have program champions played a role in sustaining the EBI?” Data from such qualitative assessments would preferably be integrated with quantitative measures of sustainability determinants (e.g., informed by the ISF or PSAT). It is important to recognize that sustainability determinants themselves are likely not static, and may change over time.

19Example Hypotheses: 1) Programs that explicitly address multi-level contextual determinants of sustainability will produce higher levels of sustainability and equity than those that do not; 2) Programs that address changing multi-level context and determinants of sustainability will be sustained longer than those addressing only one level.

20Promoting health equity1 is a central part of our conceptualization and measurement of sustainability, and RE-AIM indicators should be tracked over time to identify and address inequities when they arise (further explicated in Table 1). All RE-AIM dimensions include representativeness (heterogeneity, generalization), which should be assessed across different types of patient/population subgroups of focus (e.g., by race/ethnicity, age, disability, insurance status, literacy level, social determinants of health), and settings (e.g., urban/rural, lower vs. higher resource settings). Consistent with notions of “equitable implementation”, it is critical to document and address inequities as they emerge across all RE-AIM dimensions. Not doing so risks maintaining or even exacerbating health inequities, and ultimately inequitable use of EBIs over time.

21Issues of cost and resources required are strongly tied to health equity. For example, if an EBI is not feasible for delivery in certain settings (e.g., community health centers) due to constrained resources or insufficient staff, inequities may result. This is because these settings often reach populations that experience disproportionate social stressors and greater structural barriers to care. At the individual level, if participation requires considerable costs or burden such as travel or time off work, unintentional health inequities may result. To prevent such consequences, initial cost estimates and resource requirements should be discussed with stakeholders at the planning stage, and costs can be periodically assessed, discussed and necessary adaptations made over time.

22We consider “costs” very broadly, including understanding, planning for, and tracking economic costs, time, resources, burdens, and unintended political and social consequences (e.g., social stigma) of an EBI, especially from the perspectives of different stakeholders (e.g., implementers, administrators, community members, and patients). Recent IS research provides suggestions for cost assessment to understand the impact on sustainability. We also encourage consideration of economic factors more broadly, including the “value” and return on investment of sustaining the EBI, and the priorities of, and value to, different stakeholders, including community partners.

23Example Hypotheses: 1) Programs that explicitly and repeatedly assess health equity and equitable implementation, and make iterative adjustments guided by RE-AIM will produce higher levels of sustainability than those only considering equity at the planning stage. 2) Programs that consider and monitor costs (and RE-AIM outcomes), ‘return on investment' over time, and discuss and act on these assessments in partnership with stakeholders will produce stronger sustainable outcomes than those that do not.

24The discussion above illustrates key issues involved in extending RE-AIM to enhance sustainability. In Table 1, we outline key indicators, guiding questions, and equity and sustainability considerations in applying this extension and iterative application of RE-AIM. Consistent with complex adaptive systems, it is more complex than the discussion makes it appear, as the various factors above and the RE-AIM dimensions are interrelated. Thus, we need to consider interactions among the issues above and across RE-AIM dimensions over time. Figure 1 highlights this complexity and considerations for cross-cutting, intersecting issues and indicators that shift (like gears) over time in different combinations to guide RE-AIM for sustainability in dynamic context across the life cycle of an EBI. This summary figure illustrates the impact of EBIs and implementation strategies on the RE-AIM dimensions, and how factors such as health equity and costs influence the likelihood of sustainability across the phases of a program.

25This paper encourages iterative application of RE-AIM with early guidance on understanding, evaluating, and planning for sustainability, with a focus on changing context and health equity. While RE-AIM has previously been applied to promote health equity, this paper reinforces the importance of this focus within the context of sustainability. It advances the IS field beyond existing models and prior RE-AIM publications by providing: (1) consideration of planning for sustainability throughout the life cycle of an EBI and across multiple RE-AIM dimensions; (2) concrete guidance for operationalizing the dynamic and complex nature of sustainability, including the “evolvability” of an EBI and where adaptations and de-implementation may fit within this conceptualization; (3) attention to iterative measurement of RE-AIM indicators to inform and enhance sustainability, and (4) explicit consideration of health equity and costs/value as critical components of sustainability. In summary:

26Measuring “maintenance” as a RE-AIM dimension is important, but needs to be expanded to address longer-term conceptualizations of sustainability. The conceptualization of dynamic sustainability includes consideration of “evolvability” across the life cycle of an EBI, including continued delivery of the original EBI functions and implementation strategies, adaptations, and potential de-implementation across the EBI life cycle to produce sustained and equitable health outcomes.

27Multi-level context changes and so must EBIs and implementation strategies to meet emerging needs, resources and challenges over time. Iterative (or at least periodic) use of actionable RE-AIM assessments can guide adaptations to enhance sustainability and respond to changing context.

28Equity (both equitable implementation across RE-AIM dimensions and health equity) and costs/value are important and understudied cross-cutting issues across all RE-AIM dimensions that impact sustainability. Researchers should assess and address these factors in planning for and facilitating long-term sustainability.

29This article has both strengths and limitations. Strengths include its focus on costs and value, from the perspective of multiple stakeholders, and health equity and representativeness across all RE-AIM dimensions as key drivers of sustainability. Additionally, instead of proposing another IS model, we provide an extension of, and guidance from, a widely adopted IS framework. This paper and our recommendations address sustainability processes and planning, as well as sustainability outcomes. Finally, we make recommendations and testable hypotheses that should lead to incremental validation, revision or rejection as we refine this extension of RE-AIM. Limitations include that this proposed expansion of RE-AIM needs further empirical support. We call for future application across diverse health issues and settings, and mixed-methods research to investigate and refine this extension of RE-AIM for sustainability. There is still much to learn about sustainability, and we believe this application will provide a useful guide and addition to the IS literature.

30RS and RG initially conceptualized the paper. RS took the lead in writing an initial draft. All authors contributed to reviewing, revising, and rewriting sections for this Perspective piece.

31The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.