Below, we first summarize the results of each round of the modified Delphi process and how it supported MODIFI development. Then, we present the outcome of this development process—the final MODIFI method (see Additional File 1).
Round 1
The Round 1 survey focused on inclusion and acceptability. Most participants (61.4%) answered that the three MODIFI phases were acceptable for an adaptation how-to guide, while 38.6% answered that these overarching phases were unacceptable and provided written responses describing what they would change. For a full summary of the original MODIFI components and revision decisions based on Round 1 results, see Additional File 2.
For the specific MODIFI steps within each phase, participants rated inclusion favorably, with steps rated as “Essential” by 83.96% of participants on average, depending on the step (SD = 0.16). One step was rated as “Essential” by less than 50% of participants, and it was removed. In addition to inclusion, participants rated the acceptability of each specific MODIFI step. Steps were rated as “Acceptable” by 51.28% of participants on average, depending on the step (SD = 0.12). Given that almost half of participants considered most steps to be unacceptable as currently written, all steps were revised.
Acceptability ratings were further interpreted based on participants’ qualitative feedback. A summary of the Round 1 qualitative feedback can be viewed in Additional File 3. MODIFI was heavily revised based on this feedback, particularly the steps with lower ratings of acceptability. Following Round 1, MODIFI was streamlined, offering a narrower range of techniques. The revised version of MODIFI was designed to emphasize the creation of internally valid, locally actionable knowledge (which tends to be a more common focus in industry than in traditional academic research) and improve MODIFI’s feasibility for use in applied settings by intervention implementers (e.g., clinicians) and/or intervention decision-makers (e.g., supervisors, program/site leaders).
Round 2
For a summary of the revised MODIFI components and final revision decisions based on Round 2 results, see Additional File 4. The revised version of MODIFI participants rated in Round 2 was bookended by an introduction (MODIFI overview, definitions [see Fig. 2], prerequisites) and guidance for what users should do after completing the MODIFI process. Revised MODIFI comprised three steps: learn about the users, learn about the local context, and identify key information (Step 1); adapt the intervention (Step 2); and evaluate the adaptation (Step 3). After Step 3, MODIFI users were encouraged to return to earlier steps if the adaptation required revision to achieve the desired outcome.
Participants commented on a MODIFI phase or step if they believed it required additional revision. Higher rates of comments indicated where to apply additional revisions. Round 2 participants disproportionately commented on the introduction overview (59%) and definitions (66%), and two Step 1 components: learn about the users (53%) and identify key information about the intervention (78%). Additionally, participants rated each step’s feasibility of use in applied contexts, with all revised MODIFI components considered somewhat or completely feasible.
Similar to Round 1, Round 2 qualitative feedback was used to inform the final MODIFI revisions. A summary of the Round 2 qualitative feedback can be viewed in Additional File 5. Following Round 2, there were fewer recommended revisions, and these revisions were relatively minor in scope. Round 2 changes largely surrounded operationalization of terms/processes and the sequencing of content. In response to the Round 2 feedback, the final MODIFI presents a simplified figure illustrating its overall process (Fig. 3); front loads content regarding participatory co-design methods; includes newly clarified definitions of several key concepts (including an example of an intervention function/form table); instructs participants in how to consider and respond to potential unintended consequences of adaptation; and emphasizes iterative evaluation and development both within MODIFI and following its final steps. Additional small-scale changes were made in response to minor feedback.
The Final MODIFI Method
Round two feedback gave rise to the final version of MODIFI. The steps of MODIFI (see Fig. 3) are summarized below and presented in full in Additional File 1. MODIFI is used to make adaptations to part of an intervention—not to complete whole-intervention redesign. If the person/people applying MODIFI want to make multiple adaptations, it is recommended to complete the MODIFI steps for each adaptation (either one after another or at the same time in separate processes). MODIFI’s techniques can be carried out in many ways—to match the needs and resources of local settings, there are no prescribed numbers of participants, numbers of data collections, or time periods for data collection. MODIFI is a method that provides both structure and flexibility, because it is designed to be useful for a range of people and settings. For an illustrative case example describing what it looks like for people working in an applied setting to apply the MODIFI method for intervention adaptation, see Additional File 6.
Unlike broader implementation frameworks, MODIFI is designed to “zoom in” on intervention modification and outline clear steps for how to design adaptations. Because MODIFI is a method to be used at a particular point in the implementation lifecycle, its application carries several preconditions. First, MODIFI users should already have selected an intervention to implement that addresses a problem for the population of focus, that relevant stakeholders believe is (or has the potential to be) appropriate, and ideally that has evidence for its effectiveness. Second, users have determined that adaptation is necessary (Baumann et al., 2015; Miller et al., 2020)—the original intervention cannot be implemented successfully due to potential problems with MODIFI outcomes (e.g., fit/appropriateness, usability, cultural responsiveness; see Figs. 3 and 4) and/or implementation outcomes (e.g., low fidelity, high cost). These outcomes are prioritized because they represent a causal chain. The goal of intervention adaptation is to improve the fit such that EBPs are more feasible, usable, culturally responsive, etc., and thus are more likely to be implemented with quality. Third, the user has formed a team to support intervention adaptation (G. Moore et al., 2021). Adaptation teams work best when they contain a mix of people in different roles (Aarons et al., 2012); specifically, primary users, people with expertise in the intervention (or the topic/problem it addresses), and people with expertise in intervention adaptation methods. When this is infeasible, MODIFI may be used cautiously by an individual intervention implementer or decision-maker.
MODIFI Step 1: Learn About the Users
Step 1 of MODIFI has three components that can be completed in any order or simultaneously—learn about the users, learn about the local context, and identify key information about the intervention. Step 1 involves taking an applied anthropological approach (Hamilton & Finley, 2019)—listening, observing, and understanding experiences of the intended users within the local context—and additionally identifying the intervention’s functions and forms so the intervention can be adapted while maintaining effectiveness (Kirk et al., 2021). Learning about the users requires the person/people applying MODIFI to identify who will be the “users” of the adapted intervention (e.g., providers, service recipients; Lyon et al., 2019). At times, the person/people applying MODIFI are primary or secondary users themselves. Multiple users can be considered simultaneously by integrating information about their needs. After users have been identified, the person/people applying MODIFI conduct interviews asking users about their needs and assets related to the intervention and the topic/problem it addresses (Dopp et al., 2019b). Users might rank their unmet needs in order of priority to guide decision-making when designing adaptations. When needs conflict within or across user groups, needs should be prioritized in order of proximity to the intervention (e.g., primary users before secondary users). If needed, professional experience and research literature can be used to elaborate upon what is learned from users. After completing this component of Step 1, the person/people applying MODIFI should have a list of the highest priority unmet user needs to inform intervention adaptations in Step 2.
MODIFI Step 1: Learn About the Local Context
Another component of Step 1 is to learn about the local context. First, the person/people applying MODIFI identify which aspects of the context (e.g., workflow, routines) are most likely to impact the intervention’s implementation. Then, in the context where the adapted intervention will be implemented, observations are conducted to gather information about the identified factors (Hamilton & Finley, 2019). What gets observed depends on the reasons that adaptation is needed to improve intervention-context fit, so the person/people applying MODIFI identify which aspects of the context they need to learn about in order to address the “WHY” of adaptation (J. E. Moore et al., 2021). Observations can be conducted efficiently by selecting aspects of the context that are practical to observe (e.g., physical location, working hours). The goal is to remain unobtrusive but not necessarily trying to be invisible—observers can be friendly, ask questions, reassure people that they are there to learn (not judge), and respect confidentiality (Dale et al., 2013). After completing this component of Step 1, the person/people applying MODIFI should have a list of aspects of the local context that may interfere with intervention implementation.
MODIFI Step 1: Identify Key Information About the Intervention
Another component of Step 1 is to identify key information about the intervention. To adapt an intervention while retaining/maximizing its effectiveness, the person/people applying MODIFI must understand how it works. This can be accomplished via a function/form table that maps out how the intervention achieves its effects and is used to identify what can and cannot be changed during adaptation. Often core functions are not articulated by intervention developers but can be identified by creating a table with three columns: 1. Problems, 2. Functions, and 3. Forms (Kirk et al., 2021; Perez Jolles et al., 2019). Using the intervention materials (e.g., manual, website), consultation with intervention developers and/or experts, professional experience, and the research literature, the columns are populated with information including the problems the intervention aims to solve, the intervention’s functions—the goals or ways the intervention solves each problem—and the form(s) that each function takes within the intervention (e.g., intervention activities; see Table 1). With the function/form table completed, the person/people applying MODIFI will have a list of intervention functions (how the intervention solves problems)—these should remain unchanged in Step 2, and intervention forms—these may be adapted in Step 2. In the adapted intervention, each function is represented in at least one form.
Table 1
Example Function/Form Table for TF-CBT.
Problems | Functions | Forms |
● Posttraumatic symptoms ○ Dysfunctional trauma-related cognitions (i.e., negative beliefs about self, others, the world, & the future) ○ Reexperiencing the trauma (e.g., intrusive memories, dreams) ○ Heightened arousal & reactivity ○ Affective dysregulation ○ Avoidance of trauma reminders | ● Change trauma-related cognitions ● Improve emotion regulation ● Decrease behavioral avoidance | ● Cognitive coping skills, trauma narration & processing, psychoeducation ● Affective modulation skills, relaxation skills, cognitive coping skills ● In vivo mastery of trauma reminders, cognitive coping skills |
● Depression symptoms ○ Feeling sad, empty, hopeless, worthless ○ Decreased interest in activities ○ Suicidal ideation | ● Change trauma-related cognitions | ● Cognitive coping skills, trauma narration & processing, psychoeducation |
MODIFI Step 2
Step 2 of MODIFI uses a co-design method to adapt the intervention’s forms while leaving the functions intact. Co-design involves partnership between members of different groups to explore challenging problems and identify solutions (Metz et al., 2019). First, the person/people applying MODIFI identify who will participate in the co-design sessions. If possible, they should include at least one person with each of these roles: primary user, expert in the intervention (or the topic/problem it addresses), and expert in intervention adaptation methods. When this is not feasible, they should consider which viewpoints may be absent from the team and do their best to elevate those viewpoints as they present the information gathered during Step 1. When the group is assembled, they engage in co-design sessions (in-person or online), where they collaborate to: 1. understand the problem(s) to be solved through adaptation, 2. generate possible solutions, 3. co-create adaptations that solve the identified problem(s), 4. consider possible unintended consequences, and 5. iterate until the adaptation is ready for evaluation. In MODIFI each of these co-design aims is accompanied by a list of techniques. First, to understand the problem(s) to be solved, the person/people applying MODIFI present the information gathered in Step 1 (e.g., highest priority unmet user needs, aspects of the local context that may interfere with intervention implementation, and intervention functions), and the user/stakeholder members of the co-design sessions present information about their experiences. Then, to generate possible solutions, all co-design members contribute to brainstorming solutions to the problem(s) that they hope to solve through adaptation (IDEO, 2015). After brainstorming possible solutions, co-design members decide collaboratively which solutions (i.e., intervention adaptations) to select for co-creation. To create adaptations that solve the identified problem(s), co-design members draft intervention adaptation(s), during which they make sure that each intervention function is represented in at least one form (referring to the function/form table from Step 1). Then, co-design members iterate—co-creating further adaptation drafts, building upon each version by asking themselves, “How could this solution be just a little bit better?” (IDEO, 2015).
Next, co-design members reflect on potential impacts of the drafted adaptation(s) by discussing these questions (Kirk et al., 2020): Is this adaptation designed with specific goals in mind? Is this adaptation aligned with intervention core functions? And could there be unintended negative impacts of this adaptation (e.g., on adoption, cultural responsiveness, feasibility, cost)? Then, co-design members discuss possible negative impacts (e.g., increasing an EBP’s acceptability may reduce its effectiveness if the adaptation alters the EBP’s core functions; Kirk et al., 2020), the likelihood of negative impacts, and their severity, then consider whether these can be offset by positive impacts on other outcomes. Based on the findings of the impact analysis, further iteration may be warranted. In that case, the team co-creates further adaptation drafts. Finally, co-design members reach consensus by agreeing that the problem(s) have been solved well enough that the adaptation is ready for evaluation. After completing Step 2, the person/people applying MODIFI will know what adaptation is needed to match user needs/assets and local context realities.
MODIFI Step 3
The goal of Step 3 is to generate evidence that is relevant to the identified users within the local context, not to collect evidence that is generalizable to other users and contexts (Daleiden & Chorpita, 2005). Thus, efficient, feasible, and locally appropriate evaluation methods are recommended. The person/people applying MODIFI should begin this step by thinking about what data they need to understand whether the intervention adaptation works for the identified users within the local context. If possible, they should include both quantitative and qualitative indicators of success (e.g., ratings of acceptability, quotes about cultural responsiveness, implementation outcomes). Ultimately, they should make decisions about what data they collect based on what’s feasible in their context, alongside what they learned from the users/context in Step 1 about the most important outcomes to maximize during intervention adaptation. Next, they should decide how they will measure the outcomes they have chosen, how often they will collect data, and what they need to see in order to conclude that the adaptation works for the identified users within the local context. These decisions are based on what they can actually track in their context. Data collection may be as narrowly scoped as a provider asking a service recipient if the adaptation is acceptable during each session while the adaptation is implemented or as complex as collecting data on multiple outcomes with multiple users over time before and after the adaptation is implemented.
MODIFI Next Steps
Following the three steps of MODIFI, the person/people applying MODIFI should know whether the adapted intervention works for the identified users within the local context in a way that they find relevant and satisfying. If so, then they can implement the adapted intervention, and if resources allow, implement while collecting additional data (e.g., on the outcomes they have chosen, and/or on additional changes that occur during implementation). If the adapted intervention does not work for the identified users within the local context (or a subset of the identified users), the person/people applying MODIFI should either return to Step 1 if they need to learn more about the users, context, and/or intervention before further adaptation, or return to Step 2 if they know what further adaptation is needed, then continue from there.