Introduction

Due to its immersive characteristics, virtual reality (VR) has much promise for mental healthcare because it can bring the outside world into the treatment room. Because of the elicited sense of presence in users—resulting in the experience of real emotions—VR provides a plethora of options for treatment, such as exposure therapy or skill training (Geraets et al., 2021). While there is a need for more evaluation studies, research up until now shows that the use of virtual reality in mental healthcare can lead to at least comparable treatment outcomes as treatment as usual and can bring other benefits as well, such as increased treatment motivation, efficiency and cost-effectiveness (Geraets et al., 2021; Turner & Casey, 2014). Amongst other things, reviews have shown that the use of VR in treatment of anxiety is more effective than a waiting list condition and has similar outcomes as regular cognitive behaviour therapy (CBT), but is viewed as a more appealing and efficient option for patients and therapists (Carl et al., 2019; Turner & Casey, 2014). Furthermore, the addition of VR to CBT in psychosis has been shown to lead to less paranoid ideation and momentary anxiety (Pot-Kolder et al., 2018) and is cost-effective (Pot-Kolder et al., 2020). Additionally, patients with psychosis seem to have a greater preference for VR than in vivo exposure and VR can lead to more treatment motivation (Rus-Calafell et al., 2018). Studies show that VR has much potential for treatment of hard-to-involve patient populations that are known for their low treatment motivation, such as forensic psychiatric patients (Kip et al., 2019; Klein Tuente et al., 2020). The potential of VR is not new: the first studies that highlighted the effectiveness of VR were published over 20 years ago (Geraets et al., 2021; Jerdan et al., 2018). Nevertheless, VR is not used as an integrated element of mental healthcare yet, for example: it is not a standard part of treatment protocols (Brown et al., 2020; Geraets et al., 2021; Kip et al., 2020a, 2020b). Because implementation of VR in clinical practice is very challenging, there is a need for more knowledge on the implementation of VR in mental healthcare.

Challenges with implementation in mental healthcare are not unique for VR: implementation has been proven to be very difficult for other evidence-based eHealth interventions as well, such as internet-based interventions and mobile apps (Kip et al., 2020a, 2020b; Tossaint-Schoenmakers et al., 2021). Low uptake in practice is problematic as this results in interventions that are not used as intended and thus will not lead to the expected improvement of patient outcomes and increased efficiency of care. Up until now, not much attention has been paid to systematic, holistic implementation of eHealth interventions such as VR (Kouijzer et al., 2023). There are multiple frameworks that can be used to guide studies on implementation, such as the Diffusion of Innovations theory (Rogers, 2010), the NASSS Framework (Greenhalgh et al., 2017), and the Consolidated Framework for Implementation Research (CFIR) (Damschroder et al., 2009). The CFIR is an especially useful conceptual framework for planning and assessment of multilevel implementation processes because it is based on a broad range of existing implementation theories and implementation research (Damschroder et al., 2009). The CFIR provides a comprehensive overview of a total of 39 implementation constructs to facilitate identification of factors that influence intervention implementation. These 39 constructs are structured into five domains: (1) Intervention characteristics: the features of an intervention, e.g. its complexity and relative advantage; (2) Inner setting: the characteristics of the implementing organization, e.g. leadership engagement, implementation climate; (3) Outer setting: the characteristics of the external context or environment, e.g. external policy and incentives; (4) Characteristics of individuals: features of individuals involved in implementation, e.g. patients’ and healthcare professionals’ knowledge and beliefs about the intervention; and (5) Implementation process: strategies or tactics that might influence implementation, e.g. engaging appropriate individuals, reflecting, and evaluating.

While the CFIR has been used to study implementation factors of several eHealth interventions, such as internet-delivered CBT and internet-based communication services (Hadjistavropoulos et al., 2017; Varsi et al., 2015), it has not been used to study the implementation of VR. In the few existing studies on implementation of VR, either no implementation model was used, or the focus was on only one implementation domain, such as the skills of the practitioner, neglecting other important variables such as the organizational setting or wider context (Brown et al., 2020; Kouijzer et al., 2023; Lindner et al., 2019; Nolet et al., 2020). Because of the unique characteristics of VR compared to other types of technologies—for example, its immersive qualities and the important role of the therapist in setting up personalized virtual scenarios—findings on implementation of other eHealth interventions cannot simply be copy-pasted to the use of VR in mental healthcare (Cornet & Van Gelder, 2020). This highlights the need for research that applies the CFIR to the implementation of this unique technology in mental healthcare.

In most existing implementation research, the CFIR is used to map determinants of implementation in a reflective manner, meaning that it is used after implementation to identify factors that were proven to be related to implementation. To illustrate: studies identified factors such as limited resources from the organization or care providers’ belief in the intervention as important for implementation after the intervention was introduced in practice (Hadjistavropoulos et al., 2017; Varsi et al., 2015). However, models such as the CFIR can also be useful for the prospective planning of implementation. They can support researchers in taking all relevant implementation domains into account and prevent a focus on one implementation level, such as therapists or the intervention, and can support implementers in adopting a systematic, theory-based approach towards implementation (Kirk et al., 2015). More specifically, the CFIR can be used to identify implementation factors for its multi-level domains, to set implementation objectives for these factors, and to determine accompanying implementation strategies to guide implementation in practice (Powell et al., 2012; Varsi et al., 2019). However, the use of these models to plan implementation in practice does not seem to be widespread yet. Also, translating an abstract framework into a coherent implementation process is easier said than done. Consequently, to ensure successful implementation of evidence-based eHealth applications such as VR, more knowledge is required about how to apply implementation models to practice.

In this paper, we first describe the steps we took to plan implementation, by translating the CFIR into specific implementation factors, objectives, and strategies to implement VR in mental healthcare (Chambers et al., 2013). Second, we present the outcome of a formative evaluation of the implementation process with therapists who used VR to investigate if and how the identified implementation factors were experienced. By presenting our systematic approach towards implementation planning, we aim to support other researchers and practitioners in systematic implementation planning. To reach this overall goal, multiple sub-goals are formulated: to identify factors that influence the implementation of VR in mental healthcare, using the CFIR model, earlier implementation research, and experiences from clinical practice; to formulate implementation objectives and identify concrete strategies for the identified implementation factors and translate them into an implementation process; and to investigate if and how the identified implementation factors and objectives were addressed in the implementation process according to participating therapists.

Methods

Study Design

To reach the three sub-goals of this study, the project was divided into two phases: the development of an implementation plan based on implementation factors, and an interview study to evaluate the implementation process. To reach the first two sub-goals, an implementation plan was created by the project leaders and a researcher, who designed and executed the implementation plan in close collaboration, in which the project leaders focused mostly on the practical matters and the researcher provided scientific input and generated the study design. The initial implementation plan was based on the CFIR model, outcomes of an earlier implementation study, and experiences of the project leaders and researcher (see Online Appendix 1). For each implementation factor, accompanying objectives and strategies were identified, which were translated into an implementation process. To reach the third sub-goal, the second phase of this study was a formative evaluation of the implementation process to investigate if and how the previously identified implementation factors and objectives were addressed during the actual implementation and what points of improvement were. This was achieved by means of interviews with the therapists and project leaders who participated in the implementation process. After a short explanation of the organization, VR intervention, and project set-up, these two phases will be further explained.

Organization

This study took place within two organizations: Dimence and Transfore. Dimence offers specialized mental healthcare to both in- and outpatients that are diagnosed with—amongst other things—autism, personality disorders, trauma, anxiety disorders, depression, and psychosis. Transfore offers forensic mental healthcare to in- and outpatients who have committed or are on the verge of committing an aggressive or sexual offense and also are diagnosed with at least one psychiatric disorder. This study was only focused on outpatient clinics. Transfore and Dimence are both part of the Dimence Groep, a large mental health care organization in the east of the Netherlands that offers a broad range of mental healthcare by means of different smaller organizations. One of the strategies of Dimence Groep is to improve and optimize the quality of care using technology, which means that technologies such as internet-based interventions and mobile apps are already being used. Based on experiences from practice and research, VR was seen as a promising treatment option for mental healthcare, not just by therapists, but also by their managers—who are responsible for the functioning of therapists and the financial situation of Dimence Groep.

Virtual Reality Intervention

Based on an exploration of the possibilities of VR and rising interest from therapists, the decision to use the interactive, animated VR system of the Dutch company CleVR was made by the management of Dimence Group. The main reasons for this were the possibilities of adapting the content of VR treatment to the individual patient, the possibility of role-playing in VR, and bringing the ‘real world’ into the treatment room via multiple realistic virtual environments. In the VR system of CleVR, a personalized scenario can be built. Together, the patient and therapist can select the appropriate virtual environment (e.g. a bar, living room, or park), the avatar(s) with whom the patient can communicate (e.g. a police officer, large man, or child), and specific ‘triggers’ that can be added to the scenario (e.g. beer bottles on a table, a police siren, or barking dog). Once the scenario is built, the patient can walk through the virtual environment with controllers or can participate in a virtual roleplay in which the therapist ‘plays’ the other avatar by means of a voice-morphing microphone and a dashboard with which movements and facial expressions of the avatar can be controlled. In Fig. 1, screenshots of virtual environments, role-play with avatars, and the set-up of the VR set can be seen.

Fig. 1
figure 1

Software screenshots and CleVR system

Full size image

Project Set-Up and Pilot

Initially, the VR-project described in this paper started in 2019, with the decision of management to buy two VR sets of CleVR. Due to disappointing implementation outcomes of earlier eHealth-projects, it was decided to invest more time and effort into the implementation of VR. Consequently, in November 2019 a project manager and researcher (HK) were asked to plan and execute the implementation process of VR. In 2020, another member joined the team as project leader. This team worked closely together, with the project leaders taking on a more practical and supportive role—e.g., planning meetings, arranging financial matters, and contacting therapists for participation in the pilot. The researcher took on the role of a scientific advisor and focused mostly on the use of theory and research in setting up the implementation plan and designing and executing the data collection.

Initially, the use of VR was planned to start in March 2020, but due to COVID-19, the training of the therapists in VR use was rescheduled to July 2020, resulting in a delay of starting with the use of VR in practice as well. The implementation process was split into two parts: a pilot in which a small group of 8 therapists used VR for 9 months, and the general implementation scale-up phase in which, based on the lessons learned of the pilot, VR was implemented in the entire organization. The current study is focused on the design and evaluation of the pilot phase. A more detailed timeline and description of the implementation process is provided in Online Appendix 2.

Phase 1: Implementation Factors, Barriers, Objectives, and Strategies

In the first phase of the implementation process, an implementation plan was created to serve as the backbone/foundation of the implementation process (see Online Appendix 1). The implementation plan was structured as a table and based on the domains of the CFIR. For overview purposes, the CFIR domain ‘characteristics of individuals’ was split into a ‘therapist’ and ‘patient’ domain due to the different types of factors associated with both types of adopters. Because the domain ‘implementation process’ is focused on the identification of strategies or tactics that influence implementation (Damschroder et al., 2009), it was not used as a domain for factors, but used as the foundation for the identified implementation strategies. In the implementation plan, four categories were used: (1) implementation factor—a determinant that was expected to be relevant for the implementation of VR, (2) implementation barriers—expected problems or points of attention for each implementation factor, (3) implementation objectives—goals connected to each identified implementation barrier, and (4) implementation strategies for each objective. The template for the implementation plan is provided in Table 1.

Table 1 Implementation plan template
Full size table

To start the process of identifying factors for each domain, the implementation plan template was initially completed using factors identified in an earlier study within the Dimence Groep on the implementation of internet-based interventions (Kip et al., 2020a, 2020b). Using this overview, the two project leaders and researcher (HK) selected implementation factors and accompanying barriers that were relevant for VR and then adapted them to make them relevant for VR implementation. Second, the project leaders and researcher brainstormed about additional implementation factors and barriers based on their own experiences and added them to the implementation plan. Third, the content of the implementation plan was compared to the 39 constructs of the CFIR and several additions and changes were made. Fourth, implementation objectives that had to be reached to account for the identified barriers were formulated based on the implementation factors. Finally, the project leaders and researcher came up with implementation strategies for each implementation objective, mostly based on their own experiences since no research in which specific strategies for the use of VR in mental healthcare were matched to implementation factors was available. However, studies on implementation of internet-based interventions in mental healthcare were used as inspiration (Feijt et al., 2018; Kip et al., 2020a, 2020b). The completed implementation plan can be found in Online Appendix 1.

The strategies described in the implementation plan were translated into a concrete implementation process by the project leaders. First, the project leaders discussed the main conclusions that arose from the implementation plan with two managers responsible for the financial aspect of Dimence Groep, and the therapists that would participate. After that, the process was described in a project document (see Online Appendix 2 for the sections of this document). This project document served as the foundation for the implementation process of the pilot (see Online Appendix 2 for the timeline and main activities of the pilot).

Phase 2: Implementation Factor Evaluation

After the pilot phase of the implementation process was completed, interviews with the participating therapists and the two project leaders were conducted about their experiences with the implementation process to evaluate the relevance of the previously identified factors and search for points of improvement of the implementation process. This can be seen as a formative evaluation because the aim of this interview study was not to make statements about the effectiveness of the implementation plan, but to identify points of improvement.

Participants

While eight therapists initially participated, six remained after the pilot: one therapist found another job and one retired. Three therapists worked at Transfore and three worked at Dimence. Furthermore, the two project leaders were interviewed. The involved researcher (HK) was not interviewed because she supervised the interviewing process. Participants were invited for the interview by means of an e-mail and all eligible participants agreed to participate. The characteristics of the participants are provided in Table 2. This study was approved by the Ethical Committee of the faculty of Behavioural, Management & Social sciences (BMS) of the University of Twente (number 210108).

Table 2 Characteristics of participants of the interview study
Full size table

Materials and Procedure

A semi-structured interview scheme was developed to be able to gain in-depth insights into the experiences of the participants. The first draft of this interview scheme was based on the online interview guide, created by the authors of the CFIR to ensure that all relevant domains were covered (CFIR Research Team, n.d.). This interview guide was adapted to VR within mental healthcare and was initially structured by means of the domains of the CFIR. However, based on a pilot interview, the decision was made to structure the interview scheme in a chronological way, working from the start of the implementation process and ending with a look to the future, to allow for a more logical flow to the conversation.

After a brief introduction and signing the informed consent, the interview started. First, participants were asked several introductory questions about their current function and experiences with VR. Second, questions were asked about how they experienced the introduction of VR and were asked to reflect on this first encounter. The third set of questions was focused on the subsequent VR-training and practicing with and using VR afterwards. Fourth, participants were asked about what kind of support therapists needed for using VR in treatment and how what their role in the implementation of VR for the upcoming year would look like. Additionally, questions were asked about their ideas about the future of VR within the organization. The interview scheme can be found in Online Appendix 3.

Interviews took place at a location preferred by the participant and were conducted by a researcher who was not part of the project team (DH). Ideally, interviews were conducted face-to-face, but due to the COVID-19 pandemic, several interviews were conducted via videoconferencing via Microsoft Teams. On average, the interviews took 48 min, ranging from 36 to 54 min.

Data Analysis

After the recorded interviews were transcribed verbatim, relevant fragments were identified based on the objective of this study. A combination of deductive and inductive coding was used. First, the fragments were categorized in one of the CFIR domains that were used to structure the implementation plan: characteristics of individuals—therapists, characteristics of individuals—patients, intervention characteristics, inner setting, and outer setting. Second, implementation factors from the implementation plan were used to further cluster the fragments. This top-down, deductive coding approach allowed for structured evaluation of the implementation plan. If fragments did not fit within any of the existing implementation factors, a new factor was created. Third, an inductive approach was used to generate codes that described the experiences of participants within the implementation factors. A bottom-up coding approach was used to stick to participants’ experiences, as opposed to using the predetermined implementation objectives as a foundation for codes, which might have caused tunnel vision. Codes on the experiences with implementation were generated in an iterative way, using the method of constant comparison (Boeije, 2002). The process was first discussed by two researchers, after which one researcher (HK) coded all fragments. Next, a second researcher (GBH) checked 34% of the coded fragments. The researchers’ opinion differed on only 9.4% of the codes. Based on differences between researchers, minor changes were made to the allocation of codes. Fourth, the implementation objectives from the initial implementation plan were compared to the inductively generated codes to analyse the extent to which the initially set objectives were aligned with the experiences of the participants to evaluate whether these objectives were reached during the implementation process.

Results

Phase 1: From Theory to Objectives

All implementation factors, accompanying potential barriers, implementation objectives, and implementation strategies were generated and combined into an implementation plan. In Table 3, all implementation factors and accompanying objectives are provided. In Table 4, an example of the entire table, including barriers and implementation strategies, is presented. The entire implementation plan can be found in Online Appendix 1.

Table 3 Identified implementation factors and accompanying objectives
Full size table
Table 4 Example of table for implementation factor ‘Knowledge & Skills’
Full size table

Based on the strategies that were identified in the implementation plan, the implementation process was designed and deployed. In Online Appendix 2, an overview of the timeline of the process is provided. As can be seen in the Online Appendix, a kick-off meeting was scheduled with all therapists who wanted to participate to inform them about the project. After that, a technical training by the VR company was scheduled, but this was delayed by 4 months due to the pandemic. In July, the 1-day technical training took place with all eight participating therapists. This on-site training focused mostly on acquiring skills to set up the system and control the dashboard and voice-morphing microphone by means of small assignments and was delivered by the software developer. After this training, two different types of online bi-weekly meetings were scheduled by the project leaders. In the project meetings, all eight therapists and project leaders met bi-weekly to discuss the progress of implementation and identify points of improvement where more support was required. Additionally, bi-weekly content-related intervision sessions with smaller subgroups of two to three therapists were scheduled. Intervision refers to group sessions in which experiences with treatment are exchanged and discussed amongst colleagues in a structured, systematic manner. Therapists structured these sessions in which they exchange experiences with VR in treatment individually, and a project leader was not always present. Since intervision is an important part of therapist-training, therapists were experienced with this method. It is important to note that—despite efforts to look for experts—it was not possible to organize supervision sessions with an expert, because, at that point in time, there were no other therapists with experience with VR due to the innovative nature of this project. Furthermore, several practice sessions were organized during which therapists could practice with VR amongst each other to prepare them for actually using VR with their patients. Throughout the entire process, project leaders and the researcher (HK) met up on a weekly basis to evaluate the process. The implementation plan was used to assess if the planned activities were executed and if the implementation process was still in line with the set objectives. During the process, implementation materials such as patient flyers and an online environment in which materials could be shared were created. Throughout the entire project, technical support was available via the helpdesk of the software developer, practical support was provided by the project leaders, and content-related support was mostly provided amongst the therapists, who were learning as they were going. Consequently, while there was much material on technical support, at the beginning of the pilot, there were no treatment protocols that prescribed how VR should be integrated in treatment available. Halfway through the project, four protocols were retrieved from other organizations and researchers, but these were hardly used because they were considered to be too lengthy. Consequently, therapists mostly decided themselves on how to integrate VR within their existing treatment and how to make it a part of the protocols they used. In other words: during the pilot, therapists had to pioneer and innovate because hardly any content-related material was available.

Phase 2: Evaluation of the Implementation Plan

In the second phase of this study, interviews were conducted with therapists and project leaders to evaluate their experiences with the implementation of VR during the pilot. Below, the outcomes of these interviews are provided, structured by means of the CFIR domains and predetermined main implementation factors from the implementation plan that was developed in Phase 1. The inductively generated codes are structured by means of these factors. Finally, shortened versions of the initial implementation objectives are added to the implementation plan to facilitate the comparison of the findings from the interviews to investigate whether the set objectives were reached and identify points of improvement.

Characteristics of Individuals—Therapists

As can be seen in Table 5 below, no new implementation factors were identified regarding the characteristics of therapists. However, the identified codes did digress from the previously set implementation objectives.

Table 5 Outcomes of analysis: ‘characteristics of individuals—therapists’ domain
Full size table
Investing Time and Effort

This code within this factor refer to the extent to which therapists have and experience the possibility to spend time and effort on using VR in treatment. Almost all participants referred to the high time investment: it takes a lot of time to, amongst other things, setup a VR set, get acquainted with using it, acquire the necessary skills, and participate in project meetings. A way to ensure that this time actually is invested, is by means of scheduling VR sessions in one’s calendar, as was explained by Participant 6: “I think it is important to have a caseload from the start, so patients that can use it, and that you also really block a part of the day weekly or bi-weekly to really do this well.”. Participants mentioned waiting lists and pressure for output as a hindering factor: because therapists are required to see a predetermined number of patients, they feel like they have less time for VR. Also, they indicated that they feel guilty for spending time on VR that they also could have spent on a patient that is waiting for treatment. Participant 3 described this issue as follows: “But you will easier scale down on these things [VR] and invest less time, because you think: well, I have that production time, we have to see patients.” Three participants emphasized the importance of having the subjective perception that they were receiving time for VR from management. The project leaders indicated that all therapists officially were granted with additional time to work on VR, but this was not perceived as such by all participants. Participants indicated that it was especially important that their direct managers who supervised them gave them the feeling that they were able and allowed to spend time on VR. Finally, Participant 4 indicated the importance of dedication and commitment of therapists to the use of VR.

The initial objectives from the plan were compared to the outcomes from the interviews. Participants clearly indicated that they were not able to invest the required time due to pressure for output, but their willingness was not experienced as a problem. The previously made distinction on starting and continuing to use VR was not as clear, mostly because participants said both are closely related to each other. The interviews clearly showed the importance of scheduling time and accounting for the pressure of patients on waiting lists, which was less prominent in the original implementation plan.

Introduction of VR and Support During Use of VR of Patients

This factor was only identified once in the interviews and refers to the way in which therapists introduce and keep on communicating about VR to patients. Therapist 2 indicated the importance of clear instructions on what to do and not to do when first using VR with patients: “Because at the moment you are first wearing those glasses… For example, I tended to actually walk in real life, but that is very confusing. So these type of instructions are really important to give to patients and they help.”

When comparing the initial objectives to the codes, it became clear that this code was only mentioned once. This shows that the previously set objectives were not identified in almost all interviews.

Integration of VR in Routines and Protocols

The codes within this factor refer to the integration of VR in standard treatment procedures by therapists. Almost all participants emphasized the importance of integrating VR in treatment protocols to ensure it is embedded in treatment and not used as a separate, stand-alone tool. Participants indicated a need for CBT protocols that are extended with VR, in which it is made clear when and how VR should be used. Multiple participants indicated that in some forms of treatment, the use of VR was more straightforward than in others. Participant 3 said the following about this:

"I have to really search [how to use VR] when not using a very structured CBT protocol. In forensic mental healthcare it is way more difficult, because we are working less protocolled and the care pathways are not very distinct, as opposed to regular mental healthcare, for example with mood disorders, where there is a more standardized CBT-protocol. I think that this makes it easier to say that VR is a standard part that we implement [in regular mental healthcare]. But here we really have to search per patient what to do in treatment, and that can be very different."

Participants also referred to the difficulty of setting up personalized scenarios to conduct roleplays and referred to drama therapy as a source of inspiration for acquiring these new skills. Furthermore, two participants expressed the need for clear indication criteria for VR, which means that it should be clear for which types of patients VR should be used. Furthermore, participants indicated that therapists often had difficulties with remembering to use VR as part of treatment. Participant 6 explained this as follows:

You can be enthusiastic, I think: ‘let’s do this, let’s go live next month!’ But it doesn’t work like that […], because you are just swayed by the issues of the day. You are doing your thing, but you also have something really great lying around. But you often think: that will come someday and you unintentionally forget about it.

The comparison from the results to the objectives from the plan showed that two initial objectives were identified in the interviews: integration in treatment and remembering to use VR. However, the importance of treatment protocols became apparent from the interviews but was underemphasized in the initial objectives. The same goes for indication criteria that are used to determine for whom (not) to use VR: these were not part of the initial objectives.

Knowledge and Skills

The codes within this factor refer to the way in which therapists acquire the required knowledge and skills for successfully using VR. Almost all participants emphasized the importance of regularly practicing with colleagues: the training at the start of the process was not experienced as sufficient to acquire enough self-efficacy to start using VR with patients. Practicing was seen as necessary to improve practical skills such as getting acquainted with the control of the system, and content-related skills, such as conducting a roleplay in VR. Therapist 2 said the following about this: “It is really good to become fluent in it, […] we have practiced a lot with each other, but it now has stalled a bit, but it was very helping, because you are working on it together, almost weekly, an hour.” Participants also emphasized the importance of ‘just doing it’, which refers to starting to use VR with patients, even though one does not feel completely confident yet, as was mentioned by Therapist 4: “And I was thinking, well, I can practice yet again, but at a certain point I have to dive into the deep end and just start doing it. And the times that I am doing it are invaluable, trial and error!” Furthermore, regular use and sufficient experience with VR were viewed as important to further improve skills and ensure that they do not fade away. Another important topic was related to the consultation of experts. Multiple participants indicated that they missed having experienced colleagues that they could turn to for advice because they were the first to be trained. Consequently, four participants suggested to serve as experts for the next set of trained therapists. Finally, intervision—which refers to sessions in which experiences with VR treatment are exchanged and discussed amongst colleagues in a structured way—was mentioned several times. Participants appreciated the existing intervision groups for therapists but expressed a need to set up groups with direct colleagues in their team, as opposed to therapists that work at other locations.

The identified codes were compared to the initial objectives from the plan. This comparison showed that participants did not elaborate as much on their lack of skills but emphasized the importance of receiving more time to gain experience in using VR by practicing with colleagues and using it in treatment. They also identified multiple strategies to increase their skills that were not as present in the initially developed implementation plan, such as the importance of learning from expert-colleagues. As opposed to the objectives, they did not make a clear distinction between technical and content-related skills: they needed to be improved simultaneously since they are two sides of the same coin.

Attitude Towards Technology

The codes within this factor refer to the opinion of therapists about VR or technology in general. Almost all participants mentioned the importance of their own curiosity and enthusiasm towards VR: they viewed it as the future of treatment, were curious about if and how it can improve their treatment and wanted to try out new things. Furthermore, three participants indicated that many of their untrained colleagues are enthusiastic, which also motivated them. Participant 3 added the following footnote about the role of the attitude towards VR:

Well, I think that everyone is enthusiastic, or 80% of the people are actually enthusiastic. That doesn’t mean that they will actually do it. They do like it a lot and find it fantastic, but to actually take that step… […] But well, the feasibility for themselves or the willingness to learn something new… I think that plays a part in that of that 80%, not 80% will actually do it. If we can get 20% to really do it, that would be great!

The comparison to the initial objectives showed that the enthusiasm aspect from the objectives became very clear in the interviews. The interviews also showed that merely enthusiasm isn’t enough for successful implementation. Participants also mentioned the importance of a positive attitude towards VR of their colleagues, which was not part of the initial objectives.

Topic of Conversation Among Colleagues

The codes within this factor refer to the ways in which knowledge and experiences with VR are shared with other therapists in an informal way. Participants indicated that they often exchange experiences with other trained colleagues to learn from each other, especially by discussing how they used VR with individual patients. Several participating therapists also took initiative to share their experiences with their team, which only consisted of untrained therapists. This could be in the form of presentations, but also more informal, as was explained by Participant 2: “I am now noticing, once I’ve set up VR and colleagues who are enthusiastic drop by, and they want to know: how does it work? […] I also mention that I am working with this during staff meetings, and then people will hopefully become more enthusiastic and want to be trained as well.”

The comparison showed that the codes that arose from the interviews were very similar to the previously set objectives.

Experienced Benefits/Added Value

The codes within this factor refer to the experienced or expected benefits of VR for treatment by therapists. Participants experienced or expected advantages for treatment in general, such as increased motivation of patients, acquiring more insight into behaviour, and facilitating exposure exercises. Therapist 5 said the following about this:

Treatment of anxiety and accompanying exposure treatment… There is a certain group that is very avoidant, and their treatment often stagnates because they are not practicing with exposure in daily life. […] Well, for that group it would be desirable if you can expose people in the treatment room, so to speak.

Two participants also indicated that they sometimes had doubts about the advantages, especially before starting with VR, and thought that to get a good grasp of the potential, it is necessary to actually experience VR.

The comparison from the codes to the objectives from the plan showed that the importance of clear goals was not mentioned in the interviews, so this objective might not be a good fit with this factor. The interviews showed more detailed benefits of VR than the initial plan.

Characteristics of Individuals—Patients

As can be seen in Table 6 below, the participants did not say much about patient factors related to the use of VR in treatment. This shows that many of the previously set implementation objectives—more specifically, the factors Motivation, Conscientiousness, and Literacy and educational level—were not present in the interviews. The factors that were identified are discussed below.

Table 6 Outcomes of analysis: ‘characteristics of individuals—patients’ domain
Full size table
Experienced Benefits

The codes within this factor refer to benefits of VR for patients, as experienced or expected by therapists. Participants experienced several advantages for individual patients, such as becoming more aware of anxiety, providing more insight into behaviour, practicing with behaviour that patients often avoid or experiencing sensory overstimulation. Therapist 3 gave the following example: “Recently a man—I did two sessions with him, and both times he was very sceptical. But eventually he forgot that during the roleplay and when I saw him for a second time, he said that it really transferred to is daily life, what he learned, that he generalized it to a lot of other things. […] And his confidence received a boost, and I found that really valuable to experience. So in that sense, I believe that it can work accelerating for treatment.”

The outcomes of the interviews were again compared to the objectives that were initially set when comprising the implementation plan. Participants mostly discussed potential benefits from their perspective and did not go into feedback from their patients. Potential benefits as expected or experienced by patients were not mentioned in the interviews.

Psychosocial Situation

The codes within this factor refer to life circumstances or the well-being of the patient that can influence the use of VR. One therapist indicated that one of their patients had a psychiatric crisis, which caused a pause in the use of VR. Another therapist mentioned that they had a patient that would be suitable, but they did not want to try out VR with this patient due to epilepsy.

The comparison of the interview outcomes to the initial objectives showed that patient-related characteristics that are related to the use of VR in treatment were hardly mentioned by participants, so the objectives were not retrievable in the interviews.

Intervention Characteristics

As can be seen in Table 7, characteristics of the intervention that are related to the use of VR were not often discussed in the interviews. The identified codes are discussed below.

Table 7 Outcomes of analysis: ‘intervention characteristics’ domain
Full size table
User-Friendliness

These codes refer to the ease of setting up the VR set—which refers to hardware, and the ease of use of the software—which refers to the dashboard that therapists use to create scenarios in VR and control the virtual environment and avatars during VR sessions such as roleplaying. One participant indicated that there sometimes were issues with hardware, such as a malfunctioning microphone, and that setting up the VR set can be very complicated and takes time, especially at the beginning. Five participants indicated that the use of the software—i.e. the dashboard to create scenarios—can be complicated because there are a lot of different functionalities, which can require some searching. Participant 4 said the following about the user-friendliness:

The program itself, I didn’t find it very user-friendly, but I saw the new version that was developed, and I clicked right through that, so that was a lot easier. But it still costs quite some time. That you, let’s say, want to use the bus, which type of people, how long, and how to exactly see how many people are in the bus, to really do that well, I still found that a challenge.

The comparison of the outcomes to the initial objectives showed that, while in the objectives, the focus was mostly on the impression of the design in general, the participants made a clearer distinction between the hard- and software of the system, since this required different types of skills to use. The other factors from the plan were not identified in the interviews.

Inner Setting

As can be seen in Table 8, characteristics of the organization that are related to the use of VR were mentioned relatively often in the interviews. No new implementation factors were identified.

Table 8 Outcomes of analysis: ‘inner setting’ domain
Full size table
Introduction of VR to Therapists

This codes within this factor refer to activities organized by the organization to first introduce VR to therapists. Participants indicated that the organization needs to organize different types of activities to ensure that therapists who do not work with VR also get acquainted with it. Examples were webinars, presentations during staff meetings of team managers who were the supervisors of therapists or using the internal communication channels. Participant 1 made the following point about this:

Only how you disseminate it in the organization… There is so much going on, but the information in my team mostly has to come from me. I find that quite strange. It is really weird that colleagues don’t know that we at [the organization] work so much with VR, even though they have been told ten times, and still they don’t remember. Or that VR is becoming important. It is posted a lot on Intranet, but nobody reads that.

Multiple participants also referred to the introduction meeting in which VR and background information were discussed. Some perceived this meeting as very useful, while others felt a bit overwhelmed and were confused about the expectations from the organization. Two participants—both project leaders—referred to the decision of the organization to start using VR, and indicated that it took a lot of time and effort to convince the organization to invest. Finally, one participant indicated that the first project team meetings helped them in getting acquainted with the technology, on top of the introduction meeting.

The comparison showed that while the initial objectives were formulated quite broadly, participants mentioned more specific activities that were used to introduce them to technology. They emphasized the importance of communicating about VR in different ways, using different communication channels.

Content-Related Support for Therapists

This codes within this factor refers to the ways in which the organization offered content-related support for therapists to use VR in treatment. Almost all participants mentioned the importance of having a go-to contact person within the organization that they can go to when they require support. Participating therapists were positive about the project leaders’ reachability and support. Both project leaders expressed the need for a new contact person after the pilot with more treatment-related expertise, as Participant 7 expressed as follows: “As project leaders, we are now taking the lead. But if you want to integrate it in the normal routines of therapists, then you should ensure—just like with care programs, that there is a content-expert in the lead.” Four participants expressed the need for additional training for therapists, either in the form of a second training moment to bolster or improve their technical and content-related skills or as a longer training program with regularly recurring meetings. Participant 4 said the following about this: “I would create a sort of educational trajectory, where intervision is a scheduled part of [the process], and that you have to practice with patients.” Furthermore, three participants indicated the importance of the regular meetings with the project team to discuss matters such as existing treatment protocols, implementation materials like flyers or websites, or how to communicate about VR within one’s team. Additionally, the importance of where and how to retrieve information for support in using VR was mentioned. Finally, one therapist said that they would receive more content-related support if the organization would train entire teams, or at least more colleagues in their team, which would make it easier to learn from colleagues that treat similar patients.

The input of the participants of the interviews was again compared to the objectives from the implementation plan. These objectives were mostly focused on distributing information by the organization, while the participants emphasized the importance of setting up knowledge-exchange activities amongst therapists to learn from each other. Furthermore, in the interviews, the importance of providing additional training moments became apparent, which was not part of the initial objectives.

Integration in Organizational Structures

The codes within this factor refer to activities of the organization to ensure that VR is integrated in established organizational processes, products, and activities. Multiple participants underlined the importance of integrating VR in existing care pathways for treatment of, e.g., anxiety disorders, depression, or aggression. According to participants, this should be done by means of standards and protocols that are adopted by the entire organization, as opposed to the current ad hoc way in which VR is used, where therapists often ‘figure it out as they go’. Participants highlighted the importance of clear indication criteria because it is often not clear to them when and why to use VR. Participant 3 made the following suggestion:

That it is a standard part of treatment, just like you are indicating EMDR [Eye Movement Desensitization and Reprocessing], that you also say during the intake: ok, treatment will look like this, with three VR sessions. Instead of how it is right now, that you have to come up with it along the way, halfway during treatment.

Furthermore, participants mentioned the importance of ensuring that all supportive systems surrounding VR are well arranged to ensure integration in the organization. Examples are the administration of VR sessions in registration systems, transportation of VR sets to different locations, and integration in electronic patient files. Especially the two project leaders emphasized the important role of supportive services, e.g., those responsible for financing, technical services, and transport. They indicated that, at that point in time, many practical matters were not completely arranged yet, and that finding and involving the right persons was a very complex and time-intensive undertaking. Finally, participants suggested that the organization appoints specialized VR therapists that join treatment sessions of other therapists within their team. Participant 4 described this as follows: “I can imagine that it would be interesting to do it together, because the colleague knows a lot about the content, and I about the execution. So I can work on clicking buttons and the colleague works more patient-focused.”

The initial objectives were quite broad and focused on meetings and documents in general, but a comparison to the outcomes of the interviews showed more specific needs from participants related to the content of treatment. Also, the important role of supportive services of the organization were emphasized in the interviews, which was not reflected in the initial goals.

Practical Preconditions for Usage

The codes within this factor refer to the practical requirements that need to be met by the organization to enable therapists to use VR without experiencing thresholds. All participants indicated that the organization should ensure that therapists receive additional time to use VR since all participants perceived lack of time as a major barrier to using VR. Participants differed in the extent to which they were aware of the additional time they received from higher management and mainly attributed this to the perceived support and encouragement of their direct manager. Participant 2 said the following about this:

It would have helped me if I was clearly told that I am allowed to take that space and plan VR. […] With team leaders, what do they have to do? Well, I think that they can support people more when they start, so to speak. At this point it is still really, well, you receive time, so plan it yourself, which is easier said than done.

Another practical precondition that was mentioned by all participants was the location of the VR set. Participants indicated that the organization needed to arrange suitable VR rooms in which the VR set is always directly available. Initially, participants had to set up the entire system before starting treatment because it was not set up in a fixed room, which took them about 15 min per session. Participant 8, a project leader, said the following about arranging VR rooms: “It requires a lot more time than I ever expected. Just arranging a closet [to store the VR set in]… It is really surprising how much time this takes, but this is really something that needs to be arranged.” Finally, three participants mentioned the importance of safety of the therapist, especially when using VR with patients with aggression regulation problems; the VR set cannot be set up in an isolated room with no other therapists nearby.

While present in the initially set objectives, participants of the interview study did not mention technical problems as much because—if they occurred—they were often easily solved. Furthermore, the remarks of participants were in line with the objectives, with an essential role for enough available time and a good location.

Outer Setting

In Table 9, characteristics of the wider context that are related to the use of VR according to the participants are presented. Based on the interviews, two new factors were identified: one related to the Covid-19 pandemic that started during the pilot, and one related to the activities of the technology developer, which appeared to be very important to the participants.

Table 9 Outcomes of analysis: ‘outer setting’ domain
Full size table
Demands of Health Insurance Companies

This codes within this factor refer to the role that health insurance companies play when (partly) reimbursing organizations for using VR. One therapist referred to unclarities regarding the declaration of VR-related hours: it was unclear what has to be done to receive funding from insurers. One project leader pointed out that registration of hours in systems, which is required by health insurers, was difficult and the demands could also differ per health insurer.

When comparing the interview outcomes to the objectives, it became clear that this factor was not often mentioned in the interviews, and if it was mentioned, participants often expressed not having enough knowledge about this topic.

Costs

The code within this factor refers to the fixed financial investment of organizations that is required for the purchase and usage of VR. Two participants referred to the high costs of VR and wondered if this would be sustainable over time. Participant 4 said the following about this:

Well, it might become a problem. We have to set this up and start pilots and then a lot is possible. But I can imagine that if so to speak in five years, or even two, that it may cost a lot of money, a lot of time, for just one patient.

Compared to the initially set objectives, this code was hardly mentioned by the participants: they mostly expressed that costs were high but did not seem to have a clear overview, as was stated in the initial objectives.

Other Mental Healthcare Organizations

This code within this factor refers to the comparison with and possibility to learn from other mental healthcare organizations that use VR. The code was mentioned only once, by Participant 7, who said the following about this: “The way we register VR has to fit with, I don’t know, [other organization] registers. But at the same time, we are another organization, so our own financial department has its own demands.”

When comparing the interview outcomes to the objectives from the implementation plan, it became clear that this code was also not often identified in the interviews, and the initial objective about the state of affairs regarding VR was not represented.

Pandemic

This newly generated factor refers to the impact of the Covid-19 pandemic and accompanying lockdowns and measures on the use of VR. Four participants indicated that the pandemic caused a large delay in their uptake of VR, mostly because in-person appointments with patients were cancelled. Participant 3 said the following about this:

Covid definitely did not help and caused way more delay. We had the training day, and after that it was shut down for three months or, I don’t know for how long. So we couldn’t practice or anything, and all subject matter faded away, that was definitely the case. And I have some patients that want to do VR, but they don’t want to come to the treatment location yet.

One participant also indicated that they made appointments to use VR with a patient, but that the patient became infected and the VR appointment was cancelled.

This is a newly generated code so there were no previously set objectives to which the outcomes can be compared.

Technology Developer

This newly generated factor refers to the activities that were organized by the developer of the VR software and their impact on the start and sustained use of VR. Almost all participants referred to the 1-day training that was provided by the company. Most participants were very satisfied and became very enthusiastic about using VR. However, opinions about points of improvement differed: most participants also thought the training was too short, was overwhelming, and contained too much information, while two participants indicated that the training took too much time and wanted more information. All participants agreed that one training did not suffice, mostly because of its technical focus and lack of treatment-related information. Participant 4 said the following about the training:

I really liked that day, he was able to clearly talk about it. The pace was very high, so I’m not sure if it was suitable for everyone. I received a lot of information and I did get the idea that we were really trained during that day. But you’d actually want another day, or half a day, during which you can practice more.

Furthermore, two participants referred to the helpdesk set up by the company that they could call in case of difficulties or questions and indicated that they were very satisfied with it. Finally, two participants—both project leaders—were positive about the company’s focus on constantly improving and further developing its VR software.

This also is a newly generated code so no previously set objectives were present to compare the outcomes to.

Discussion

In this paper, we provided an overview of factors, objectives, and strategies that are relevant for the implementation of VR in mental healthcare and developed and formatively evaluated an implementation process based on these factors. Our study was based on the domains of the CFIR, earlier implementation research, and experience from practice. Even though there were many points of improvement, the interviews with therapists and project leaders showed that many of the initially set factors and objectives were identified and fit the needs of the therapists relatively well. The interviews helped in identifying points of improvement for accompanying implementation strategies that will be used to shape a larger implementation process. An important factor appeared to be the importance of having enough time to gain sufficient experience with VR. The interviews showed that the initially selected strategy of merely providing therapists with time did not suffice: clear communication from management and specific strategies such as blocking time in one’s agenda were found to be important to ensure that therapists actually experienced room to take the necessary time. This shows that the implementation factor ‘Investing time & effort’ was even more important than expected and that there is a need for more and better implementation strategies to address this issue. Another important factor was the integration of VR in existing care structures: while attention to this was paid in the plan, the interviews showed how difficult it was for therapists to get VR ‘in their system’, again highlighting the need for better implementation strategies. Furthermore, technical and content-related skills to use VR proved to be essential: this was part of the implementation plan, but therapists underlined the importance of more opportunities to improve their skills. The interviews showed that therapists valued a multiple-method approach, in which intervision to learn from each other, multiple skill-training sessions, and learning from experts should be combined. However, while these strategies were introduced from the start and appreciated, there was room for improvement in terms of available time to spend on skill-training.

Furthermore, differences and points of improvement between the factors and experiences of therapists were identified. Amongst other things, the role of the VR-developer in providing training and technical support was much larger than expected, highlighting the importance of including a factor that addresses the collaboration with the technology developer in planning and executing implementation processes. Additionally, the importance of explicitly integrating VR in existing treatment protocols was found to be more important than initially expected. Furthermore, the interviews showed that the patient perspective was hardly mentioned by the therapists, while multiple patient-related factors were included. In the interviews, therapists mostly focused on their own skills, insecurities, and required support during this first implementation phase. Finally, the major impact that the Covid-19 pandemic had on the implementation process became very clear: not just in terms of practical problems due to lockdowns, but also because adapting to the pandemic required much mental energy, which had a negative impact on the energy required to start using a new technology such as VR. To summarize: while the implementation plan offered a good first foundation with factors and accompanying strategies, our formative evaluation shows that it requires multiple adaptations to better fit the context and needs from the therapists and project leaders.

Interpretation of Findings

Implementation is an Iterative Process

While the initially formulated implementation factors and strategies provided useful input and ensured that the multiple levels involved in implementation were accounted for, multiple new factors and strategies arose during the process. Especially in the initial stages of implementation, practical matters such as having a room to use VR, communication about available time for VR use, scheduling VR sessions, and integration into administrative systems proved to be essential. The importance of these practical boundary conditions is in line with findings from other implementation research on technology in healthcare (Feijt et al., 2018; Kip et al., 2020a, 2020b; Schreiweis et al., 2019; Titzler et al., 2018; Vis et al., 2018). It seems to illustrate a kind of ‘implementation hierarchy’, comparable to that of Maslow’s Hierarchy of Needs (Maslow, 1981), where these practical conditions first need to be met before being able to progress to higher levels of implementation, such as self-efficacy and an experimental mindset. While ideally, these practical matters should be accounted for as early as possible (Schreiweis et al., 2019), our study showed that this was more difficult than expected: new problems or points of attention only became apparent during the use of VR by therapists. This illustrated that an implementation plan cannot be finished before starting the process but is a ‘living document’ that has to be adapted throughout. Consequently, implementation should be viewed as an iterative process, during which constant adaptations have to be made to the identified factors, objectives, and strategies (Boustani et al., 2018; Kip, 2021). This is also in line with how implementation often takes place in practice: a technology is first adopted by a small number of therapists, i.e. the innovators, after which it is used by an increasingly large number of therapists, i.e. the early adopters, early majority, late majority and laggards from the Diffusion of Innovation theory (Rogers, 2010). The different phases in the process might require different implementation strategies. Looking back on the process described in this paper, we could have integrated more iterations—e.g., the interviews that were used as a formative evaluation step could have been conducted after several weeks as opposed to several months—to ensure that important points were identified earlier on. The importance of such shorter iteration cycles with constant evaluations is in line with the principles of agile science (Conboy et al., 2015; Hekler et al., 2016). This is in line with the importance of ‘just doing it’, which was emphasized by therapists—it seems to be preferable to start with something small and then evaluate and improve it, instead of creating a very elaborate, large project plan that might not fit the needs from practice. While our study showed that implementation could (or should) be viewed as a long-lasting, iterative process, this has not received much attention in scientific literature and implementation models (Boustani et al., 2018).

Implementation Requires Major Behaviour Change

Participating therapists gave a broad range of examples of behaviour that needed to change, such as the integration of VR in their treatment routines, acquiring and applying new skills, and remembering to use VR during their busy days. However, while the interviews showed that therapists had a positive attitude towards VR and expressed the intention to use it, these ‘implementation behaviours’ were often not carried out. This points to an intention-behaviour gap, which means that even though people may express an intention to change their behaviour, they might not take any actual action to do so (Sniehotta et al., 2005). Research in other domains has, e.g., shown that intention only predicts 46% of actual physical activity behaviour (Rhodes & de Bruijn, 2013), which implies that this could also be the case for implementation. An implication is that behaviour change of adopters is a large part of implementation. Despite its importance, the complexity of changing behaviour is often underestimated and underrepresented in implementation studies (Presseau et al., 2015). For future implementation projects, implementation science could draw from behavioural science to sufficiently address the required change in behaviour of adopters. In behavioural science, multi-modal interventions with evidence-based behaviour change techniques (BCTs) are developed to change intentions and accompanying behaviour (Bartholomew et al., 2006; Mohr et al., 2017; Sniehotta et al., 2005). To ensure a coherent approach, a set of BCTs to change implementation behaviour of adopters could be combined in a coherent implementation intervention (Eccles et al., 2005; Presseau et al., 2015). However, there is not much knowledge on how to systematically develop such implementation interventions (ICEBeRG, 2006). Intervention development frameworks used in behavioural science could be applied, such as Intervention Mapping (Bartholomew et al., 2006), the CeHRes Roadmap (van Gemert-Pijnen et al., 2011), or the Accelerated Creation-to-Sustainment (ACTS) model (Mohr et al., 2017). In line with the multi-level nature of implementation models such as the CFIR, these implementation interventions should not merely focus on individual behaviour change, but also factors related to the organization, intervention, and wider context (Damschroder et al., 2009; Greenhalgh et al., 2017; Kip et al., 2020a, 2020b). While there has been some work on behaviour change and implementation interventions, there is a need for more insight into how knowledge from behavioural sciences can be used to develop implementation interventions that target the behaviour of adopters in a suitable way (Patey et al., 2021; Presseau et al., 2015).

From Model to Practice

In this study, we used the domains of the CFIR to guide the implementation process. Translating this broad model to a specific plan appeared to be quite challenging due to the major shift from an abstract model to an implementation process. To bridge this gap, we used existing research on implementation factors for the use of an internet-based intervention in the same organization as an intermediate step (Kip et al., 2020a, 2020b). Consequently, to apply models such as the CFIR to practice, it seems that there first is a need for an overview of specific context- and/or technology-dependent implementation factors. These factors can be derived from earlier, comparable studies or systematic reviews, but also by collecting new data by means of for example interviews or focus groups with intended adopters. A potential avenue for future research is the exploration of more specific versions of implementation models that are tailored to settings, such as mental healthcare, occupational therapy, or hospital care. Besides tailoring the CFIR to a specific context, another challenge in this study was to translate factors into concrete implementation strategies in the implementation plan. Compared to research on for example implementation barriers, relatively few studies have been conducted on implementation strategies (Proctor et al., 2013). Some taxonomies have been created, but as is the case for the CFIR model, these remain quite abstract and it is not clear which types of strategies might work well in different types of contexts and for what type of factors (Baker et al., 2015; Waltz et al., 2015). To illustrate: some strategies might be very suitable for the implementation of a new software system in a commercial company but might not fit the implementation of VR in mental healthcare. This highlights the need for more reporting and reflecting on implementation strategies within specific domains, e.g. by following the guidelines for reporting by Proctor et al. (Proctor et al., 2013). In line with this, more knowledge on how to match implementation strategies to implementation factors should be generated (Powell et al., 2017). An option might be to use a similar approach as used in behavioural science, in which there is an overview in which BCTs are connected to behavioural determinants (Bartholomew et al., 2006). Consequently, there is a need for more domain-specific knowledge on how context-specific barriers can be connected to suitable to strategies.

Strengths and Limitations

A strength of this study is the systematic approach in setting up the implementation plan and conducting the formative evaluation interviews. By grounding the factors and objectives in earlier research and using the factors as deductive codes to guide the qualitative data analysis, a structured approach was used, in which frameworks, previous research and new data collection were combined. Additionally, as opposed to focusing on only one point of implementation processes such as identifying barriers—which often happens in implementation research (Kouijzer et al., 2023), we took on a broader perspective and also identified implementation objectives and strategies. A limitation of this case study is the relatively low number of participants in the pilot and interviews. Additionally, even though they all worked in different teams and specializations of mental healthcare, all participants were employed by the same organization. Furthermore, because this study focused on a small implementation pilot and due to the pandemic and accompanying measures, most therapists did not have much experience with using VR. This of course limits the generalizability of the current findings, but it is important to note that the main goal of this case study was not to create an exhaustive overview of implementation factors and strategies but to reflect on the use of implementation models, research, and new data in shaping an implementation process, thus putting the used methodology central. The outcomes of this study serve as the foundation for a more elaborate and exhaustive overview of implementation factors, objectives, and strategies. Future research is needed to develop a more comprehensive implementation plan, based on the outcomes of this study, using an iterative and participatory approach.

Another limitation is that it is not possible to draw conclusions on the effectiveness of this approach: because this study was not experimental and we did not conduct a summative evaluation of the implementation outcomes and, we cannot conclude if our systematic approach towards implementation planning resulted in better implementation outcomes than a ‘laissez-faire’ approach. Nevertheless, the lessons learned provide guidelines for future implementation endeavours and can serve as the foundation for future implementation research, in which for example an implementation intervention is evaluated on its effectiveness via an experimental design (Brown et al., 2017). A final limitation is that no interviews with other stakeholders such as patients and team managers were conducted, which might partly explain why the patient- and wider context categories were underrepresented. The main reason for this was that during the pilot, VR was not yet intensively used with many patients. This warrants the need for future research, in which the implementation process is evaluated with stakeholders from different levels, such as patients, team managers and technology developers, to do justice to the multi-level nature of implementation.

Conclusion

While our systematic approach resulted in a broad range of relevant factors and an implementation process that was evaluated quite positively by participants, the interviews showed that there were multiple points of improvement and underrepresented factors in the original implementation plan. Overall, it proved to be challenging to use abstract frameworks to shape specific implementation plans. Our case study highlighted the need for more guidelines on how to connect domain-specific barriers and facilitators to concrete implementation strategies that fit within a specific context. Based on our findings, we recommend an iterative, agile and systematic approach towards development of coherent, multi-level implementation interventions in which multiple evaluation cycles are needed to ensure that factors and strategies fit within a specific context. This approach can be guided by models and frameworks such as the CFIR, intervention mapping, BCTs, and agile science, but future research is necessary to determine if and how this should be done. Overall, there is a need for a more coherent, systematic approach to get from implementation factors to a multi-level, successful implementation program.