Using a realist lens to understand the Victorian Family Preservation and Reunification Response in the first year of implementation — Towards a better understanding of practice

Highlights

• Understanding what drives outcomes in family preservation programs is important.

• Realist methods applied to focus group data found important demi-regularities.

• Practitioners easily share their knowledge of what works for programs and families.

• Practitioners clearly articulate what leads to intended and unintended outcomes.

Abstract

Programs for families where children are at risk aim to develop the conditions for safety and care that is rewarding, loving, stable and secure. Understanding what leads to outcomes is less clear, and realist research methods are a useful in unpacking this. The Social Care Theorising Model informed the collection of focus group data from practitioners and team leaders in a family preservation and reunification program being scale up across the state of Victoria in Australia. Demi-regularities (patterns of a program’s function that are reasonably stable) were generated to provide a rapid understanding of practice knowledge to inform data-driven decision making. This study combined the methodology from two studies to balance the need for realist informed findings without time and resource intensive processes of program theory development. Twelve demi-regularities were found which were grouped into three socio-ecological themes of family, organisation and system. At the family level, practitioners described what enabled engagement, readiness, the development of a therapeutic relationship and the need for supported closure. The organisational level suggested the preconditions to engagement, the caseload needed to work intensively, management of staff stress and work needed to operationalise goals. The systems level largely described the essential role of the Child Protection Navigator and how they enable service delivery. This study provides an understanding of what works, for whom, when and why in an innovative family preservation and reunification program.

Introduction

Preservation and reunification programs use intensive and targeted service provision to provide support to vulnerable families. The overarching aim of these programs is to keep families safe and together, or create the conditions for reunification by improving the quality of relationships between parents and children and increasing parenting knowledge, skills and confidence to reduce child maltreatment and neglect (Bezeczky et al., 2020, Schweitzer et al., 2015). These effective programs are a widely used public health approach (Channa et al., 2012), however the necessary conditions and importance of context, including how these impact service delivery and ultimately outcomes is not well articulated. Indeed, understanding the social ‘work’ is stubbornly elusive (Ferguson et al., 2020). Realist research methods are a useful in unpacking the black-box of what is happening when the practice in a certain context, leads to an outcome that is either intended or not. This article explores the realist evaluation of an intensive family services program delivered by MacKillop Family Services across regional and metropolitan regions of Victoria, Australia.

Realist research is a form of theory-driven enquiry underpinned by the principles of realist philosophy. Comprising both realist evaluation and realist review (Pawson, Greenhalgh, Harvey, & Walshe, 2004; Pawson, Tilley, & Tilley, 1997), realist research recognises the complex social systems that interventions are delivered within, such as health and social care (Baginsky, Moriarty, & Manthorpe, 2019; Greenhalgh et al., 2009, Kazi et al., 2011, Tennant et al., 2020), and the multiple pathways from implementation to outcomes (Forster, Dalkin, Lhussier, Hodgson, & Carr, 2016). Recognising that programs will work for some people and in some contexts, but not others, realist evaluation shifts from asking ‘does this program work?’, which relies on capturing the average answer, to ‘what works, for whom, in what circumstances and to what extent’? It does this by exploring the underlying mechanisms that interact with features of the context to influence an observable outcome (Pawson et al., 1997). Mechanisms are a critical element in realist evaluation, and comprise the concrete resources offered by a program (for example, brokerage funds (money) allocated to families to purchase items that address immediate needs or facilitate the achievement of goals e.g., rent, food, childcare) combined with stakeholders (for example, frontline practitioners) change in reasoning in response. Mechanisms only activate under certain conditions, thus it is the interaction between contexts and mechanisms, that lead to the observed outcomes (either intended or unintended). The explanatory framework and main unit of analysis in realist evaluation are Context-Mechanism-Outcome (CMO) configurations that attempt to capture what happens within the system to connect inputs with outputs (Dalkin, Greenhalgh, Jones, Cunningham, & Lhussier, 2015).

There are many ways to undertake robust and rigorous realist evaluation. The Quality Standards for Realist Evaluation (Wong et al., 2016) do not propose standardised processes, but emphasise flexibility and transparency to iteratively improve understanding of the key contexts and mechanisms that contribute to how and why programs work. Typically, realist evaluation spans multiple phases, beginning with the development of a set of initial program theories (CMO configurations), drawing on the knowledge and expertise of program designers and staff, and existing program documentation. Broader literature can refine and validate the program theories and formulate middle-range theories. A middle-range theory is explanatory and may be used to explain specific parts of an intervention or program that can also be tested using data (Jagosh et al., 2014). The initial CMOs and middle-range theories inform data collection and analysis, usually utilising a mixed-methods approach that seeks quantitative and qualitative data. Based on this testing, CMOs are refined, forming the basis for a new cycle of realist evaluation (Forster et al., 2016, Goicolea et al., 2013). Using this approach, concepts from higher-order theories can inform program theories regarding a specific intervention (Westhorp, 2012), while the testing of program theories in different contexts can in turn refine higher-order theories (Shearn, Allmark, Piercy, & Hirst, 2017).

Demi-regularities are recurring forms of CMO configurations representing patterns of a program’s functionality with some predictability (Pawson et al., 2004). Often demi-regularities are searched for in realist synthesis, reviews and evaluations where multiple studies from independent sources have been conducted on the one topic but are less frequently sought in the analysis of qualitative data. Two recent studies have searched for demi-regularities within their qualitative data analysis. Mutschler et al (2018) used a two phased approach to understand how a Clubhouse model worked for psychosocial rehabilitation (Mutschler, Rouse, McShane, & Habal-Brosek, 2018). Phase 1 utilised a participatory approach throughout the research project to obtain data and refine findings. The authors conducted interviews and focus groups with staff and executives to understand what CMOs existed when the program was employed. These data were thematically analysed and later organised into the CMO format. Hypotheses about mechanisms and outcomes were formed based on literature (Rouse, Mutschler, McShane, & Habal-Brosek, 2017). Phase 2 involved program participants using visual analogue scales to validate the CMOs identified in Phase 1. Hierarchical regression analyses statistically determined what mechanisms predicted certain outcomes. Similarly, Gilmore et al.’s (2019) paper explicitly describes Phases 3–5 (prior phases previously published) of their process to uncover middle range theories from case studies (Gilmore, McAuliffe, Power, & Vallières, 2019). Phase 3 involved data preparation and coding of transcripts for observable CMOs. Phase 4 refined the initial program theories from earlier phases and refined them based on evidence. These studies found strong and evident patterns from the data that outlined functional aspects of the respective programs under review.

Realist approaches can be difficult to conceive, develop and apply leading to confusion when operationalising the method in practice (Dalkin et al., 2015, Forster et al., 2016). There exists varying and sometimes divergent understanding of contexts, mechanisms and outcomes, and realist scholars have discussed the challenge in disentangling concepts (Greenhalgh and Manzano, 2021, Shaw et al., 2018). Indeed, clear methodological information has been lacking (Gilmore et al., 2019) and individual interpretation of published standards has led to variation in application (Shearn, Allmark, Piercy, & Hirst, 2017). Further, the iterative and repeated approaches that realist research uses, does not lend itself to strict descriptions — providing both flexibility and vague boundaries (Greenhalgh, Wong, Westhorp, & Pawson, 2011). Interviewing is an important method to develop and test theories within realist-informed inquiry (O’Rourke, Abdulghani, Yelland, Newton, & Shafiei, 2022), however it relies on tailored interview schedules, the use of ‘teacher-learner cycles’ (where the interviewer presents theories to participants who are invited to comment, confirm, deny or refine the theory), and various interview phases (Manzano, 2016). This suggests that a team with high-level research expertise, time and financial resources, who can undertake this work is needed, which effectively excludes organisations with limited resources. As such a realist approach that can identify CMOs in existing qualitative data, derived without the need for realist ‘style’ interview schedules (Manzano, 2016, Pawson, 1996) may offer an alternative method for smaller research teams in real-world practice settings to begin the type of mid-level theorising that will support deeper knowledge and understanding of the program and its processes.

In 2020, a new program called the Family Preservation and Reunification Response (hereafter, the Response) that aimed to develop and maintain strong resilient families was launched (Department of Families Fairness and Housing, 2022). Service providers delivered evidence-based or -informed practices and/or programs which were designed to be targeted, responsive and intensive for families in crisis and/or at risk of having their children removed for reasons of abuse or neglect. Recipients received approximately 200 hours of intensive family support by a key practitioner, characterised as three home visits a week for an hour (on average) and other support work as necessary. The families worked towards achieving their goals and strengthening relationships with their children, with brokerage funds accelerating this achievement. For example, the goal of ensuring the safety of the child may be facilitated by funding childcare at the start of the program to immediately decrease the child’s exposure to family violence. An additional 40 hours of step-down support were provided as families move towards closure and referrals to other programs were made as needed. See theory of change in Fig. 1.

MacKillop Family Services (hereafter, MacKillop) was chosen as one of many providers across regional and metropolitan Victoria Australia to deliver the Response, which at the time of this study was in nine geographically diverse areas. MacKillop were the only community organisation also conducting a longitudinal effectiveness and implementation study (the program, measures and planning is described in depth elsewhere (Morris, Savaglio, et al., 2021)). MacKillop’s program provided training, supervision, and infrastructure to practitioners as they engaged with families intensively. Team leaders guided practitioners and worked with child protection staff as they accepted new families into MacKillop’s program. Multisystemic Therapy- Psychiatric (MST-Psych)(Henggeler & Schaeffer, 2016), an evidence-based program, was offer to metropolitan families with children who are 10 years or older, and younger children received a modified Cradle to Kinder program (O’Donnell et al., 2020). For more information, please see (Morris, Savaglio, et al., 2021). As the Response was recently initiated, a timely and accurate description of what does and does not work, for whom, when, how and why, was needed for program modification and adaption as the Response was scaled up across the state.

This study utilised the Social Care Theorising Model (SCTM) as the framework through which data were obtained (Morris, Blewitt, O'Connor, & Skouteris, 2021). The SCTM focuses on the theorising of frontline practitioners and the mid-range theories they develop using everyday interactions, tacit knowledge, and program theory. Higher order theories are developed through supervision, coaching, and the practice expertise of team leaders and management thereby building capacity in frontline practitioners and strengthening program delivery with fidelity. A PhD qualified researcher in residence who is independent of the program provider, mobilises the latest academic knowledge, fosters data driven practice, and links higher order theories to grand theories. Grand theories are abstract depictions of the social world which cannot be measured but offer an overarching theoretical view (Morris, Blewitt, et al., 2021). As a critical enabler, the researcher in residence offers timely findings and analysis, supports the implementation of new practices and skills and a drives rapid, iterative processes of knowledge development, mobilisation, and translation. The SCTM acknowledges the tacit knowledge that practitioners employ in combination with their understanding of program theory and daily interactions with families. Mobilising this knowledge is critical to understand what works for whom, when, and why. Indeed, it allows for the rapid generation of new knowledge that can be disseminated back into the system for verification and use. Organisational decision making can be informed by data that is derived from the frontline which has been synthesised by the researcher in residence.

We sought to better understand the context and mechanisms that contribute to outcomes of the Response by using a realist research lens to analyse qualitative focus group data. Undertaking this pragmatic research supports the generation of new knowledge about implementation to support decision making.