Measuring the work environment among healthcare professionals: Validation of the Dutch version of the Culture of Care Barometer

Susanne Maassen; Catharina van Oostveen; Anne Marie Weggelaar; Anne Marie Rafferty; Marieke Zegers; Hester Vermeulen

doi:10.1371/journal.pone.0298391

Abstract

Objectives

A positive work environment (WE) is paramount for healthcare employees to provide good quality care. To stimulate a positive work environment, employees’ perceptions of the work environment need to be assessed. This study aimed to assess the reliability and validity of the Dutch version of the Culture of Care Barometer (CoCB-NL) survey in hospitals.

Methods

This longitudinal validation study explored content validity, structural validity, internal consistency, hypothesis testing for construct validity, and responsiveness. The study was conducted at seven departments in two Dutch university hospitals. The departments were included based on their managers’ motivation to better understand their employees’ perception of their WE. All employees of participating departments were invited to complete the survey (n = 1,730).

Results

The response rate was 63.2%. The content of the CoCB-NL was considered relevant and accessible by the respondents. Two factor models were found. First, confirmative factor analysis of the original four-factor structure showed an acceptable fit (X² 2006.49; df 399; p = <0.001; comparative fit index [CFI] 0.82; Tucker-Lewis index [TLI] 0.80; root mean square error of approximation [RMSEA] 0.09). Second, explanatory factor analysis revealed a five-factor model including ‘organizational support’, ‘leadership’, ‘collegiality and teamwork’, ‘relationship with manager’, and ‘employee influence and development’. This model was confirmed and showed a better fit (X² 1552.93; df 395; p = < 0.00; CFI 0.87; TLI 0.86; RMSEA 0.07). Twelve out of eighteen hypotheses were confirmed. Responsiveness was assumed between the measurements.

Conclusions

The CoCB-NL is a valid and reliable instrument for identifying areas needing improvement in the WE. Furthermore, the CoCB-NL appears to be responsive and therefore useful for longitudinal evaluations of healthcare employees’ work environments.

Citation: Maassen S, van Oostveen C, Weggelaar AM, Rafferty AM, Zegers M, Vermeulen H (2024) Measuring the work environment among healthcare professionals: Validation of the Dutch version of the Culture of Care Barometer. PLoS ONE 19(2): e0298391. https://doi.org/10.1371/journal.pone.0298391

Editor: Elif Ulutaş Deniz, Ataturk University, Faculty of Pharmacy, TURKEY

Received: May 17, 2023; Accepted: January 23, 2024; Published: February 29, 2024

Copyright: © 2024 Maassen et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: Data cannot be shared publicly because of Dutch privacy legislation. Data are available from the TRANZO Tilburg University Institutional Data Access Committee for researchers who meet the criteria for access to confidential data. Request can be submitted via: https://dataverse.nl/dataset.xhtml?persistentId=doi:10.34894/WCX0WV.

Funding: The author(s) received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

A positive work environment (WE) is paramount for providing high-quality and safe patient care and for attracting and maintaining engaged healthcare professionals [1–7] The WE is the internal setting of the organization where employees work [8] and consists of the physical environment, culture, social climate, and job context [9]. A positive WE is characterized by respect, support, and trust between employees at all levels; effective collaboration and communication; recognition for good work; support from management; and a healthy work place [4, 9–11].

Healthcare organizations have become more aware of how the WE can influence patient and employee outcome measures and are committed to improving their WE [4, 12]. To better understand employees’ experiences of their WE, surveys have been developed to gain a systematic insight into the WE. Periodic and valid measurements of the WE helps management learn from best practices and to understand which WE components need improving [4, 13–15]. However, the WE is multidimensional so is not easy to measure [9]. In addition, WEs differ within healthcare organizations and each department is likely to have its own characteristic WE [16, 17]. Achieving a positive WE is not just up to the members of one profession, but a challenge for the multidisciplinary team [6, 12]. This means that healthcare organizations need to use an instrument that encompasses important WE features from the perspective of all employees to measure the WE. This questionnaire should also be sensitive enough to differentiate between team and departmental responses and should be succinct and easy to understand by everyone [14].

A literature review [14] and Delphi study [9] found that the Culture of Care Barometer (CoCB) [15] is a complete, succinct, and applicable survey for all employees within healthcare organizations. The CoCB is designed to assess the WE within healthcare organizations. Rafferty et al. [15] described a positive WE as ‘a caring culture’; a place where employees feel valued, respected, and supported and where relationships between employees, management, teams, and departments are good and where concerns can be discussed without fear of blame. The CoCB is a diagnostic and dialogic tool comprising 30 positively formulated items on four factors (organizational values, team support, relationships with colleagues, and job constraints) and one open question (what, if any, action needs to be taken to improve the culture of care environment where you work?) [15].

The validity and reliability of measurement instruments is determined by multiple measurement properties, each of which requires a different study design (Fig 1) [18, 19]. Rafferty et al. [15] determined content validity of the CoCB by asking healthcare professionals to assess the relevance and clarity of the items. Based on this, Rafferty et al. [15] defined the structural validity and internal consistency of the CoCB using exploratory factor analysis (EFA). They found a four-factor structure with sufficient factor loadings > 0.40 [15, 18, 20]. These four factors were organizational values (12 items, 0.84–0.40), team support (11 items, 0.87–0.40), relationships with colleagues (4 items, 0.81–0.56), and job constraints (3 items, 0.79–0.41) [15]. The factors’ internal consistency was proven by Cronbach’s alphas between 0.70 and 0.93 [15].

Download:

Fig 1. Overview measurement properties based on COSMIN guideline.

In italics is indicated which studies have examined this measurement property.

https://doi.org/10.1371/journal.pone.0298391.g001

The usability of a factor model found in an EFA is rarely absolute and finding a single model in a single data set does not prove that other plausible models do not also fit [20]. Therefore, EFA results need to be further evaluated by confirmatory factor analysis (CFA) on new data, to see whether other plausible models occur. Recently, Ying, Fitzpatrick [21] performed a CFA to validate the Chinese version of the CoCB. This CFA showed that the four-factor structure had acceptable fit (Chi-square [χ2] = 5975.22, 399 df, comparative fit index [CFI] 1.00, Tucker-Lewis index [TLI] 1.00, root mean square error of approximation [RMSEA] 0.07) and good internal consistency with Cronbach’s alphas between 0.77 and 0.96 [21]. However, the authors did not examine alternative factor models. Additionally, Ying, Fitzpatrick [21] examined construct validity by testing hypotheses based on the relationships between the CoCB and job satisfaction and organizational culture. They found evidence of good content, structural and construct validity, and internal consistency between the English CoCB and the Chinese CoCB [15, 21]. However, the reliability and validity of the Dutch version (the CoCB-NL) has not yet been examined. Furthermore, no information is known about the responsiveness of the CoCB. The aim of the present study was to assess the reliability and validity of the CoCB-NL to determine whether broad uptake of the instrument should be recommended. For cross-cultural validation, the content and structural validity and internal consistency of the CoCB-NL were also examined. The instrument was further validated by assessing convergent validity, discriminant validity, and responsiveness.

Methods

Design

This validation study contained two parts and addressed five measurement properties:

Translation and cultural adaptation by determining the content validity of the translated CoCB items in the Dutch hospital context.
A longitudinal survey to confirm structural validity and internal consistency, testing hypotheses to determine the convergent validity and discriminant validity, and assessment of responsiveness of the CoCB-NL.

The study was developed and reported according to the Consensus-Based Standards for the Selection of Health Measurement Instruments (COSMIN) study design checklist [19] and reporting guideline [22].

Setting and participants

The study was conducted in two Dutch hospitals that are part of the researchers’ network and purposively approached hospitals that were planning to improve their work environment: the Radboud University Medical Center and the Erasmus University Medical Center. The survey was distributed to all employees working in the following departments at these hospitals: intensive care unit (ICU) (2), pediatric surgery (1), oncology (1), pharmacy (1), clinical chemistry (1), and endoscopy (1). The departments were included based on their managers’ motivation to better understand their employees’ perception of the WE.

Part 1: Translation and content validity

The CoCB was translated into Dutch according to the translation procedure described by Sousa and Rojjanasrirat [23]. Three Dutch healthcare professionals (nurse, physiotherapist and researcher) independently provided the forward translation. These versions were compared by the research team (SM, AMW, CvO, HV) and checked for semantic and cross-cultural equivalence. Context-specific terminology such as ’trust’ was changed to ’organization’. This first version of the instrument was back-translated into English by three independent native UK English speakers and was compared with the original UK version [15]. Discrepancies were discussed in the research team and resolved in cooperation with one of the co-authors and co-developer of the CoCB (AMR).

To determine content validity, 50 hospital employees, working in various positions in one of the hospitals affiliated with the researchers, were asked to assess the relevance and accessibility of the CoCB-NL items using a dichotomous yes/no scale. The threshold for content validity at the item level was set at an 80% ‘yes’ score [23]. Thirty-seven respondents (response rate 74%) (S1 Table) completed the content validity questionnaire and the threshold was reached for 26/30 items with scores between 89.2% and 100% on relevance and comprehensibility (Table 1). The four items that did not reach the threshold were: ‘Unacceptable behavior is consistently tackled’ (78.4%), ‘The organization values the service we provide’ (76.3%), ‘I have the resources I need to do a good job’ (71.1%), and ‘Managers at the top of the organization know how things really are’(62.2%). Considering the expected relevance of these items for practice, we decided to further evaluate them in the second part of the study. Footnotes were added to the questionnaire to clarify what was meant by ‘the organization’ and who was meant by ‘managers at the top’.

Download:

Table 1. Content validity assessment on comprehensibility.

https://doi.org/10.1371/journal.pone.0298391.t001

Part II: Survey

The survey contained the CoCB-NL to which the nine-item version of the Utrecht Work Engagement scale (UWES-9) [24] and the team and safety climate subscales of the Safety Attitudes Questionnaire (SAQ-NL) [25, 26] were also added for hypothesis testing [18]. The UWES-9 and SAQ-NL are psychometrically sound instruments [24–26]. Both the CoCB-NL and SAQ-NL use a five-point Likert scale ranging from ’do not agree at all’ to ’totally agree’ [15, 25, 26]. The UWES-9 is rated on a seven-point scale ranging from ‘never’ to ‘always’ [24]. Being latent variables, the scores are calculated as subscales in the CoCB-NL and SAQ-NL and as scales in the UWES-9. The survey was sent out twice, at an interval of 9 to 12 months, in line with the continuous improvement cycle of the respective department. Data from the first survey were used to assess the structural validity, internal consistency, convergent validity, and discriminant validity. Data from the second survey were used to assess responsiveness.

To minimize the measurement burden, administration of the UWES-9 and SAQ-NL subscales was optional. All departments chose to extend the survey with the SAQ safety climate subscale. One ICU also added the team climate subscale. The pharmacy, clinical chemistry, pediatric surgery, and oncology departments volunteered to measure engagement with the UWES-9.

All employees were informed about the survey by their manager before receiving an invitation to participate by email from the researchers. Data were collected for six weeks via the online survey tool LimeSurvey^®. Automatic reminders were sent after two and four weeks to those participants who had not yet completed the survey. All questions on the CoCB-NL, UWES-9, and SAQ-NL subscales were obligatory. Questions about personal characteristics like age group, gender, type of contract, mother language, and profession were voluntary as required by the Dutch privacy legislation. Data collection took place in 2018 and 2019.

Ethical considerations.

Ethical approval was waived by the Medical Ethic Committee Brabant (NW2023-32) since this study did not include patients and did not affect the participants’ wellbeing. The data collection and data storage plan was approved by the local GDPR committee of the Erasmus Medical Center Rotterdam. Participation to the study was anonymous and voluntary. Participants were informed about this both in the announcement e-mail and at the beginning of the questionnaire itself (page 1). By proceeding to the second page with the substantive questions, participants gave consent. Technically, a strict separation was made between the LimeSurvey^® input file containing personal data and the output file, which did not include this data. Researchers only had access to the output file.

Structural validity and internal consistency.

Structural validity and internal consistency were assessed in the CoCB-NL in two stages. First, an EFA was performed to see if other plausible factor models exist. Second, both the UK factor model and other potential models were evaluated with a CFA. Internal consistency was calculated in all models.

Convergent validity, discriminant validity, and responsiveness.

In total, 18 hypotheses were formulated for testing. Fifteen hypotheses on both the item level (1–9) and subscale level (10–15) were formulated to assess convergent validity (Table 2). These hypotheses were based on associations between a poor perceived WE and low employee engagement [3, 17] or poor safety climate [4, 7, 27] and an expected association between the WE and team climate [27]. Hypotheses to test discriminant validity were based on existing differences in positions and secondary working conditions between professional groups [18]. For example, the influence of nurses on hospital policy was perceived as less than that of physicians [28]. Three hypotheses (16–18) were formulated based on the differences in position and secondary working conditions between physicians, nurses, pharmacists, and laboratory staff.

Download:

Table 2. Hypothesis testing for construct validity.

https://doi.org/10.1371/journal.pone.0298391.t002

The responsiveness of the CoCB-NL for detecting change over time was explored by comparing the development and trends in the CoCB-NL, UWES-9, and SAQ-NL safety climate and team climate between the first and second measurement on the department level.

Analysis.

Data analysis was performed using IBM SPSS Statistics version 25 and IBM SPSS AMOS version 25. For optimal use of SPSS AMOS, only complete cases were analyzed. The minimum sample size was set at seven times the number of items [19, 29]. Data were analyzed on the total sample and department level. For each CoCB-NL item normal distribution as a prerequisite for further analysis was calculated (S2 Table). EFA was performed without a predefined number of factors using principal component analysis with a varimax rotation method, including Kaiser normalization. Factor loadings above 0.4 were considered acceptable [30, 31]. The CFA model fit was assessed by multiple fit indices including Χ², degrees of freedom (df), CFI, TLI, RMSEA, and the upper and lower 90% confidence interval (CI) of the RMSEA. These indices comprehensively evaluate model fit, with CFI and TLI values >0.95 and RMSEA values <0.06 indicating excellent model fit [32–35]. For an instrument with 30 items, the df should be 405 minus the number of factors m (m × (m − 1)/2) and the ratio of X² to df should be ≤ 2 [32, 33, 35]. No post hoc modifications were applied in the CFA [33, 35]. Internal consistency of the factors in both models was assessed with Cronbach’s α coefficient and good internal consistency was indicated by α values between 0.70 and 0.95 [18, 34].

The hypotheses for convergent validity of CoCB-NL items were tested using Spearman’s correlation coefficients for ordinal data. The hypotheses on CoCB-NL subscales were tested using Pearson correlation coefficients for continuous data. Hypotheses were accepted or rejected based on cut-off values defined in the COSMIN guidelines for construct validity assessment [18, 34]. All but one hypothesis addressed related but dissimilar constructs; therefore, correlation coefficients of 0.30–0.50 between the CoCB-NL and UWES-9 or SAQ-NL were considered acceptable [18, 34]. Only hypothesis 7 compares two nearly equal items, so we expected a strong correlation (>0.50). The hypotheses for discriminant validity were tested using a Student t-test for independent samples with a 95% CI. A p-value of <0.05 indicated a significant difference between groups. The CoCB-NL’s ability to detect change over time at the department level was analyzed using a Student t-test for independent samples with a 95% CI. The trend of each outcome measure was visually analyzed as a bar chart.

Results

All employees of the participating departments (n = 1,730) were invited to participate in the study. Of these, 1,094 (63.2%) respondents completed survey 1 and 1,062 of these were complete cases (Table 3). Most respondents were female (71.9%) and were equally distributed among age groups and types of contracts. Eight professional groups participated in the survey; nurses were the largest group (47.9%), followed by physicians (18.5%). Survey 2 was completed by 590 respondents (48.4%).

Download:

Table 3. Sample characteristics.

https://doi.org/10.1371/journal.pone.0298391.t003

Structural validity and internal consistency

The CFA of the original CoCB-based factor model had factor correlations between 0.63 and 0.84. The standardized item loadings of the factors ranged from 0.43 to 0.85 (Table 4). The model fit indices did not reach the thresholds for an excellent fit (X² 2006.49; df 399; p = <0.00; CFI 0.82; TLI 0.80; RMSEA 0.09; 90% CI 0.08–0.09). Three out of four factors showed internal consistency with Cronbach’s α coefficients of 0.76–0.91. For the factor ‘job constraint’ a Cronbach’s α coefficient of 0.51 was found.

Download:

Table 4. Confirmative factor analysis and internal consistency of the four-factor model.

https://doi.org/10.1371/journal.pone.0298391.t004

The EFA identified a five-factor model (Table 5). The first factor contained six items that emerged from the original factors ‘organizational values’ and ‘job constraint’ (items 1, 3, 4, 6, 7, and 26; loadings 0.47–0.67). The items reflected the respondents’ experience of organizational support and their valuation of the organization so we named this factor ‘organizational support’. The second factor included seven items concerning ‘leadership’ (items 10, 12, 13, 15, and 27–29; loadings 0.46–0.66). These items were a combination of items from three factors in the original model. This was also the case for the third factor, which included six items related to ‘collegiality and teamwork’ (2, 14, 16, 17, 19, and 20; loadings 0.54–0.72). The fourth factor included five items about the ‘relationship with manager’ and originated from the original factor ‘team support’ (5, 8, 11, 21, and 30; loadings 0.63–0.77). The last factor contained six items on ‘employee influence and development (9, 18, 22–25; loadings 0.47–0.77). These items emerged from the factors ‘organizational values’ and ‘team support’ in the original model. The correlations between the five factors ranged from 0.68 to 0.88. The model fit indices were X² 1552.93, df 395, p = < 0.00, CFI 0.87, TLI 0.86, and RMSEA 0.07 (90% CI 0.07–0.08). Although the CFI, TLI, and RSMEA did not reach the threshold for an excellent model fit, the values were more favorable than those of the four-factor model. All factors showed internal consistency with Cronbach’s α coefficients varying between 0.79 and 0.88.

Download:

Table 5. Exploratory and confirmative factor analysis and internal consistency of the five-factor model.

https://doi.org/10.1371/journal.pone.0298391.t005

Convergent validity, discriminant validity, and responsiveness

For convergent validity, six out of nine hypotheses were accepted on the item level (2, 3, 4, 6, 7, and 9) based on strong correlation coefficients (Table 2). The hypotheses 2, 3, 4, 6, and 9 showed moderate coefficients between r 0.30 (6) and r 0.41 (3). Hypothesis 7 revealed the expected strong correlation (r 0.57). The remaining three hypotheses showed weak correlation coefficients on the item level (hypotheses 1, 5 and 8; r 0.11 –r 0.25) and were therefore rejected. Three out of six factor-level hypotheses were accepted (13, 14, and 15 out of 10–15). The hypotheses concerning work engagement (UWES-9) and CoCB-NL (13–15) had moderate correlation coefficients of r 0.41 to r 0.46 and were thus accepted. A strong correlation coefficient was observed between safety climate and the CoCB-NL factors ‘relationship with manager’ and ‘leadership’ (r 0.53 and r 0.64) as well as between team climate and the CoCB-NL factor ‘collegiality and teamwork’ (r 0.54) instead of the expected moderate correlation, so these hypotheses were rejected.

All three hypotheses for discriminant validity (16–18) were accepted based on a significant Student t-test (p <0.01). The mean scores for the factor ‘employee influence and development’ ranged from 3.45 (physicians) to 2.81 (laboratory staff), and differed between professional groups (p <0.01).

Four departments completed two measurements during this study; n = 792 participants responded to survey 1 and n = 569 participants responded to survey 2. Both groups were comparable with regard to department, age group, and having Dutch as the first language (Table 3). The scores ranged from 2.93 (employee influence and development) to 4.14 (collegiality and teamwork) for survey 1 and 3.00 (leadership) to 4.17 (employee influence and development) for survey 2.

Responsiveness was assumed because the bar charts presenting outcome measures between surveys 1 and 2 showed similar patterns in all departments (Fig 2). There was a significant change in ‘organizational support’ (department A p = 0.042 and department C p = 0.026) and ‘safety climate’ (department D p = 0.045) between survey 1 and 2.

Download:

Fig 2. Bar charts on department level with difference between survey 1 (T0) and 2 (T1).

* = a significant change between T0 and T1 with p < 0.05 (95% CI).

https://doi.org/10.1371/journal.pone.0298391.g002

Discussion

This study aimed to assess the cross-cultural validity of the CoCB in Dutch hospitals and to psychometrically validate the instrument. The content of the CoCB-NL was considered relevant and accessible by the respondents. The original four-factor model, including ‘organizational values’, ‘team support’, ‘relationships with colleagues’, and ‘job constraints’, and the new five-factor model, including ‘organizational support’, ‘leadership’, ‘collegiality and teamwork’, ‘relationship with manager’, and ‘employee influence and development’, were found to be sufficiently valid and reliable for use in Dutch hospitals. All hypotheses for discriminant validity and more than half of the hypotheses for convergent validity were confirmed, which provided satisfactory construct validity. Furthermore, the CoCB-NL appears to be responsive in detecting changes over time so is a useful instrument for longitudinal evaluations of the WE.

Reflections on structural validity and internal consistency

CFA of the four-factor model was performed in accordance with the original factor structure reported by Rafferty et al. [15]. This model showed an acceptable fit in our Dutch sample; however, this fit was not as strong as that found for the Chinese version of the CoCB [21]. We performed an EFA to check for other plausible models as recommended by several authors [20, 32, 33, 35] and found a five-factor model. The CFA of this model showed a better fit in our Dutch sample. These differences between the four- and five-factor structures can be explained by the context in which the validation studies were conducted and the level at which the data were analyzed. The original four-factor instrument was developed in the UK and measures the WE at the trust, management, and ward level [15]. In the UK, hospital trusts can comprise multiple hospitals, such as the five hospitals making up Barts Health in London [36]. In contrast, hospitals in the Netherlands are independent entities. Therefore, we translated the original instrument into one that can measure the WE in a single organization, as was also done in the Chinese translation [21]. This resulted in factors on organizational foundations like ‘leadership’ and ‘employee influence and development’ in the Dutch version. These factors have been defined as essential elements of the WE [9].

Based on the psychometrics, the five-factor model is preferable in the Dutch context whereas the four-factor structure is better for international benchmarking. This justifies our decision not to drop or adjust any of the original CoCB items, although item 11 (‘I know who my line manager is’) does qualify for adjustment. The weak presentation of this item in both models can be explained by the lack of variation in the responses (95.8% agreement). If future research points out that this item remains weak in other language versions of the CoCB, dropping this item could be considered.

Reflections on convergent validity, discriminant validity, and responsiveness

In total, 67% (12 out of 18) of the tested hypotheses for construct validity were confirmed. Three hypotheses were rejected because contrary to the expected moderate correlations, strong correlations were found between factors of the CoCB-NL and the SAQ-NL. This is commensurate findings of Brubakk et al. [37] who found that positive work environments associate with a positive safety climate and favorable patient safety outcomes. The remaining three hypotheses were rejected because 1) support from managers did not appear to be a prerequisite for the support of colleagues or other personnel, and 2) working as a well-coordinated team did not necessarily mean being part of a well-managed team.

Overall, the CoCB-NL had moderate to high scores. Participants scored the organizational support factor evenly; however, there was variation in department specific factors. This is important for two reasons. First, because participants generally agreed with the positively formulated items used to score their WE. Second, because the variation in scores demonstrated the power to detect best and bad practices, which created learning opportunities among departments [16, 25].

Implications for practice and future research

These findings provide first insights into the WE. Now, further investigation on how to improve the WE is needed. Huebner and Zacher [38] found that active follow-up discussions and action planning after employee surveys benefitted improvement of the WE. The open question in the CoCB-NL could be a useful place to start such active follow-up discussions [39]. This should be part of continuous efforts to improve the WE since a positive WE is a cornerstone for high-performance hospitals [4, 10]. Our finding that the CoCB-NL is responsive indicates that it can also be used for long-term evaluation of WE improvement.

Limitations

This study has some limitations. First, although substantial effort was made to validate the instrument by assessing five measurement properties, the instrument’s measurement error is still unknown (Fig 1). Establishing measurement error requires two measurements in the same group under stable conditions [19]. These conditions are difficult to achieve because employees’ perceptions of the WE is influenced by multiple factors, such as staffing ratios [1] or a pandemic [40]. Second, responsiveness was determined on the team level. Although determining responsiveness on the individual employee level may have given stronger evidence [41], the current results show promising responsiveness. Future studies may continue to measure responsiveness at the team level to ensure the privacy and safety of respondents, which is a requirement of these surveys. Third, there was an interval of 9–12 months between survey 1 and survey 2, which might not have been long enough to significantly change the participants’ experiences of their WE. However, we did observe a significant difference in one factor among the three departments. Fourth, data were collected in two large-scale university hospitals and therefore may not be generalizable to other type of institutions. These large organizations are characterized by an extensive management line, politics, and hierarchy. This means our respondents could distinguish between the ‘top of the organization’ and the ‘line manager’, as requested in several items. This may be more difficult for respondents from smaller organizations.

Conclusions & recommendations

This study demonstrated that the CoCB-NL is sufficiently valid and reliable for assessing employees’ perceptions of their WE. Both a four-factor and five-factor model showed acceptable fit. We recommend using the five-factor model in the Dutch setting based on our findings and the four-factor model in the international setting based on previous validations in two other languages.

A positive WE is paramount for good patient outcomes and for attracting and keeping healthcare staff. Therefore, more research is needed to continuously improve the WE. Continuous improvement of the WE should start with discussions based on the results and open question in the CoCB-NL.

Supporting information

S1 Table. Sample characteristics of content validity on comprehensibility.

https://doi.org/10.1371/journal.pone.0298391.s001

(PDF)

S2 Table. CoCB-NL item normal distribution.

https://doi.org/10.1371/journal.pone.0298391.s002

(XLSX)

S3 Table. Dutch language version of the CoCB-NL.

https://doi.org/10.1371/journal.pone.0298391.s003

(XLSX)

Acknowledgments

The authors thank Hilco van Elten, Erasmus School of Health Policy and Management, Rotterdam, The Netherlands, for his assistance with the confirmative factor analysis in SPSS AMOS. We also thank all participating teams and their management for their willingness and enthusiasm to participate.

References

1. Aiken LH, Sermeus W, Van den Heede K, Sloane DM, Busse R, McKee M, et al. Patient safety, satisfaction, and quality of hospital care: cross sectional surveys of nurses and patients in 12 countries in Europe and the United States. Bmj-Brit Med J. 2012;344. ARTN e1717 pmid:22434089
- View Article
- PubMed/NCBI
- Google Scholar
2. Mathisen J, Nguyen TL, Jense JH, Rugulies R, Rod NH. Reducing employee turnover in hospitals: estimating the effects of hypothetical improvements in the psychosocial work environment. Scand J Work Environ Health. 2021;47(6):456–65. pmid:34052852.
- View Article
- PubMed/NCBI
- Google Scholar
3. Slatten T, Lien G, Mutonyi BR. Precursors and outcomes of work engagement among nursing professionals-a cross-sectional study. BMC Health Serv Res. 2022;22(1):21. Epub 2022/01/06. pmid:34983510.
- View Article
- PubMed/NCBI
- Google Scholar
4. Taylor N, Clay-Williams R, Hogden E, Braithwaite J, Groene O. High performing hospitals: a qualitative systematic review of associated factors and practical strategies for improvement. BMC Health Services Research. 2015;15(1):244. pmid:26104760
- View Article
- PubMed/NCBI
- Google Scholar
5. Van Bogaert P, Clarke S, Willems R, Mondelaers M. Staff engagement as a target for managing work environments in psychiatric hospitals: implications for workforce stability and quality of care. J Clin Nurs. 2013;22(11–12):1717–28. pmid:23216757.
- View Article
- PubMed/NCBI
- Google Scholar
6. Wan Q, Li Z, Zhou W, Shang S. Effects of work environment and job characteristics on the turnover intention of experienced nurses: The mediating role of work engagement. J Adv Nurs. 2018;74(6):1332–41. pmid:29350781.
- View Article
- PubMed/NCBI
- Google Scholar
7. Wei H, Sewell KA, Woody G, Rose MA. The state of the science of nurse work environments in the United States: A systematic review. Int J Nurs Sci. 2018;5(3):287–300. pmid:31406839.
- View Article
- PubMed/NCBI
- Google Scholar
8. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50. pmid:19664226
- View Article
- PubMed/NCBI
- Google Scholar
9. Maassen SM, van Oostveen C, Vermeulen H, Weggelaar AM. Defining a positive work environment for hospital healthcare professionals: A Delphi study. Plos One. 2021;16(2):e0247530. pmid:33630923.
- View Article
- PubMed/NCBI
- Google Scholar
10. Sutcliffe KM. High reliability organizations (HROs). Best Pract Res Clin Anaesthesiol. 2011;25(2):133–44. pmid:21550539.
- View Article
- PubMed/NCBI
- Google Scholar
11. Schmutz JB, Meier LL, Manser T. How effective is teamwork really? The relationship between teamwork and performance in healthcare teams: a systematic review and meta-analysis. BMJ Open. 2019;9(9):e028280. pmid:31515415
- View Article
- PubMed/NCBI
- Google Scholar
12. Sikka R, Morath u M, Leape L. The Quadruple Aim: care, health, cost and meaning in work. BMJ Quality & Safety. 2015;24(10):608–10. pmid:26038586
- View Article
- PubMed/NCBI
- Google Scholar
13. Grant S, Davidson J, Manges K, Dermenchyan A, Wilson E, Dowdell E. Creating Healthful Work Environments to Deliver on the Quadruple Aim: A Call to Action. JONA: The Journal of Nursing Administration. 2020;50(6):314–21. pmid:32427662
- View Article
- PubMed/NCBI
- Google Scholar
14. Maassen SM, Weggelaar Jansen A, Brekelmans G, Vermeulen H, van Oostveen CJ. Psychometric evaluation of instruments measuring the work environment of healthcare professionals in hospitals: a systematic literature review. Int J Qual Health Care. 2020;32(8):545–57. pmid:32648902.
- View Article
- PubMed/NCBI
- Google Scholar
15. Rafferty AM, Philippou J, Fitzpatrick JM, Pike G, Ball J. Development and testing of the ’Culture of Care Barometer’ (CoCB) in healthcare organisations: a mixed methods study. BMJ Open. 2017;7(8):e016677. Epub 2017/08/20. pmid:28821526.
- View Article
- PubMed/NCBI
- Google Scholar
16. Cho SH, Mark BA, Yun SC, June KJ. Differences in intensive care unit work environments among and within hospitals using subscales and a composite measure of the Revised Nursing Work Index. J Adv Nurs. 2011;67(12):2637–48. pmid:21645043.
- View Article
- PubMed/NCBI
- Google Scholar
17. Rugulies R. What is a psychosocial work environment? Scandinavian Journal of Work, Environment & Health. 2019;(1):1–6. pmid:30643919
- View Article
- PubMed/NCBI
- Google Scholar
18. Mokkink LB, de Vet HCW, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, et al. COSMIN Risk of Bias checklist for systematic reviews of Patient-Reported Outcome Measures. Qual Life Res. 2018;27(5):1171–9. pmid:29260445.
- View Article
- PubMed/NCBI
- Google Scholar
19. Mokkink LB, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, de Vet HCW, et al. COSMIN Study Design checklist for Patient-reported outcome measurement instruments2019 08-25-2023:[1–32 pp.]. https://www.cosmin.nl/wp-content/uploads/COSMIN-study-designing-checklist_final.pdf#.
20. Henson RK, Roberts JK. Use of Exploratory Factor Analysis in Published Research: Common Errors and Some Comment on Improved Practice. Educational and Psychological Measurement. 2006;66(3):393–416.
- View Article
- Google Scholar
21. Ying L, Fitzpatrick JM, Philippou J, Zhang Y, Murrells T, Rafferty AM. Front-line staff perspectives on a caring culture in Chinese hospitals: Validation of a Chinese version of the Culture of Care Barometer. J Nurs Manag. 2022;30(6):2093–102. pmid:35506434.
- View Article
- PubMed/NCBI
- Google Scholar
22. Gagnier JJ, Lai J, Mokkink LB, Terwee CB. COSMIN reporting guideline for studies on measurement properties of patient-reported outcome measures. Qual Life Res. 2021;30(8):2197–218. pmid:33818733.
- View Article
- PubMed/NCBI
- Google Scholar
23. Sousa VD, Rojjanasrirat W. Translation, adaptation and validation of instruments or scales for use in cross-cultural health care research: a clear and user-friendly guideline. J Eval Clin Pract. 2011;17(2):268–74. pmid:20874835.
- View Article
- PubMed/NCBI
- Google Scholar
24. Schaufeli WB, Bakker AB, Salanova M. The Measurement of Work Engagement With a Short Questionnaire. Educational and Psychological Measurement. 2006;66:701–16.
- View Article
- Google Scholar
25. Haerkens MHTM, van Leeuwen W, Sexton JB, Pickkers P, van der Hoeven JG. Validation of the Dutch language version of the Safety Attitudes Questionnaire (SAQ-NL). BMC Health Services Research. 2016;16(1):385. pmid:27528393
- View Article
- PubMed/NCBI
- Google Scholar
26. Sexton JB, Helmreich RL, Neilands TB, Rowan K, Vella K, Boyden J, et al. The Safety Attitudes Questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Services Research. 2006;6(1):44. pmid:16584553
- View Article
- PubMed/NCBI
- Google Scholar
27. Guo L, Ryan B, Leditschke IA, Haines K J, Cook K, Eriksson L, et al. Impact of unacceptable behaviour between healthcare workers on clinical performance and patient outcomes: a systematic review. BMJ Quality & Safety. 2022;31(9):679–87. pmid:35046101
- View Article
- PubMed/NCBI
- Google Scholar
28. Galbany-Estragués P, Comas-d’Argemir D. Care, Autonomy, and Gender in Nursing Practice: A Historical Study of Nurses’ Experiences. Journal of Nursing Research. 2017;25(5):361–7. 00134372-201710000-00007. pmid:28877123
- View Article
- PubMed/NCBI
- Google Scholar
29. Mokkink LB, Terwee CB, Knol DL, Stratford PW, Alonso J, Patrick DL, et al. The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: A clarification of its content. BMC Medical Research Methodology. 2010;10(1):22. pmid:20298572
- View Article
- PubMed/NCBI
- Google Scholar
30. Costello A, Osborne JW. Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Practical Assessment, Research and Evaluation. 2005;10:1–9.
- View Article
- Google Scholar
31. Howard MC. A Review of Exploratory Factor Analysis Decisions and Overview of Current Practices: What We Are Doing and How Can We Improve? International Journal of Human–Computer Interaction. 2016;32(1):51–62.
- View Article
- Google Scholar
32. Crede M, Harms P. Questionable research practices when using confirmatory factor analysis. Journal of Managerial Psychology. 2019;34(1):18–30.
- View Article
- Google Scholar
33. Jackson DL, Gillaspy JA, Purc-Stephenson R. Reporting practices in confirmatory factor analysis: an overview and some recommendations. Psychol Methods. 2009;14(1):6–23. pmid:19271845.
- View Article
- PubMed/NCBI
- Google Scholar
34. Prinsen CAC, Mokkink LB, Bouter LM, Alonso J, Patrick DL, de Vet HCW, et al. COSMIN guideline for systematic reviews of patient-reported outcome measures. Qual Life Res. 2018;27(5):1147–57. pmid:29435801.
- View Article
- PubMed/NCBI
- Google Scholar
35. Schreiber JB, Nora A, Stage FK, Barlow EA, King J. Reporting Structural Equation Modeling and Confirmatory Factor Analysis Results: A Review. The Journal of Educational Research. 2006;99(6):323–38.
- View Article
- Google Scholar
36. BHN. T. Our hospitals: Barts Health NHS Trust;; 2022 [cited 2022 05-30-2022]. https://www.bartshealth.nhs.uk/our-hospitals.
37. Brubakk K, Svendsen MV, Deilkås ET, Hofoss D, Barach P, Tjomsland O. Hospital work environments affect the patient safety climate: A longitudinal follow-up using a logistic regression analysis model. Plos One. 2021;16(10):e0258471. pmid:34653217.
- View Article
- PubMed/NCBI
- Google Scholar
38. Huebner LA, Zacher H. The role of mean item ratings, topic distance, direct leadership, and voice climate in action planning after employee surveys. Acta Psychol (Amst). 2023;238:103950. pmid:37379784.
- View Article
- PubMed/NCBI
- Google Scholar
39. Boussat B, Kamalanavin K, François P. The contribution of open comments to understanding the results from the Hospital Survey on Patient Safety Culture (HSOPS): A qualitative study. Plos One. 2018;13(4):e0196089. pmid:29672577.
- View Article
- PubMed/NCBI
- Google Scholar
40. Raso R, Fitzpatrick JJ, Masick K. Nurses’ Intent to Leave their Position and the Profession During the COVID-19 Pandemic. JONA: The Journal of Nursing Administration. 2021;51(10):488–94. pmid:34519700
- View Article
- PubMed/NCBI
- Google Scholar
41. Mokkink L, Terwee C, de Vet H. Key concepts in clinical epidemiology: Responsiveness, the longitudinal aspect of validity. J Clin Epidemiol. 2021;140:159–62. pmid:34116141.
- View Article
- PubMed/NCBI
- Google Scholar

[ref1] 1. Aiken LH, Sermeus W, Van den Heede K, Sloane DM, Busse R, McKee M, et al. Patient safety, satisfaction, and quality of hospital care: cross sectional surveys of nurses and patients in 12 countries in Europe and the United States. Bmj-Brit Med J. 2012;344. ARTN e1717 pmid:22434089
View Article
PubMed/NCBI
Google Scholar

[2] View Article

[3] PubMed/NCBI

[4] Google Scholar

[ref2] 2. Mathisen J, Nguyen TL, Jense JH, Rugulies R, Rod NH. Reducing employee turnover in hospitals: estimating the effects of hypothetical improvements in the psychosocial work environment. Scand J Work Environ Health. 2021;47(6):456–65. pmid:34052852.
View Article
PubMed/NCBI
Google Scholar

[6] View Article

[7] PubMed/NCBI

[8] Google Scholar

[ref3] 3. Slatten T, Lien G, Mutonyi BR. Precursors and outcomes of work engagement among nursing professionals-a cross-sectional study. BMC Health Serv Res. 2022;22(1):21. Epub 2022/01/06. pmid:34983510.
View Article
PubMed/NCBI
Google Scholar

[10] View Article

[11] PubMed/NCBI

[12] Google Scholar

[ref4] 4. Taylor N, Clay-Williams R, Hogden E, Braithwaite J, Groene O. High performing hospitals: a qualitative systematic review of associated factors and practical strategies for improvement. BMC Health Services Research. 2015;15(1):244. pmid:26104760
View Article
PubMed/NCBI
Google Scholar

[14] View Article

[15] PubMed/NCBI

[16] Google Scholar

[ref5] 5. Van Bogaert P, Clarke S, Willems R, Mondelaers M. Staff engagement as a target for managing work environments in psychiatric hospitals: implications for workforce stability and quality of care. J Clin Nurs. 2013;22(11–12):1717–28. pmid:23216757.
View Article
PubMed/NCBI
Google Scholar

[18] View Article

[19] PubMed/NCBI

[20] Google Scholar

[ref6] 6. Wan Q, Li Z, Zhou W, Shang S. Effects of work environment and job characteristics on the turnover intention of experienced nurses: The mediating role of work engagement. J Adv Nurs. 2018;74(6):1332–41. pmid:29350781.
View Article
PubMed/NCBI
Google Scholar

[22] View Article

[23] PubMed/NCBI

[24] Google Scholar

[ref7] 7. Wei H, Sewell KA, Woody G, Rose MA. The state of the science of nurse work environments in the United States: A systematic review. Int J Nurs Sci. 2018;5(3):287–300. pmid:31406839.
View Article
PubMed/NCBI
Google Scholar

[26] View Article

[27] PubMed/NCBI

[28] Google Scholar

[ref8] 8. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50. pmid:19664226
View Article
PubMed/NCBI
Google Scholar

[30] View Article

[31] PubMed/NCBI

[32] Google Scholar

[ref9] 9. Maassen SM, van Oostveen C, Vermeulen H, Weggelaar AM. Defining a positive work environment for hospital healthcare professionals: A Delphi study. Plos One. 2021;16(2):e0247530. pmid:33630923.
View Article
PubMed/NCBI
Google Scholar

[34] View Article

[35] PubMed/NCBI

[36] Google Scholar

[ref10] 10. Sutcliffe KM. High reliability organizations (HROs). Best Pract Res Clin Anaesthesiol. 2011;25(2):133–44. pmid:21550539.
View Article
PubMed/NCBI
Google Scholar

[38] View Article

[39] PubMed/NCBI

[40] Google Scholar

[ref11] 11. Schmutz JB, Meier LL, Manser T. How effective is teamwork really? The relationship between teamwork and performance in healthcare teams: a systematic review and meta-analysis. BMJ Open. 2019;9(9):e028280. pmid:31515415
View Article
PubMed/NCBI
Google Scholar

[42] View Article

[43] PubMed/NCBI

[44] Google Scholar

[ref12] 12. Sikka R, Morath u M, Leape L. The Quadruple Aim: care, health, cost and meaning in work. BMJ Quality & Safety. 2015;24(10):608–10. pmid:26038586
View Article
PubMed/NCBI
Google Scholar

[46] View Article

[47] PubMed/NCBI

[48] Google Scholar

[ref13] 13. Grant S, Davidson J, Manges K, Dermenchyan A, Wilson E, Dowdell E. Creating Healthful Work Environments to Deliver on the Quadruple Aim: A Call to Action. JONA: The Journal of Nursing Administration. 2020;50(6):314–21. pmid:32427662
View Article
PubMed/NCBI
Google Scholar

[50] View Article

[51] PubMed/NCBI

[52] Google Scholar

[ref14] 14. Maassen SM, Weggelaar Jansen A, Brekelmans G, Vermeulen H, van Oostveen CJ. Psychometric evaluation of instruments measuring the work environment of healthcare professionals in hospitals: a systematic literature review. Int J Qual Health Care. 2020;32(8):545–57. pmid:32648902.
View Article
PubMed/NCBI
Google Scholar

[54] View Article

[55] PubMed/NCBI

[56] Google Scholar

[ref15] 15. Rafferty AM, Philippou J, Fitzpatrick JM, Pike G, Ball J. Development and testing of the ’Culture of Care Barometer’ (CoCB) in healthcare organisations: a mixed methods study. BMJ Open. 2017;7(8):e016677. Epub 2017/08/20. pmid:28821526.
View Article
PubMed/NCBI
Google Scholar

[58] View Article

[59] PubMed/NCBI

[60] Google Scholar

[ref16] 16. Cho SH, Mark BA, Yun SC, June KJ. Differences in intensive care unit work environments among and within hospitals using subscales and a composite measure of the Revised Nursing Work Index. J Adv Nurs. 2011;67(12):2637–48. pmid:21645043.
View Article
PubMed/NCBI
Google Scholar

[62] View Article

[63] PubMed/NCBI

[64] Google Scholar

[ref17] 17. Rugulies R. What is a psychosocial work environment? Scandinavian Journal of Work, Environment & Health. 2019;(1):1–6. pmid:30643919
View Article
PubMed/NCBI
Google Scholar

[66] View Article

[67] PubMed/NCBI

[68] Google Scholar

[ref18] 18. Mokkink LB, de Vet HCW, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, et al. COSMIN Risk of Bias checklist for systematic reviews of Patient-Reported Outcome Measures. Qual Life Res. 2018;27(5):1171–9. pmid:29260445.
View Article
PubMed/NCBI
Google Scholar

[70] View Article

[71] PubMed/NCBI

[72] Google Scholar

[ref19] 19. Mokkink LB, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, de Vet HCW, et al. COSMIN Study Design checklist for Patient-reported outcome measurement instruments2019 08-25-2023:[1–32 pp.]. https://www.cosmin.nl/wp-content/uploads/COSMIN-study-designing-checklist_final.pdf#.

[ref20] 20. Henson RK, Roberts JK. Use of Exploratory Factor Analysis in Published Research: Common Errors and Some Comment on Improved Practice. Educational and Psychological Measurement. 2006;66(3):393–416.
View Article
Google Scholar

[75] View Article

[76] Google Scholar

[ref21] 21. Ying L, Fitzpatrick JM, Philippou J, Zhang Y, Murrells T, Rafferty AM. Front-line staff perspectives on a caring culture in Chinese hospitals: Validation of a Chinese version of the Culture of Care Barometer. J Nurs Manag. 2022;30(6):2093–102. pmid:35506434.
View Article
PubMed/NCBI
Google Scholar

[78] View Article

[79] PubMed/NCBI

[80] Google Scholar

[ref22] 22. Gagnier JJ, Lai J, Mokkink LB, Terwee CB. COSMIN reporting guideline for studies on measurement properties of patient-reported outcome measures. Qual Life Res. 2021;30(8):2197–218. pmid:33818733.
View Article
PubMed/NCBI
Google Scholar

[82] View Article

[83] PubMed/NCBI

[84] Google Scholar

[ref23] 23. Sousa VD, Rojjanasrirat W. Translation, adaptation and validation of instruments or scales for use in cross-cultural health care research: a clear and user-friendly guideline. J Eval Clin Pract. 2011;17(2):268–74. pmid:20874835.
View Article
PubMed/NCBI
Google Scholar

[86] View Article

[87] PubMed/NCBI

[88] Google Scholar

[ref24] 24. Schaufeli WB, Bakker AB, Salanova M. The Measurement of Work Engagement With a Short Questionnaire. Educational and Psychological Measurement. 2006;66:701–16.
View Article
Google Scholar

[90] View Article

[91] Google Scholar

[ref25] 25. Haerkens MHTM, van Leeuwen W, Sexton JB, Pickkers P, van der Hoeven JG. Validation of the Dutch language version of the Safety Attitudes Questionnaire (SAQ-NL). BMC Health Services Research. 2016;16(1):385. pmid:27528393
View Article
PubMed/NCBI
Google Scholar

[93] View Article

[94] PubMed/NCBI

[95] Google Scholar

[ref26] 26. Sexton JB, Helmreich RL, Neilands TB, Rowan K, Vella K, Boyden J, et al. The Safety Attitudes Questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Services Research. 2006;6(1):44. pmid:16584553
View Article
PubMed/NCBI
Google Scholar

[97] View Article

[98] PubMed/NCBI

[99] Google Scholar

[ref27] 27. Guo L, Ryan B, Leditschke IA, Haines K J, Cook K, Eriksson L, et al. Impact of unacceptable behaviour between healthcare workers on clinical performance and patient outcomes: a systematic review. BMJ Quality & Safety. 2022;31(9):679–87. pmid:35046101
View Article
PubMed/NCBI
Google Scholar

[101] View Article

[102] PubMed/NCBI

[103] Google Scholar

[ref28] 28. Galbany-Estragués P, Comas-d’Argemir D. Care, Autonomy, and Gender in Nursing Practice: A Historical Study of Nurses’ Experiences. Journal of Nursing Research. 2017;25(5):361–7. 00134372-201710000-00007. pmid:28877123
View Article
PubMed/NCBI
Google Scholar

[105] View Article

[106] PubMed/NCBI

[107] Google Scholar

[ref29] 29. Mokkink LB, Terwee CB, Knol DL, Stratford PW, Alonso J, Patrick DL, et al. The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: A clarification of its content. BMC Medical Research Methodology. 2010;10(1):22. pmid:20298572
View Article
PubMed/NCBI
Google Scholar

[109] View Article

[110] PubMed/NCBI

[111] Google Scholar

[ref30] 30. Costello A, Osborne JW. Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Practical Assessment, Research and Evaluation. 2005;10:1–9.
View Article
Google Scholar

[113] View Article

[114] Google Scholar

[ref31] 31. Howard MC. A Review of Exploratory Factor Analysis Decisions and Overview of Current Practices: What We Are Doing and How Can We Improve? International Journal of Human–Computer Interaction. 2016;32(1):51–62.
View Article
Google Scholar

[116] View Article

[117] Google Scholar

[ref32] 32. Crede M, Harms P. Questionable research practices when using confirmatory factor analysis. Journal of Managerial Psychology. 2019;34(1):18–30.
View Article
Google Scholar

[119] View Article

[120] Google Scholar

[ref33] 33. Jackson DL, Gillaspy JA, Purc-Stephenson R. Reporting practices in confirmatory factor analysis: an overview and some recommendations. Psychol Methods. 2009;14(1):6–23. pmid:19271845.
View Article
PubMed/NCBI
Google Scholar

[122] View Article

[123] PubMed/NCBI

[124] Google Scholar

[ref34] 34. Prinsen CAC, Mokkink LB, Bouter LM, Alonso J, Patrick DL, de Vet HCW, et al. COSMIN guideline for systematic reviews of patient-reported outcome measures. Qual Life Res. 2018;27(5):1147–57. pmid:29435801.
View Article
PubMed/NCBI
Google Scholar

[126] View Article

[127] PubMed/NCBI

[128] Google Scholar

[ref35] 35. Schreiber JB, Nora A, Stage FK, Barlow EA, King J. Reporting Structural Equation Modeling and Confirmatory Factor Analysis Results: A Review. The Journal of Educational Research. 2006;99(6):323–38.
View Article
Google Scholar

[130] View Article

[131] Google Scholar

[ref36] 36. BHN. T. Our hospitals: Barts Health NHS Trust;; 2022 [cited 2022 05-30-2022]. https://www.bartshealth.nhs.uk/our-hospitals.

[ref37] 37. Brubakk K, Svendsen MV, Deilkås ET, Hofoss D, Barach P, Tjomsland O. Hospital work environments affect the patient safety climate: A longitudinal follow-up using a logistic regression analysis model. Plos One. 2021;16(10):e0258471. pmid:34653217.
View Article
PubMed/NCBI
Google Scholar

[134] View Article

[135] PubMed/NCBI

[136] Google Scholar

[ref38] 38. Huebner LA, Zacher H. The role of mean item ratings, topic distance, direct leadership, and voice climate in action planning after employee surveys. Acta Psychol (Amst). 2023;238:103950. pmid:37379784.
View Article
PubMed/NCBI
Google Scholar

[138] View Article

[139] PubMed/NCBI

[140] Google Scholar

[ref39] 39. Boussat B, Kamalanavin K, François P. The contribution of open comments to understanding the results from the Hospital Survey on Patient Safety Culture (HSOPS): A qualitative study. Plos One. 2018;13(4):e0196089. pmid:29672577.
View Article
PubMed/NCBI
Google Scholar

[142] View Article

[143] PubMed/NCBI

[144] Google Scholar

[ref40] 40. Raso R, Fitzpatrick JJ, Masick K. Nurses’ Intent to Leave their Position and the Profession During the COVID-19 Pandemic. JONA: The Journal of Nursing Administration. 2021;51(10):488–94. pmid:34519700
View Article
PubMed/NCBI
Google Scholar

[146] View Article

[147] PubMed/NCBI

[148] Google Scholar

[ref41] 41. Mokkink L, Terwee C, de Vet H. Key concepts in clinical epidemiology: Responsiveness, the longitudinal aspect of validity. J Clin Epidemiol. 2021;140:159–62. pmid:34116141.
View Article
PubMed/NCBI
Google Scholar

[150] View Article

[151] PubMed/NCBI

[152] Google Scholar

Figures

Abstract

Objectives

Methods

Results

Conclusions

Introduction

Methods

Design

Setting and participants

Part 1: Translation and content validity

Part II: Survey

Ethical considerations.

Structural validity and internal consistency.

Convergent validity, discriminant validity, and responsiveness.

Analysis.

Results

Structural validity and internal consistency

Convergent validity, discriminant validity, and responsiveness

Discussion

Reflections on structural validity and internal consistency

Reflections on convergent validity, discriminant validity, and responsiveness

Implications for practice and future research

Limitations

Conclusions & recommendations

Supporting information

S1 Table. Sample characteristics of content validity on comprehensibility.

S2 Table. CoCB-NL item normal distribution.

S3 Table. Dutch language version of the CoCB-NL.

Acknowledgments

References