Article Text
PDF
Abstract
Objectives To validate and test the dimensionality of six constructs from the Workplace Integrated Safety and Health (WISH) assessment, an instrument that assesses the extent to which organisations implement integrated systems approaches for protecting and promoting worker health, safety and well-being, in a sample of nursing homes in the USA.
Design Validation of an assessment scale using data from a cross-sectional survey.
Setting Nursing homes certified by the Centers for Medicaid and Medicare services in three states of the USA: Ohio, California and Massachusetts.
Participants 569 directors of nursing from nursing homes serving adults and with more than 30 beds participated in the study.
Results Graded response Item Response Theory (IRT) models showed that five out of six constructs were unidimensional based on balanced interpretation of model fit statistics—M2 or C2 with p value >0.05, Comparative Fit Index >0.95, lower bound of the root mean squared error of approximation 90% CI <0.06 and standardised root mean square residual <0.08. Overall measure and construct reliability ranged from acceptable to good. Category boundary location parameters indicated that items were most informative for respondents in lower range of latent scores (ie, β1, β2, β3 typically below 0). A few items were recommended to be dropped from future administrations of the instrument based on empirical and substantive interpretation.
Conclusions The WISH instrument has utility to understand to what extent organisations integrate protection and promotion of worker health, safety and well-being; however, it is most informative in organisations that present lower scores.
- organisation of health services
- occupational & industrial medicine
- statistics & research methods
Data availability statement
Data are available upon reasonable request and by e-mail to: jwilliams13@kumc.edu.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
The sample was large and heterogeneous: Validation of the Workplace Integrated Health and Safety (WISH) assessment used survey data from 569 nursing homes in three states of the USA: Ohio, California and Massachusetts.
The respondents were knowledgeable: Directors of nursing responded on behalf of nursing homes as previous studies showed that they are more knowledgeable about policies, practices and programmes than any other manager; directors of nursing had the option to pass the survey to a health and safety representative at the facility if they judged them to be more knowledgeable about the survey topic.
The methods for selecting items was rigorous: The study used graded response Item Response Theory modelling to identify items for unidimensional WISH domains based on discrimination parameters, and information curves to assess conditional reliability along the latent scores.
The generalisability of the measure is limited by the study sample which included nursing homes with at least 30 beds located in the USA.
Background
Increasing evidence supports the integration of protection and promotion of workers’ health, safety and well-being as best practices to achieve a healthy workplace.1–3 Accordingly, the National Institute for Occupational Health and Safety, the agency focused on worker safety and health from the US Centers for Disease and Control Prevention, developed the Total Worker Health (TWH) Initiative that has as a goal to promote worker well-being by focusing on ‘programmes, policies and practices that integrate protection from work-related safety and health hazards with promotion of injury and illness prevention efforts to advance worker well-being.’4 Understanding the scope of TWH approaches, the extent to which they are implemented and how they relate with workers’ health, safety and well-being is essential for employers and for occupational health and safety organisations in setting priorities when developing interventions, policies and programmes.
Investigators from the Harvard T.H. Chan School of Public Health Center for Work, Health and Well-being (CWHW) have conducted research on the causal pathways through which policies, programmes and practices impact workers’ health, safety and well-being outcomes.2 3 5 6 Evidence from this research informed the development of a conceptual model that serves as a framework to conduct research and workplace interventions using TWH approaches.3 The conceptual model underscores the central role of working conditions and their impact on worker (health and well-being) and enterprise outcomes such as turnover and healthcare costs and the role that policies, programmes and practices have in shaping these working conditions.3 This conceptual model, related literature, and an expert panel guided the development of two measures to assess the extent to which organisations implement evidence-based integrated systems approaches for protecting and promoting worker health, safety and well-being: the Workplace Integrated Safety and Health (WISH) assessment7 and, its predecessor, the Indicators of Integration.2
The Indicators of Integration were validated in a sample of small-sized to medium-sized employers8 9 and a sample of the Department of Veterans Affairs (VA) administrative parents in the USA showing satisfactory convergent validity and reliability.8 9 However, the Indicators of Integration were primarily designed to measure the integration of health protection and health promotion. The WISH is more comprehensive and measures additional policies, programmes and practices that affect worker safety, health and well-being.7 It consists of six constructs that were identified as central in the integration of protection and promotion of workers’ health, safety and well-being: Leadership Commitment; Participation; Policies, Programmes and Practices that foster positive working conditions; Comprehensive and Collaborative Strategies; Adherence to federal and state regulations and ethical norms; and Data-driven Change. The WISH tool aims to assess the extent to which organisations adhere to best practice recommendations in terms of an integrated systems approach of programmes, policies and practices that address protection and promotion of health, safety and well-being.
This study aims to validate each domain comprising the WISH instrument in the nursing home industry. Specifically, we test each domain in the WISH assessment instrument as a unidimensional latent construct, estimate discrimination and location parameters, and inspect information curves to assess the conditional reliability and utility of each scale along the respective latent scores. The nursing home industry is of interest as its main goal is to provide high-quality patient care to an increasing ageing population. However, in the USA, the industry is sustained by a low-wage labour force and under-represented minorities and is largely dependent on payments from Medicaid and Medicare,10 11 which are federal health programmes for low-income individuals and individuals over age 65. Evidence from research on nursing home facilities shows that placing workers’ health as secondary to patient care may have deleterious effects on quality of patient care by increasing employee turnover.10 12 Measuring the implementation of integrated approaches to protect and promote workers’ health, safety and well-being in a nursing home population could help us understand how TWH approaches are associated with workers’, patients’ and enterprise outcomes.
Methods
The present study used data from the Enterprise Outcomes (EO) Study, a project from the Harvard T.H. Chan School of Public Health CWHW designed to assess TWH implementation in nursing homes and its relationship with worker and enterprise outcomes. The study uses a cross-sectional survey of directors of nursing (DONs) in nursing homes across three states in the USA: Ohio (OH), California (CA) and Massachusetts (MA). The survey included the WISH assessment and additional related questions.
Instrument
The items comprising the WISH constructs were developed through an extensive literature review and modified Delphi approach to ensure content validity, and cognitive pretesting with nursing home directors to ensure suitability, comprehension, information retrieval and response mapping.7 Methods on the approach are detailed elsewhere.7 The WISH assessment consists of six constructs ranging from 4 to 14 items with closed-ended questions and four-point ordinal response scale (not at all, somewhat, mostly, completely/not at all, some of the time, most of the time, all of the time) (table 1).
Domains and their definitions of the Workplace Integrated Safety and Health (WISH) assessment
Survey administration
All 2388 nursing homes in OH, CA and MA with at least 30 beds, serving adults and listed in Nursing Home Compare as accepting Medicare or Medicaid on 23 August 2018, were invited to participate in the survey (Nursing Home Compare (https://www.medicare.gov/nursinghomecompare). The states of OH, CA and MA were chosen for the survey as part of a larger project that links nursing home data on injuries from state Bureaus of Worker’s Compensation. DONs were initially invited to respond to the survey based on previous evidence indicating that nursing home directors are more knowledgeable about workplace policies, programmes and practices than any other managers in the organisation.10 Their knowledge is due to their involvement in the development and implementation of nursing home policies and procedures13 and exposure to the clinical operations of nursing homes.14 However, DONs were instructed to seek input or pass the survey to an appropriate health and safety representative for their nursing home, if needed. Data were collected via a sequential mixed mode survey, where a web survey protocol was implemented followed by a mail survey among non-respondents. An initial invitation was sent to DONs via email at each nursing home. The invitation explained the purpose of the survey, that the survey was voluntary and confidential, and that the survey was intended to be completed by the DON or a health and safety representative if they choose to pass it along. Up to three reminder emails were sent to non-respondents, followed by a mailed paper survey. There were some DONs without working email addresses—these individuals only received paper copies. Five hundred sixty-nine surveys were returned for a 24% response rate. A low response rate may be indicative of non-response bias; however, a previous survey study with Chiefs of Medicine showed that answers from a first survey wave with 29.7% response rate did not represent any statistical significant differences from answers collected during the first to the fifth survey waves that achieved an 85% response rate.15 The only parameter that changed was precision which increased as respondents accumulated.15 In other analyses (not shown), we used nursing home characteristics, such as for-profit status, state, number of beds, Medicare quality rating, Medicare staffing rating, Medicare health inspection rating, survey wave and rurality, to predict whether a nursing home responded to the survey.16
Statistical analysis
Data were initially characterised by frequencies to identify items with severely skewed distributions. We collapsed response categories with an adjacent category if fewer than 10% of respondents selected it. We analysed each domain of the WISH separately because they represent different latent constructs. We chose IRT analysis because it characterises the relationship between items and their respective constructs on several important parameters—discrimination (ie, strength of relationship of item with construct), category location (ie, location along the construct score where), and their information or reliability (ie, their contribution to measurement precision).
For each domain, we fitted unidimensional graded response models to select items based on empirical information (discrimination and local dependence) and, importantly, substantive interpretation to ensure content validity. Items were flagged for discussion if they had small discrimination parameters relative to the rest of the items in their domains or exhibited local dependence. Local dependence was assessed using the G2 statistic, with items flagged if the p value was below Bonferroni-corrected alpha of 0.05. When local dependence was identified in a pair of items, the item with greater discrimination was retained unless the other item better ensured content validity. Likewise, items with smaller discrimination relative to other items were retained if their elimination compromised content validity. Substantive interpretation was conducted by the study team with input from other investigators at the Center for Work, Health, and Well-being.17
We examined measurement precision—which in IRT is summarised with information curves where higher numbers indicate better precision or, equivalently, greater information—for each construct across the range of their latent scores. We also present the empirical reliability as a measure of the overall reliability of the latent scores.
Limited information goodness-of-fit χ2 statistic (M2 or C2 depending on model df), root mean square error of approximation (RMSEA), standardised root mean square residuals (SRMSR) and Comparative Fit Index (CFI)18 were used to evaluate fit of unidimensional model for each construct. These fit indices evaluate models relative to a saturated model (M2 and C2), improvement relative to a null model (CFI), as well as absolute measures of fit (RMSEA and SRMSR). We use Hu and Bentler’s (1999) threshold recommendations to consider good model fit—lower bound of the 90% CI for RMSEA (RMSEA LB)<0.06, SRMS <0.08 and CFI>0.95. Item fit was assessed by the signed χ2 test and by interpreting empirical plots.19 The analyses were conducted using the mirt package in R, using marginal maximum likelihood via Expectation Maximisation estimation20 (R Core Team, 2019).
Patient or public involvement
No patients were involved in this study.
Results
Sample
Table 2 depicts the characteristics of the participating nursing homes. Most surveys were completed by the DON (n=480, 84.4%)%), 7.6% (n=43) reported a title other than DON (13 were administrators and 11 were assistant/interim DONs). Response rates did not differ significantly between profit and non-for-profit nursing homes or based on the mode (paper/email) of the survey.16 The majority of nursing homes in OH and CA had 51–100 beds with plurality in MA having 101–150 beds, and the majority in all states being for-profit.
Nursing home characteristics
Table 3 shows the distribution of responses by category for each WISH item retained in the final scales. All variables were skewed towards selection of Mostly/Most of the time and Complete/All of the time. ‘Not at all’ consistently had less than 10% selection, with the exception of two items (Participation q2 and Policies, Programmes and Practices q11), thus all variables collapsed ‘not at all’ with ‘somewhat’ or ‘some of the time’, as applicable, resulting in a three-point analytical scale. Online supplemental appendix table 1 includes items that were deleted due to lower discrimination.
Supplemental material
Final items in the Workplace Integrated Safety and Health (WISH) assessment response distribution
Table 4 shows model fit statistics, discrimination (α) and category boundary location (βi) parameters for each item by domain. Leadership Commitment, Comprehensive and Collaborative Strategies, and Adherence had good fitting unidimensional models indicating it fits the data as well as a saturated model (C2 or M2), improvement from a null model (CFI >0.95), deviation of the model implied correlation matrix (SRMSR <0.08), and absolute fit (lower bound of RMSEA 90% CI (RMSEA LB)<0.06). Participation showed good fit relative to a null model (CFI=0.99), deviation of the model implied correlation matrix (SRMSR=0.03) and the RMSEA (RMSEA=0.10, RMSEA LB=0.06), but indicated some misfit relative to a saturated model (M2(df=2)=14.2). Policies, Programmes and Practices showed acceptable fit of the model implied correlation matrix (SRMSR=0.05) and RMSEA (RMSEA=0.08, RMSEA LB=0.06), but showed some lack of fit relative to a null model (CFI=0.93) and saturated model (M2(df=22)=84.93). Adherence had good fit of the model implied correlation matrix, RMSEA (RMSEA=0.07, RMSEA LB=0.02) and relative to a null model (CFI=1.0), but some misfit relative to a saturated model (C2=6.640, df=2). No reasonably fitting model could be identified for the dimension Data-driven Change, thus it is not presented further.
Discrimination and category location parameters by domain
Discrimination parameters (α) were large for all constructs, indicating each item is sensitive to differences on the latent scores for each construct. These discrimination parameters ranged from 2.55 to 4.07 for Leadership, 2.1 to 3.31 for Participation, 1.19 to 4.34 for Policies, Programmes and Practices, 3.3 to 4.84 for Comprehensive and Collaborative Strategies, and 2.96 to 5.1 for Adherence. Category boundary location parameters (β1, β2, β3) were mostly in the negative range of the latent scores, indicating most items are most informative for nursing homes scoring lower on the constructs.
Figure 1 shows the test information curve for each domain. These curves show the amount of information (or measurement precision) in the sets of items at different scores of the latent construct (locations). Each domain has limited range on the latent trait within which it has high information. The peak of the information curve was above 15 for Policies, Programmes and Practices, and Adherence, and above 10 for Leadership and Comprehensive and Collaborative Strategies. The sets of items measuring these constructs were most informative (thus providing the most reliable measurement) among nursing homes with lower scores on the respective latent traits, typically in the range of −2 to 0. The range for Participation was more informative in the range of −1 to 1, with its peak at 8. The empirical reliability of the scores on the latent traits was 0.81 for Leadership, 0.82 for Participation, 0.87 for Policies, Programmes and Practices, 0.82 for Comprehensive and Collaborative Strategies, and 0.70 for Adherence.
Test information curves by domain of the Workplace Integrated Safety and Health (WISH) assessment.
Discussion
This paper aimed to validate the WISH assessment by examining the dimensionality of its six constructs based on a survey conducted with DONs in nursing homes in three states in the USA. We identified unidimensional scales for Leadership Commitment, Participation, Policies, Programmes and Practices, Comprehensive and Collaborative Strategies, and Adherence, with acceptable fit on most or all fit measures. Empirical reliability suggested overall acceptable to good reliability among all constructs. Additional analyses (category boundary locations and test information curves) revealed that most items, and thus the scales, were most informative among nursing homes with low levels working conditions that support safety, health and well-being. The WISH can further our understanding of best practices for an integrated systems approach to protect and promote workers’ health, safety and well-being.
The WISH has several strengths that make it well suited to different applications. Due to the large discrimination parameters, reliable estimates of the respective constructs can be estimated with relatively few items. This makes the assessment particularly well suited for studies covering the WISH and additional topics, reducing participation time.
There were two domains that would benefit from further study. Although all domains exhibited some ceiling effect, Adherence’s was pronounced as 53% of nursing homes had a maximum total score. Future work on the Adherence domain should seek to develop items that discriminate and are informative among higher scoring nursing homes. Additionally, we could not identify a reasonable and good fitting model for Data-driven Change. At present, we cannot claim these items measure a latent data-driven change construct. However, the individual items may still be of interest to researchers for their individual measurement potential. They can be directly interpreted on an item-by-item basis. For example, the item ‘Integrated data on employee safety and health outcomes are coordinated across all relevant departments’ can be used and interpreted as a measure of integrated data coordination without any claim that it is a manifestation of data-driven change. Indeed, these items (table 3) have been thoroughly pretested previously.7
The study has limitations regarding the nature of the sample. The sample only included nursing homes with at least 30 beds or more, from three states. The survey was sent to DONs; 84.4% of the respondents were indeed DONs, 7.4% did not report a title, and 7.6% reported a different, but often similar, title. We assume that the individuals answering the survey had requisite knowledge to complete the task, particularly for non-DOs respondents. Responses from nursing homes with a DON respondent did not differ significantly from nursing homes with a different type of respondent.16 After controlling for for-profit status, state, number of beds, Medicare quality rating, Medicare staffing rating, Medicare health inspection rating, survey wave and rurality, we did not find any statistically significant predictors of whether a nursing home responded.16 The WISH should be validated in additional samples for robustness to geographic location and setting. Our findings should be regarded as a first version of these scales, to be expanded on and modified in future studies. This should, as the body of research grows, result in scales that are generalisable with respect to type of establishment studied.
Despite ceiling effects on some domains, the WISH instrument has utility for many research and applied settings. For example, research among organisations with low performance on Adherence can get reliable estimates of the respective latent scores. Organisations such as these are often of most concern for researchers and practitioners trying to improve workplace safety, health and well-being. As such, an intervention targeting workplace safety, health and well-being can still detect meaningful changes and non-intervention longitudinal studies can similarly observe changes over time. Moreover, the tools can have tremendous utility in applied work as a screening instrument to identify low-performing organisations that need intervention or further study. Findings from this validation study indicate that the WISH assessment is adequate to measure the extent to which organisations adhere to best practices related to integrated systems approaches that protect and promote workers’ health, safety and well-being. Measuring the domains of the WISH can help organisations and researchers in their prioritisation and decision-making to improve workplace conditions, using resources to affect the areas with the worst performance. Using the WISH tool at baseline with follow-up assessments has the potential to provide organisations a way to identify improvements or setbacks in the organisation in relation to recommended best practices. The WISH tool may also complement surveys directed to workers which can inform managers and researchers alike of any discordances between organisational policies, practices and programmes and workers’ perceptions of these.
Data availability statement
Data are available upon reasonable request and by e-mail to: jwilliams13@kumc.edu.
Ethics statements
Ethics approval
This study received ethics approval from the Harvard T.H. Chan School of Public Health (IRB file number 18-1245).
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors MALG, DAG, LIB, GS, JNK, JEC, GW, MGV and JARW participated in the conceptualisation and design of the study. MGV, JARW and MALG organised the data collection process with update meetings and advice from all authors. DAG, MALG and JARW analysed data and presented findings to the rest of the group for discussion. MALG, DAG, LIB, GS, JNK, JEC, GW, MGV and JARW participated in the interpretation of results and critically reviewed and edited the manuscript and approved the final version for submission to BMJ Open.
Funding This work was supported by a grant from the National Institute for Occupational Safety and Health (U19 OH008861) for the Harvard T.H. Chan School of Public Health Center for Work, Health and Well-being. The funding agency has no role in the design and conduct of the research.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.