Skip to main content

Advertisement

SpringerLink
Log in

Psychometric evaluation of a patient-reported item bank for healthcare engagement

  • Published:
Quality of Life Research Aims and scope Submit manuscript

Abstract

Purpose

Healthcare engagement is a core measurement target for efforts to improve healthcare systems. This construct is broadly defined as the extent to which healthcare services represent collaborative partnerships with patients. Previous qualitative work operationalized healthcare engagement as generalized self-efficacy in four related subdomains: self-management, collaborative communication, health information use, and healthcare navigation. Building on this work, our objective was to establish a healthcare engagement instrument that is sufficiently unidimensional to yield a single score.

Method

We conducted cognitive interviews followed by a nation-wide mail survey of US Veteran Administration (VA) healthcare users. Data were collected on 49 candidate healthcare engagement items, as well as measures of self-efficacy for managing symptoms, provider communication, and perceived access. Items were subjected to exploratory bifactor, statistical learning, and IRT analyses.

Results

Cognitive interviews were completed by 56 patients and 9552 VA healthcare users with chronic conditions completed the mail survey. Participants were mostly white and male but with sizable minority participation. Psychometric analyses and content considerations reduced the item pool to 23 items, which demonstrated a strong general factor (OmegaH of .89). IRT analyses revealed a high level of reliability across the trait range and little DIF across groups. Most health information use items were removed during analyses, suggesting a more independent role for this domain.

Conclusion

We provide quantitative evidence for a relatively unidimensional measure of healthcare engagement. Despite developed with VA healthcare users, the measure is intended for general use. Future work includes short-form development and validation with other patient groups.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

Data will be made available in accordance with the data management plan submitted to the VA Office of Research and Development. Within one year of publication of manuscripts addressing the aims of the grant, investigators will make a deidentified, anonymized dataset available to the public.

Code availability

Code for the analyses presented in this paper will be made available upon request.

References

  1. Blumenthal, D., & McGinnis, J. M. (2015). Measuring vital signs: An IOM report on core metrics for health and health care progress. JAMA, 313(19), 1901–1902. https://doi.org/10.1001/jama.2015.4862.

    Article  PubMed  Google Scholar 

  2. Carman, K. L., Dardess, P., Maurer, M., Sofaer, S., Adams, K., Bechtel, C., & Sweeney, J. (2013). Patient and family engagement: A framework for understanding the elements and developing interventions and policies. Health Affairs (Millwood), 32(2), 223–231. https://doi.org/10.1377/hlthaff.2012.1133.

    Article  Google Scholar 

  3. Frampton, S. B., Guastello, S., Hoy, L., Naylor, M., Sheridan, S., & Johnston-Fleece, M. (2017). Harnessing evidence and experience to change culture: A guiding framework for patient and family engaged care. NAM Perspectives, 7(1), 1–37. https://doi.org/10.31478/201701f.

    Article  Google Scholar 

  4. Berwick, D. M., Nolan, T. W., & Whittington, J. (2008). The triple aim: Care, health, and cost. Health Affairs (Millwood), 27(3), 759–769. https://doi.org/10.1377/hlthaff.27.3.759.

    Article  Google Scholar 

  5. Castellon-Lopez, Y., Skrine Jeffers, K., Duru, O. K., Moreno, G., Moin, T., Grotts, J., et al. (2020). Psychometric properties of the altarum consumer engagement (ACE) measure of activation in patients with prediabetes. Journal of General Internal Medicine, 35(11), 3159–3165. https://doi.org/10.1007/s11606-020-05727-z.

    Article  PubMed  Google Scholar 

  6. Graffigna, G., Barello, S., Bonanomi, A., & Lozza, E. (2015). Measuring patient engagement: Development and psychometric properties of the Patient Health Engagement (PHE) Scale. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2015.00274.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Wasson, J. H., & Coleman, E. A. (2014). Health confidence: An essential measure for patient engagement and better practice. Family Practice Management, 21(5), 8–12.

    PubMed  Google Scholar 

  8. Gruman, J., Rovner, M. H., French, M. E., Jeffress, D., Sofaer, S., Shaller, D., & Prager, D. J. (2010). From patient education to patient engagement: Implications for the field of patient education. Patient Education and Counseling, 78(3), 350–356. https://doi.org/10.1016/j.pec.2010.02.002.

    Article  PubMed  Google Scholar 

  9. Tzeng, H. M., & Marcus Pierson, J. (2017). Measuring patient engagement: Which healthcare engagement behaviours are important to patients? Journal of Advanced Nursing, 73(7), 1604–1609. https://doi.org/10.1111/jan.13257.

    Article  PubMed  Google Scholar 

  10. Powell, R. E., Doty, A., Casten, R. J., Rovner, B. W., & Rising, K. L. (2016). A qualitative analysis of interprofessional healthcare team members’ perceptions of patient barriers to healthcare engagement. BMC Health Services Research. https://doi.org/10.1186/s12913-016-1751-5.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Tai-Seale, M., Sullivan, G., Cheney, A., Thomas, K., & Frosch, D. L. (2016). The language of engagement: “Aha!” moments from engaging patients and community partners in two pilot projects of the patient-centered outcomes research institute. The Permanente Journal, 20(2), 89–92. https://doi.org/10.7812/TPP/15-123.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Grant, R. W., Altschuler, A., Uratsu, C. S., Sanchez, G., Schmittdiel, J. A., Adams, A. S., & Heisler, M. (2017). Primary care visit preparation and communication for patients with poorly controlled diabetes: A qualitative study of patients and physicians. Primary Care Diabetes, 11(2), 148–153. https://doi.org/10.1016/j.pcd.2016.11.003.

    Article  PubMed  Google Scholar 

  13. Austin, E. J., Lee, J. R., Bergstedt, B., Mitchell, A. I., Javid, S. H., Ko, C. W., & Gore, J. L. (2020). “Help me figure this out”: Qualitative explorations of patient experiences with cancer pathology reports. Patient Education and Counseling. https://doi.org/10.1016/j.pec.2020.07.020.

    Article  PubMed  Google Scholar 

  14. Smith, S. K., Dixon, A., Trevena, L., Nutbeam, D., & McCaffery, K. J. (2009). Exploring patient involvement in healthcare decision making across different education and functional health literacy groups. Social Science and Medicine, 69(12), 1805–1812. https://doi.org/10.1016/j.socscimed.2009.09.056.

    Article  PubMed  Google Scholar 

  15. Bokhour, B. G., Cohn, E. S., Cortes, D. E., Solomon, J. L., Fix, G. M., Elwy, A. R., et al. (2012). The role of patients’ explanatory models and daily-lived experience in hypertension self-management. Journal of General Internal Medicine, 27(12), 1626–1634. https://doi.org/10.1007/s11606-012-2141-2.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Kimerling, R., Lewis, E. T., Javier, S. J., & Zulman, D. M. (2019). Opportunity or burden? A behavioral framework for patient engagement. Medical Care. https://doi.org/10.1097/MLR.0000000000001240.

    Article  Google Scholar 

  17. Bombard, Y., Baker, G. R., Orlando, E., Fancott, C., Bhatia, P., Casalino, S., et al. (2018). Engaging patients to improve quality of care: A systematic review. Implementation Science, 13(1), 1–22. https://doi.org/10.1186/s13012-018-0784-z.

    Article  Google Scholar 

  18. Bandura, A. (2006). Guide for constructing self-efficacy scales. In F. Pajares & T. Urdan (Eds.), Self-efficacy beleifs of adolescents. (pp. 307–337). Information Age Publishing.

    Google Scholar 

  19. Bandura, A. (2004). Health promotion by social cognitive means. Health Education and Behavior, 31(2), 143–164. https://doi.org/10.1177/1090198104263660.

    Article  PubMed  Google Scholar 

  20. Reeve, B. B., Hays, R. D., Bjorner, J. B., Cook, K. F., Crane, P. K., Teresi, J. A., et al. (2007). Psychometric evaluation and calibration of health-related quality of life item banks: Plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Medical Care, 45(5 Suppl 1), S22-31. https://doi.org/10.1097/01.mlr.0000250483.85507.04.

    Article  PubMed  Google Scholar 

  21. Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S., et al. (2010). The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology, 63(11), 1179–1194. https://doi.org/10.1016/j.jclinepi.2010.04.011.

    Article  PubMed  PubMed Central  Google Scholar 

  22. DeWalt, D. A., Rothrock, N., Yount, S., Stone, A. A., & PROMIS Cooperative Group. (2007). Evaluation of item candidates: The PROMIS qualitative item review. Medical care, 45(5 Suppl 1), S12–S21. https://doi.org/10.1097/01.mlr.0000254567.79743.e2.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7(3), 309–319.

    Article  Google Scholar 

  24. Fihn, S. D., Francis, J., Clancy, C., Nielson, C., Nelson, K., Rumsfeld, J., et al. (2014). Insights from advanced analytics at the veterans health administration. Health Affairs, 33(7), 1203–1211. https://doi.org/10.1377/hlthaff.2014.0054.

    Article  PubMed  Google Scholar 

  25. Wong, E. S., Wang, V., Liu, C. F., Hebert, P. L., & Maciejewski, M. L. (2015). Do Veterans health administration enrollees generalize to other populations? Medical Care Research and Review. https://doi.org/10.1177/1077558715617382.

    Article  PubMed  Google Scholar 

  26. Hernandez, S. E., Sylling, P. W., Mor, M. K., Fine, M. J., Nelson, K. M., Wong, E. S., et al. (2019). Developing an algorithm for combining race and ethnicity data sources in the Veterans health administration. Military Medicine, 185(3–4), 3495-e500. https://doi.org/10.1093/milmed/usz322.

    Article  Google Scholar 

  27. Nelson, K. M., Chang, E. T., Zulman, D. M., Rubenstein, L. V., Kirkland, F. D., & Fihn, S. D. (2019). Using predictive analytics to guide patient care and research in a national health system. Journal of General Internal Medicine, 34(8), 1379–1380. https://doi.org/10.1007/s11606-019-04961-4.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Wagner, T. H., Almenoff, P., Francis, J., Jacobs, J., & Pal Chee, C. (2018). Assessment of the Medicare advantage risk adjustment model for measuring veterans affairs hospital performance. JAMA Network Open, 1(8), e185993. https://doi.org/10.1001/jamanetworkopen.2018.5993.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Gruber-Baldini, A. L., Velozo, C., Romero, S., & Shulman, L. M. (2017). Validation of the PROMIS® measures of self-efficacy for managing chronic conditions. Quality of Life Research, 26(7), 1915–1924. https://doi.org/10.1007/s11136-017-1527-3.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Dyer, N., Sorra, J. S., Smith, S. A., Cleary, P. D., & Hays, R. D. (2012). Psychometric properties of the consumer assessment of healthcare providers and systems (CAHPSs) clinician and group adult visit survey. Medical Care, 50(11), 7.

    Google Scholar 

  31. Kimerling, R., Pavao, J., Greene, L., Karpenko, J., Rodriguez, A., Saweikis, M., & Washington, D. L. (2015). Access to mental health care among women Veterans: Is VA meeting women’s needs? Medical Care, 53(4 Suppl 1), S97–S104. https://doi.org/10.1097/MLR.0000000000000272.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Bollen, K., & Lennox, R. (1991). Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin, 110(2), 305–314. https://doi.org/10.1037/0033-2909.110.2.305.

    Article  Google Scholar 

  33. Streiner, D. L., Norman, G. R., & Cairney, J. (2015). Health measurement scales: A practical guide to their development and use. . Oxford University Press.

    Book  Google Scholar 

  34. Revelle, W. (1979). Hierarchical cluster analysis and the internal structure of tests. Multivariate Behavioral Research, 14(1), 57–74. https://doi.org/10.1207/s15327906mbr1401_4.

    Article  CAS  PubMed  Google Scholar 

  35. Reise, S. P., Morizot, J., & Hays, R. D. (2007). The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Quality of Life Research, 16(1), 19–31. https://doi.org/10.1007/s11136-007-9183-7.

    Article  PubMed  Google Scholar 

  36. Reise, S. P., Bonifay, W. E., & Haviland, M. G. (2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95(2), 129–140. https://doi.org/10.1080/00223891.2012.725437.

    Article  PubMed  Google Scholar 

  37. Chapman, B. P., Weiss, A., & Duberstein, P. (2016). Statistical learning theory for high dimensional prediction: Application to criterion-keyed scale development. Psychological Methods, 21(4), 603–620. https://doi.org/10.1037/met0000088.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Reise, S. P., Moore, T. M., & Haviland, M. G. (2010). Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scale scores. Journal of Personality Assessment, 92(6), 544–59. https://doi.org/10.1080/00223891.2010.496477.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Smits, N., van der Ark, L. A., & Conijn, J. M. (2018). Measurement versus prediction in the construction of patient-reported outcome questionnaires: Can we have our cake and eat it? Quality of Life Research, 27(7), 1673–1682. https://doi.org/10.1007/s11136-017-1720-4.

    Article  PubMed  Google Scholar 

  40. Elleman, L. G., McDougald, S. K., Condon, D. M., & Revelle, W. (2020). That takes the BISCUIT: A comparative study of predictive accuracy and parsimony of four statistical learning techniques in personality data, with data missingness conditions. European Journal of Personality Assessment. https://doi.org/10.31234/osf.io/tuqap.

    Article  Google Scholar 

  41. Samejima, F. (1968). Estimation of latent ability using a response pattern of graded scores. ETS Research Bulletin Series, 1968(1), i–169. https://doi.org/10.1002/j.2333-8504.1968.tb00153.x.

    Article  Google Scholar 

  42. Gibbons, R. D., Bock, R. D., Hedeker, D., Weiss, D. J., Segawa, E., Bhaumik, D. K., et al. (2007). Full-information item bifactor analysis of graded response data. Applied Psychological Measurement, 31(1), 4–19. https://doi.org/10.1177/0146621606289485.

    Article  Google Scholar 

  43. Toland, M. D., Sulis, I., Giambona, F., Porcu, M., & Campbell, J. M. (2017). Introduction to bifactor polytomous item response theory analysis. Journal of School Psychology, 60, 41–63. https://doi.org/10.1016/j.jsp.2016.11.001.

    Article  PubMed  Google Scholar 

  44. Stucky, B. D., Thissen, D., & Edelen, M. O. (2013). Using logistic approximations of marginal trace lines to develop short assessments. Applied Psychological Measurement, 37(1), 41–57. https://doi.org/10.1177/0146621612462759.

    Article  Google Scholar 

  45. Choi, S. W., Gibbons, L. E., & Crane, P. K. (2011). lordif : An R package for detecting differential item functioning using iterative hybrid ordinal logistic regression/item response theory and Monte Carlo simulations. Journal of Statistical Software. https://doi.org/10.18637/jss.v039.i08.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Hays, R. D., Calderón, J. L., Spritzer, K. L., Reise, S. P., & Paz, S. H. (2018). Differential item functioning by language on the PROMIS® physical functioning items for children and adolescents. Quality of Life Research, 27(1), 235–247. https://doi.org/10.1007/s11136-017-1691-5.

    Article  PubMed  Google Scholar 

  47. Revelle, W. (2020) psych: Procedures for Personality and Psychological Research. 2.0.12  Evanston. Illinois: Northwestern University. https://CRAN.R-project.org/package=psych.

  48. Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software. https://doi.org/10.18637/jss.v048.i02.

    Article  Google Scholar 

  49. Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software. https://doi.org/10.18637/jss.v048.i06.

    Article  Google Scholar 

  50. Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431–444. https://doi.org/10.1177/014662168200600405.

    Article  Google Scholar 

  51. Koh, H. K., Brach, C., Harris, L. M., & Parchman, M. L. (2013). A proposed “health literate care model” would constitute a systems approach to improving patients’ engagement in care. Health Affairs (Millwood), 32(2), 357–67. https://doi.org/10.1377/hlthaff.2012.1205.

    Article  Google Scholar 

  52. Roberts, E. T., Zaslavsky, A. M., Barnett, M. L., Landon, B. E., Ding, L., & McWilliams, J. M. (2018). Assessment of the effect of adjustment for patient characteristics on hospital readmission rates: Implications for pay for performance. JAMA Internal Medicine, 178(11), 1498–1507. https://doi.org/10.1001/jamainternmed.2018.4481.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Shakir, M., Armstrong, K., & Wasfy, J. H. (2018). Could pay-for-performance worsen health disparities? Journal of General Internal Medicine, 33(4), 567–569. https://doi.org/10.1007/s11606-017-4243-3.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Hausmann, L. R. M., Canamucio, A., Gao, S., Jones, A. L., Keddem, S., Long, J. A., & Werner, R. (2017). Racial and ethnic minority concentration in veterans affairs facilities and delivery of patient-centered primary care. Population Health Management, 20(3), 189–198. https://doi.org/10.1089/pop.2016.0053.

    Article  PubMed  Google Scholar 

  55. Ginsburg, P. B. (2013). Achieving health care cost containment through provider payment reform that engages patients and providers. Health Affairs (Project Hope), 32(5), 929–934. https://doi.org/10.1377/hlthaff.2012.1007.

    Article  Google Scholar 

Download references

Acknowledgements

This manuscript is supported by grants #I21HX001855 and 1I01HX002317 from the United States (US) Department of Veterans Affairs Health Services Research and Development Service. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.

Funding

This manuscript is supported by Grants #I21HX001855 and 1I01HX002317 from the United States (US) Department of Veterans Affairs Health Services Research and Development Service.

Author information

Authors and Affiliations

  1. Department of Medical Social Sciences, Northwestern University Feinberg School of Medicine, Chicago, USA

    Benjamin D. Schalet

  2. Department of Psychology, University of California, San Diego, USA

    Steven P. Reise

  3. National Center for PTSD, VA Palo Alto Health Care System, Menlo Park, USA

    Rachel Kimerling

  4. Center for Innovation to Implementation, VA Palo Alto Health Care System, Menlo Park, USA

    Donna M. Zulman, Eleanor T. Lewis & Rachel Kimerling

  5. Division of Primary Care and Population Health, Stanford University School of Medicine, Stanford, USA

    Donna M. Zulman

Authors
  1. Benjamin D. Schalet
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Steven P. Reise
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Donna M. Zulman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eleanor T. Lewis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rachel Kimerling
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Benjamin D. Schalet.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schalet, B.D., Reise, S.P., Zulman, D.M. et al. Psychometric evaluation of a patient-reported item bank for healthcare engagement. Qual Life Res 30, 2363–2374 (2021). https://doi.org/10.1007/s11136-021-02824-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11136-021-02824-2

Keywords

Search

Navigation