Skip to main content

Advertisement

SpringerLink
Log in

Comparison of anchor-based methods for estimating thresholds of meaningful within-patient change using simulated PROMIS PF 20a data under various joint distribution characteristic conditions

  • Special Section: Methodologies for Meaningful Change
  • Published:
Quality of Life Research Aims and scope Submit manuscript

Abstract

Purpose

To compare the performance of anchor-based methods for estimating thresholds of meaningful within-patient change (i.e., individual change) of clinical outcome assessments in conditions reflecting data characteristics of small- to medium-sized clinical trials.

Methods

Datasets were generated from the joint distributions of the PROMIS PF 20a T-score changes and a seven-point global change anchor measure. The 108 simulation conditions (1000 replications per condition) included combinations of three marginal distributions of T-score changes, three improvement percentages in the anchor measure, four levels of responsiveness correlations, and three sample sizes. Threshold estimation methods included mean change, median change, ROC curve, predictive modeling, half SD, and SEM. Relative bias, precision, accuracy, and measurement significance of the estimates were evaluated based on comparison with true thresholds and IRT-based individual reliable changes of PROMIS scores. Quantile regression models were applied to select and interpret effects of simulation conditions on estimation bias.

Results

When PROMIS T-score changes were distributed normally, the predictive modeling method performed best with 50% or more responders identified by the anchor; the mean and median methods were preferred with 30% responders. For skewed distributions, the median method and ROC method gained more advantages. Among the evaluated study conditions, the improvement percentage condition had the most obvious effects on estimation bias.

Conclusion

To establish accurate and precise thresholds, clinical researchers are recommended to prioritize study designs with at least 50% anchor-defined responders and strongly responsive target endpoints with highly reliable scoring calibration and to select optimal anchor-based methods given the data characteristics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. U.S. Department of Health and Human Services, Food and Drug Administration. (2009). Guidance for industry: Patient-reported outcome measures: use in medical product development to support labeling claims. Retrieved November 25, 2021, from https://www.fda.gov/media/77832/download

  2. Food and Drug Administration. (2018). Patient-focused drug development guidance public workshop: methods to identify what is important to patients & select, develop or modify fit-for-purpose clinical outcomes assessments. Guidance 3 discussion document. Retrieved November 25, 2021, from https://www.fda.gov/media/116277/download

  3. Food and Drug Administration. (2019). Patient-focused drug development guidance public workshop: incorporating clinical outcome assessments into endpoints for regulatory decision making. Guidance 4 discussion document. Retrieved April 18, 2021, from https://www.fda.gov/media/132505/download

  4. Chan, E. K. H., Edwards, T. C., Haywood, K., Mikles, S. P., & Newton, L. (2019). Implementing patient-reported outcome measures in clinical practice: A companion guide to the ISOQOL user’s guide. Quality of Life Research, 28(3), 621–627. https://doi.org/10.1007/s11136-018-2048-4

    Article  PubMed  Google Scholar 

  5. Cappelleri, J. C., & Bushmakin, A. G. (2014). Interpretation of patient-reported outcomes. Statistical methods in medical research, 23(5), 460–483. https://doi.org/10.1177/0962280213476377

    Article  PubMed  Google Scholar 

  6. McLeod, L. D., Coon, C. D., Martin, S. A., Fehnel, S. E., & Hays, R. D. (2011). Interpreting patient-reported outcome results: US FDA guidance and emerging methods. Expert Review of Pharmacoeconomics & Outcomes Research, 11(2), 163–169. https://doi.org/10.1586/erp.11.12

    Article  Google Scholar 

  7. Coon, C. D. (2016). Telling the interpretation story: The case for strong anchors and multiple methods. Plenary presentation at the ISOQOL 23rd Annual Conference; October 19, 2016. Copenhagen, Denmark

  8. Coon, C. D., & Cook, K. F. (2018). Moving from significance to real-world meaning: Methods for interpreting change in clinical outcome assessment scores. Quality of Life Research, 27(1), 33–40. https://doi.org/10.1007/s11136-017-1616-3

    Article  PubMed  Google Scholar 

  9. Terluin, B., Eekhout, I., & Terwee, C. (2017). The anchor-based minimal important change, based on receiver operating characteristic analysis or predictive modeling, may need to be adjusted for the proportion of improved patients. Journal of Clinical Epidemiology, 83, 90–100. https://doi.org/10.1016/j.jclinepi.2016.12.015

    Article  PubMed  Google Scholar 

  10. Norman, G. R., Sloan, J. A., & Wyrwich, K. W. (2003). Interpretation of changes in health-related quality-of-life: The remarkable universality of half a standard deviation. Medical Care, 4, 582–592. https://doi.org/10.1097/01.MLR.0000062554.74615.4C

    Article  Google Scholar 

  11. Wyrwich, K. W., Tierney, W. M., & Wolinsky, F. D. (1999). Further evidence supporting an SEM-based criterion for identifying meaningful intra-individual changes in health-related quality of life. Journal of Clinical Epidemiology, 52, 861–873. https://doi.org/10.1016/s0895-4356(99)00071-2

    Article  CAS  PubMed  Google Scholar 

  12. Hays, R. D., Brodsky, M., Johnston, M. F., Spritzer, K. L., & Hui, K. K. (2005). Evaluating the statistical significance of health-related quality-of-life change in individual patients. Evaluation and the Health Professions, 28(2), 160–171. https://doi.org/10.1177/0163278705275339

    Article  PubMed  Google Scholar 

  13. Crosby, R. D., Kolotkin, R. L., & Williams, G. R. (2003). Defining clinically meaningful change in health-related quality of life. Journal of Clinical Epidemiology, 56(5), 395–407. https://doi.org/10.1016/s0895-4356(03)00044-1

    Article  PubMed  Google Scholar 

  14. Revicki, D., Hays, R. D., Cella, D., & Sloan, J. (2008). Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. Journal of Clinical Epidemiology, 61(2), 102–109. https://doi.org/10.1016/j.jclinepi.2007.03.012

    Article  PubMed  Google Scholar 

  15. Hays, R. D., Farivar, S. S., & Liu, H. (2005). Approaches and recommendations for estimating minimally important differences for health-related quality of life measures. COPD: Journal of Chronic Obstructive Pulmonary Disease, 2, 63–67. https://doi.org/10.1081/copd-200050663

    Article  PubMed  Google Scholar 

  16. de Vet, H. C., Terluin, B., Knol, D. L., Roorda, L. D., Mokkink, L. B., Ostelo, R. W., Hendriks, E. J. M., Bouter, L. M., & Terwee, C. B. (2010). Three ways to quantify uncertainty in individually applied “minimally important change” values. Journal of Clinical Epidemiology, 63(1), 37–45. https://doi.org/10.1016/j.jclinepi.2009.03.011

    Article  PubMed  Google Scholar 

  17. Health Measures. (2022). PROMIS® Score Cut Points. Retrieved April 18, 2022, from: www.healthmeasures.net/score-and-interpret/interpret-scores/promis

  18. Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S., Amtmann, D., Bode, R., Buysse, D., Choi, S., Cook, K., Devellis, R., DeWalt, D., Fries, J. F., Gershon, R., Hahn, E. A., Lai, J. S., Pilkonis, P., Revicki, D., … PROMIS Cooperative Group. (2010). The patient-reported outcomes measurement information system (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005–2008. Journal of Clinical Epidemiology, 63(11), 1179–1194. https://doi.org/10.1016/j.jclinepi.2010.04.011

    Article  PubMed  PubMed Central  Google Scholar 

  19. Iman, R. L., & Conover, W.-J. (1982). A distribution-free approach to inducing rank correlation among input variables. Communications in Statistics—Simulation and Computation, 11(3), 311–334. https://doi.org/10.1080/03610918208812265

    Article  Google Scholar 

  20. Wicklin, R. (2013). Simulating Data with SAS®. SAS Institute Inc.

    Google Scholar 

  21. Coles, T. M., Chen, W., Nelson, L. M., Williams, V. S., Williams, N. J., & McLeod L. D. Current sample size practices in the psychometric evaluation of patient-reported outcome measures for use in clinical trials. Poster presented at the 2014 ISPOR 17th Annual European Congress; November 8-12, 2014. Amsterdam

  22. SAS Institute Inc. (2012). SAS proprietary software, version 9.4. SAS Institute Inc.

    Google Scholar 

  23. Froud, R., & Abel, G. (2014). Using ROC curves to choose minimally important change thresholds when sensitivity and specificity are valued equally: the forgotten lesson of Pythagoras. Theoretical considerations and an example application of change in health status. PLoS ONE, 9(12), e114468. https://doi.org/10.1371/journal.pone.0114468

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Sully, K., Trigg, A., Bonner, N., Moreno-Koehler, A., Trennery, C., Shah, N., Yucel, E., Panjabi, S., & Cocks, K. (2019). Estimation of minimally important differences and responder definitions for EORTC QLQ-MY20 scores in multiple myeloma patients. European Journal of Haematology, 103(5), 500–509. https://doi.org/10.1111/ejh.13316

    Article  PubMed  PubMed Central  Google Scholar 

  25. Musoro, J. Z., Bottomley, A., Coens, C., Eggermont, A. M., King, M. T., Cocks, K., Sprangers, M. A., Groenvold, M., Velikova, G., Flechtner, H. H., Brandberg, Y., EORTC Melanoma Group and EORTC Quality of Life Group. (2018). Interpreting European Organisation for Research and Treatment for Cancer Quality of life Questionnaire core 30 scores as minimally importantly different for patients with malignant melanoma. European Journal of Cancer, 104, 169–181. https://doi.org/10.1016/j.ejca.2018.09.005

    Article  PubMed  Google Scholar 

  26. Webster, K. E., & Feller, J. A. (2021). Evaluation of the responsiveness of the Anterior Cruciate Ligament Return to Sport After Injury (ACL-RSI) scale. Orthopedic Journal of Sports Medicine, 9(8), 23259671211031240. https://doi.org/10.1177/23259671211031240

    Article  Google Scholar 

  27. Zou, H. (2006). The adaptive LASSO and its oracle properties. Journal of the American Statistical Association, 101(476), 1418–1429. https://doi.org/10.1198/016214506000000735

    Article  CAS  Google Scholar 

  28. Morozova, O., Levina, O., Uusküla, A., et al. (2015). Comparison of subset selection methods in linear regression in the context of health-related quality of life and substance abuse in Russia. BMC Medical Research Methodology, 15, 71. https://doi.org/10.1186/s12874-015-0066-2

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  29. Mehta, L., McNeill, M., Hobart, J., Wyrwich, K. W., Poon, J. L., Auguste, P., Zhong, J., & Elkins, J. (2015). Identifying an important change estimate for the Multiple Sclerosis Walking Scale-12 (MSWS-12v1) for interpreting clinical trial results. Multiple Sclerosis Journal—Experimental, Translational, and Clinical, 1, 2055217315596993. https://doi.org/10.1177/2055217315596993

    Article  PubMed  Google Scholar 

  30. Symonds, T., Spino, C., Sisson, M., Soni, P., Martin, M., Gunter, L., & Patrick, D. L. (2007). Methods to determine the minimum important difference for a sexual event diary used by postmenopausal women with hypoactive sexual desire disorder. Journal of Sexual Medicine, 4(5), 1328–1335. https://doi.org/10.1111/j.1743-6109.2007.00562.x

    Article  PubMed  Google Scholar 

  31. Fayers, P. M., & Hays, R. D. (2014). Don’t middle your MIDs: Regression to the mean shrinks estimates of minimally important differences. Quality of Life Research, 23(1), 1–4. https://doi.org/10.1007/s11136-013-0443-4

    Article  PubMed  Google Scholar 

  32. Hopkins, C., Gillett, S., Slack, R., Lund, V. J., & Browne, J. P. (2009). Psychometric validity of the 22-item sinonasal outcome test. Clinical Otolaryngology, 34, 447–454. https://doi.org/10.1111/j.1749-4486.2009.01995.x

    Article  CAS  PubMed  Google Scholar 

  33. Spearman, C. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15(1), 72–101. https://doi.org/10.2307/1412159

    Article  Google Scholar 

Download references

Acknowledgements

The authors sincerely thank Theresa Coles of Duke University’s Department of Population Health Sciences in the early simulation design, and Caitlyn Matuska and John Forbes of RTI Health Solutions for their editorial reviews on this paper.

Funding

The study was supported by RTI Health Solutions.

Author information

Authors and Affiliations

  1. RTI Health Solutions, 3040 Cornwallis Road, Research Triangle Park, NC, 27709, USA

    Shanshan Qin, Lauren Nelson, Valerie Williams, Randall Bender & Lori McLeod

  2. ICF International, 1902 Reston Metro Plaza, Reston, VA, 20190, USA

    Nicole Williams

Authors
  1. Shanshan Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lauren Nelson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicole Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Valerie Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Randall Bender
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lori McLeod
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shanshan Qin.

Ethics declarations

Conflict of interest

SQ, LN, NW, VW, RB, and LM are researchers employed by RTI Health Solutions, which provides clinical outcome assessment development and psychometric evaluation support to pharmaceutical companies.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Consent to participate

Not applicable.

Consent to publish

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 2182 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qin, S., Nelson, L., Williams, N. et al. Comparison of anchor-based methods for estimating thresholds of meaningful within-patient change using simulated PROMIS PF 20a data under various joint distribution characteristic conditions. Qual Life Res 32, 1277–1293 (2023). https://doi.org/10.1007/s11136-022-03285-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11136-022-03285-x

Keywords

Search

Navigation