Psychometric evaluation of the PROMIS Physical Function Computerized Adaptive Test in comparison to the American Shoulder and Elbow Surgeons score and Simple Shoulder Test in patients with rotator cuff disease

doi:10.1016/j.jse.2015.06.025

Journal of Shoulder and Elbow Surgery

Volume 24, Issue 12, December 2015, Pages 1961-1967

https://doi.org/10.1016/j.jse.2015.06.025 Get rights and content

Background

The Patient-Reported Outcomes Measurement Information System (PROMIS) Physical Function Computerized Adaptive Test (PF CAT) is a newly developed patient-reported outcome instrument designed by the National Institutes of Health to measure generalized physical function. However, the measurement properties of the PF CAT have not been compared with established shoulder-specific patient-reported outcomes.

Methods

Patients with clinical diagnosis of rotator cuff disease completed the American Shoulder and Elbow Surgeons (ASES) score, Simple Shoulder Test (SST), and PF CAT. Responses to each of the 3 instruments were statistically analyzed with a Rasch partial credit model. Associations between instruments, convergent validity, item and person reliability, ceiling and floor effects, dimensionality, and survey length were determined.

Results

Responses from 187 patients were analyzed. The PF CAT required fewer questions than the ASES or SST (PF CAT, 4.3; ASES, 11; SST, 12). Correlation between all instruments was moderately high. Item reliability was excellent for all instruments, but person reliability of the PF CAT was superior (0.93, excellent) to the SST (0.71, moderate) and ASES (0.48, fair). Ceiling effects were similar among all instruments (PF CAT, 0.53%; SST, 6.1%; ASES, 2.3%). Floor effects were found in 21% of respondents to the SST but in only 3.2% of PF CAT and 2.3% of ASES respondents.

Conclusion

The measurement properties of the PROMIS PF CAT compared favorably with the ASES and SST despite requiring fewer questions to complete. The PROMIS PF CAT had improved person reliability compared with the ASES score and fewer floor effects compared with the SST.

Section snippets

Participants and data collection

A total of 609 consecutive patients who were seen by a single shoulder and elbow surgeon at a single academic institution completed the ASES, SST, and PF CAT (Appendices A-C) between April 2013 and February 2014 at their initial patient encounter. We obtained Institutional Review Board approval to analyze the data as part of our institution's musculoskeletal outcomes registry before conducting this observational cross-sectional study. Patients were included in the analysis if a clinical

Results

We included 187 patients in the final analysis. There were 106 men and 81 women. Average age of the patients was 60 years. Demographic data are summarized in Table I.

The distribution of responses was evaluated for normalcy by computing the skewness statistic for each instrument: PF CAT (skewness, −0.28; 95% CI, −0.59 to 0.09); ASES (skewness, −0.92; 95% CI, −1.89 to 0.33); and SST (skewness, 0.06; 95% CI, −0.21 to 0.31). Results indicated that all 3 scales have nonsignificant skewness. After

Discussion

The psychometric properties of the PROMIS PF CAT compared favorably with the ASES and SST in this population of patients with rotator cuff disease. The PF CAT required approximately one third of the questions that are used in the ASES and SST yet showed excellent person and item reliability, with few ceiling or floor effects. Relative weaknesses of the ASES in this study included its comparatively lower person reliability; relative weaknesses of the SST in this study included a large floor

Conclusion

The psychometric properties of the PROMIS PF CAT compared favorably with the ASES and SST in this population of patients with rotator cuff disease. The PF CAT required fewer questions than the ASES and SST, and it showed excellent reliability and unidimensionality with fewer ceiling and floor effects.

Acknowledgment

This investigation was supported by the University of Utah Department of Orthopaedics and the Center for Outcomes Research and Assessment, with funding in part from the National Institute of Arthritis and Musculoskeletal and Skin Diseases of the National Institutes of Health under Award Number U01AR067138. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Study data were collected and managed using

Disclaimer

The authors, their immediate families, and any research foundation with which they are affiliated have not received any financial payments or other benefits from any commercial entity related to the subject of this article.

References (28)

D. Amtmann et al.
Development of a PROMIS item bank to measure pain interference
Pain
(2010)
J.C. Cappelleri et al.
Overview of classical test theory and item response theory for the quantitative assessment of items in developing patient-reported outcomes measures
Clin Ther
(2014)
G.M. Gartsman et al.
Self-assessment of general health status in patients with five common shoulder conditions
J Shoulder Elbow Surg
(1998)
J. Godfrey et al.
Reliability, validity, and responsiveness of the simple shoulder test: psychometric properties by age and injury type
J Shoulder Elbow Surg
(2007)
C. Jepson et al.
In a mailed physician survey, questionnaire length had a threshold effect on response rate
J Clin Epidemiol
(2005)
L.A. Michener et al.
American Shoulder and Elbow Surgeons Standardized Shoulder Assessment Form, patient self-report section: reliability, validity, and responsiveness
J Shoulder Elbow Surg
(2002)
M. Rose et al.
The PROMIS Physical Function item bank was calibrated to a standardized metric and shown to improve measurement efficiency
J Clin Epidemiol
(2014)
G.P. Slobogean et al.
Measuring shoulder injury function: common scales and checklists
Injury
(2011)
A.R. Tyser et al.
Evaluation of the PROMIS physical function computer adaptive test in the upper extremity
J Hand Surg Am
(2014)
D. Bryant et al.
A comparison of pain, strength, range of motion, and functional outcomes after hemiarthroplasty and total shoulder arthroplasty in patients with osteoarthritis of the shoulder. A systematic review and meta-analysis
J Bone Joint Surg Am
(2005)

D. Cella et al.

A discussion of item response theory and its applications in health status assessment

Med Care

(2000)

K.F. Cook et al.

Dynamic assessment of health outcomes: time to let the CAT out of the bag?

Health Serv Res

(2005)

C.O. Davidshofer et al.

Psychological testing: principles and applications

(2005)

J.F. Fries et al.

Item response theory, computerized adaptive testing, and PROMIS: assessment of physical function

J Rheumatol

(2014)

Cited by (120)

Many patients fail to achieve MCID for PROMIS upper extremity and pain interference following nonoperative management of rotator cuff tears
2023, JSES International
Efficacy of nonoperative treatment for rotator cuff tears has been debated, especially for full-thickness tears. The purpose of this study was to a) define the minimal clinically important difference (MCID) of nonoperative treatment with regard to Patient-Reported Outcomes Measurement Information System (PROMIS) pain interference (PI) and upper extremity (UE), and b) determine the proportion of patients with both partial and full-thickness tears (PTRCT, FTRCT) who achieve this improvement following initial nonoperative treatment. We hypothesized that >75% of PTRCT and FTRTC patients would achieve MCID for PROMIS PI and UE.
We performed a retrospective cohort study evaluating nonoperatively managed patients with image-confirmed PTRCT and FTRCT. Treatment modalities and follow-up PROMIS scores at least 6 months after their initial visit were recorded. Using a distribution technique, MCID was calculated.
A total of 111 FTRCT and 68 PTRCT patients were included with at least 6 months of follow-up. At 6 months from initial presentation, the MCID for PROMIS UE was 3.75 and 3.95 for FTRCT and PTRCT patients, respectively. For PROMIS PI, the MCID was 3.35 and 3.90 for FTRCT and PTRCT, respectively. In total, 41% of FTRCT and 41% of PTRCT achieved MCID for PROMIS UE. Thirty-four percent of FTRCT and 35% of PTRCT achieved MCID for PROMIS PI.
The majority of patients undergoing nonoperative treatment for supraspinatus/infraspinatus rotator cuff tears did not achieve MCID at 6 months for PROMIS PI (34% for FTRCT and 35% for PTRCT) or UE (41% for FTRCT and 41% for PTRCT).
Effects of increased body mass index on one year outcomes following soft tissue arthroscopic shoulder instability repair
2023, JSES International
The purpose of this study was to investigate the impact of high body mass index on the 1-year minimal outcome following arthroscopic shoulder stabilization.
Patients who underwent arthroscopic Bankart repair (ABR) between 2017 and 2021 were identified and assigned to 1 of 3 cohorts based on their preoperative body mass index: normal (18-25), overweight (25-30), and obese (>30). The primary outcomes assessed were postoperative shoulder instability and revision rates. The 3 groups were compared using the Patient-Reported Outcomes Measurement Information System (PROMIS) upper extremity, pain interference, pain intensity, Clinical Global Impression scores, visual analog scale pain scores, and shoulder range of motion at 1 year postoperatively.
During the study period, 142 patients underwent ABR and had an average age of 35 ± 10 years. Obese patients had a higher percentage of partial rotator cuff tears (60% vs. 27%, odds ratio: 3.2 [1.1, 9.2]; P = .009), longer mean operative time (99.8 ± 40.0 vs. 75.7 ± 28.5 minutes; P < .001), and shorter time to complication (0.5 ± 0 vs. 7.0 ± 0 months; P = .038). After controlling for confounding factors, obesity was associated with a lesser improvement in upper extremity function scores (obese vs. normal: −4.9 [−9.4, −0.5]; P = .029); although this difference exists, found future studies are needed to determine the clinical significance. There were no differences in patient reported outcome measures, recurrence rate, or revision surgery rates between cohorts at any time point (P > .05).
Obesity is an independent risk factor for longer operative times but does not confer a higher risk of recurrent instability, revision surgery, or lower outcome scores 1 year following ABR.
Editorial Commentary: Legacy Patient-Reported Outcome Measures Remain Important Today Despite Responder Burden, but With Further Refinement, Patient-Reported Outcomes Measurement Information System Could Replace Legacy Instruments in the Future
2023, Arthroscopy - Journal of Arthroscopic and Related Surgery
Measuring patient-reported outcomes (PROs) is a key tenet in the drive toward value-based care. A more detailed understanding of outcomes has traditionally been obtained with increasingly frequent and more in-depth patient questionnaires. Legacy PROs require patients to complete a predetermined set of questions. Administering lengthy surveys repeatedly can generate respondent fatigue that compromises quality of the survey responses and overall compliance. In addition, these legacy scales have notable ceiling and floor effects, which limit their capacity to describe the condition of patients who are functioning at relatively high or low levels. Modern computer adaptive instruments, such as PROMIS, strive to minimize responder burden and mitigate floor and ceiling effects associated with many legacy instruments. However, although these new newer instruments correlate (moderately) with legacy scores, PROMIS also shows floor and ceiling effects, and PROMIS responsiveness is currently understudied. Today, we collect both PROMIS and legacy scores for tracking patient outcomes and for research purposes. We predict that with further refinement, PROMIS will replace legacy patient-reported outcome measures.
Race and Socioeconomic Status Are Associated With Inferior Patient-Reported Outcome Measures Following Rotator Cuff Repair
2023, Arthroscopy - Journal of Arthroscopic and Related Surgery
To investigate the impact social determinants of health (SDOH) have on National Institutes of Health Patient-Reported Outcomes Measurement Information System (PROMIS) computer adaptive test scores and postoperative health care use in patients who undergo rotator cuff repair (RCR).
All patients who underwent RCR surgery by 3 shoulder and/or sports medicine fellowship-trained orthopaedic surgeons between July 2017 and January 2020 were included. The electronic medical record (EMR) was used to identify SDOH for each patient. PROMIS computer adaptive test measures of Upper Extremity function, Pain Interference, and Depression were completed preoperatively and postoperatively (6 months and 1 year). Postoperative health care use (clinical visits, virtual encounters, imaging encounters, and physical therapy visits) were recorded as well. Univariate associations, multiple linear regressions, and Wilcoxon rank-sum tests were used to analyze mean differences between patient groups based on SDOH.
Three hundred thirty-eight patients who underwent RCR were included. Patients who were Black, in lower median household income quartiles, had public insurance, and female reported lower PROMIS scores compared with their counterparts. Smokers and White patients attended fewer postoperative office visits whereas Black patients had more physical therapy and nonvisit encounters compared with their respective counterparts.
Black race and lower socioeconomic status are associated with worse function and pain outcomes post-RCR compared with White race. Similarly, Black race and positive smoking status are associated with differential use of health care following RCR. Further attention may be required for these patients to address health care disparities.
III, retrospective cohort study.
Correlation of patient-reported outcome measurement information system-upper extremity with American Shoulder and Elbow Surgeons scores in shoulder arthroplasty
2022, Seminars in Arthroplasty JSES
Multiple outcome measure surveys are available to assess preoperative and postoperative outcomes for a variety of orthopedic procedures such as shoulder arthroplasty. Although legacy instruments such as American Shoulder and Elbow Surgeons (ASES) remain popular, there remains no singular gold standard survey instrument for shoulder arthroplasty patients, and alternative instruments have been developed to better capture clinical outcomes. The goal of this study is to compare the efficacy of Patient-Reported Outcome Measurement Information System-Upper Extremity (PROMIS-UE) using computer adaptive technology with ASES scores both preoperatively and postoperatively in shoulder arthroplasty patients. Our hypothesis is that there would be a strong correlation between PROMIS-UE and ASES scores both preoperatively and postoperatively in total shoulder arthroplasty patients.
Patients who underwent total shoulder arthroplasty and agreed to complete baseline and 1-year follow-up of ASES and PROMIS-UE scores were included. Patients also completed PROMIS-Physical Function (PROMIS-PF) as a third point comparison. Responses to these instruments were statistically analyzed and compared using Pearson correlation coefficients. Floor and ceiling effects were then calculated.
Ninety patients were included in this study, all of whom completed the PROMIS-UE, PROMIS-PF, and ASES surveys both preoperatively and 12 months postoperatively. The mean age in this cohort was 68.9 years (standard deviation [SD] 8.4 years, range 39-89). The mean preoperative and postoperative PROMIS-UE scores were 26.6 (SD 6.7, range 14.7-44.6) and 41.8 (SD 10.3, range 20.2-56.4) respectively. Mean preoperative and postoperative PROMIS-PF computer adaptive technology scores were 35.7 (SD 9.54, range 34.2-64) and 44.3 (SD 9.22, range 23.5-73.3) respectively. The mean ASES score was 37.5 preoperatively (SD 18.0, range 5-99.5) and 77.5 postoperatively (SD 20.5, range 25.0-100.0). PROMIS-UE demonstrated a moderate correlation with ASES preoperatively and a strong correlation postoperatively (r = 0.52, confidence interval 0.27-0.60; r = 0.70, confidence interval 0.55-0.78, respectively). PROMIS-UE demonstrated a minor floor effect preoperatively (7.8%) but significant ceiling effect postoperatively (24.4%) and ASES demonstrated a mild ceiling effect at final follow-up (8.9%). There were otherwise no other floor or ceiling effects at all other time points across each survey.
PROMIS-UE correlates well with ASES at both baseline and 1-year postoperation for patients undergoing total shoulder replacement. In addition, the change seen between both baseline and 1 year outcome scores for both scoring systems also correlates strongly, suggesting that PROMIS-UE may be a suitable alternative to ASES for this patient population.
Risk factors for failure to achieve minimal clinically important difference and significant clinical benefit in PROMIS computer adaptive test domains in patients undergoing rotator cuff repair
2022, Journal of Shoulder and Elbow Surgery
The Patient-Reported Outcomes Measurement Information System (PROMIS) has emerged as a valid and efficient means of collecting outcomes in patients with rotator cuff tears. The purpose of this study was to establish threshold score changes to determine minimal clinically important difference (MCID) and substantial clinical benefit (SCB) in PROMIS computer adaptive test (CAT) scores following rotator cuff repair (RCR). Additionally, we sought to identify potential risk factors for failing to achieve MCID and SCB.
Patients undergoing arthroscopic RCR were identified over a 24-month period. Only patients who completed both preoperative and postoperative PROMIS CAT assessments were included in this cohort. PROMIS CAT forms for upper extremity physical function (PROMIS-UE), pain interference (PROMIS-PI), and depression (PROMIS-D) were used with a minimum of 1.5-year follow-up. Statistical analysis was performed to determine threshold score changes to determine anchor-based MCID and SCB, as well as risk factors for failure to achieve significant clinical improvement following surgery.
Of 198 eligible patients, 168 (84.8%) were included in analysis. ΔPROMIS-UE values of 5.8 and 9.7 (area under the curve [AUC] = 0.906 and 0.949, respectively) and ΔPROMIS-PI values of –11.4 and –12.9 (AUC = 0.875 and 0.938, respectively) were identified as threshold predictors of MCID and SCB achievement. On average, 81%, 65%, and 55% of patients achieved MCID for PROMIS-UE, PROMIS-PI, and PROMIS-D whereas 71%, 61%, and 38% of patients in the cohort, respectively, achieved SCB. MCID achievement in PROMIS-UE significantly differed according to risk factors, including smoking status (likelihood ratio [LR]: 9.8, P = .037), tear size (LR: 10.4, P < .001), distal clavicle excision (LR: 6.1, P = .005), and prior shoulder surgery (LR: 19.2, P < .001). Factors influencing SCB achievement for PROMIS-UE were smoking status (LR: 9.3, P = .022), tear size (LR: 8.0, P = .039), and prior shoulder surgery (11.9, P < .001). Significantly different rates of MCID and SCB achievement in PROMIS-PI for smoking status (LR: 7.0, P = .030, and LR: 5.2, P = .045) and prior shoulder surgery (LR: 9.1, P = .002, and LR: 7.4, P = .006) were also identified.
The majority of patients showed clinically significant improvements that exceeded the established MCID for PROMIS-UE and PROMIS-PI following RCR. Patients with larger tear sizes, a history of prior shoulder surgery, tobacco users, and those who received concomitant distal clavicle excision were at risk for failing to achieve MCID in PROMIS-UE. Additionally, smokers and patients who underwent prior shoulder surgery demonstrated significantly lower improvements in pain scores following surgery.

View all citing articles on Scopus

: Institutional Review Board approval was obtained from the University of Utah School of Medicine before initiation of this study: No. 00071740.

View full text

Basic SciencePsychometric evaluation of the PROMIS Physical Function Computerized Adaptive Test in comparison to the American Shoulder and Elbow Surgeons score and Simple Shoulder Test in patients with rotator cuff disease

Background

Methods

Results

Conclusion

Section snippets

Participants and data collection

Results

Discussion

Conclusion

Acknowledgment

Disclaimer

Pain

Clin Ther

J Shoulder Elbow Surg

J Shoulder Elbow Surg

J Clin Epidemiol

J Shoulder Elbow Surg

J Clin Epidemiol

Injury

J Hand Surg Am

A comparison of pain, strength, range of motion, and functional outcomes after hemiarthroplasty and total shoulder arthroplasty in patients with osteoarthritis of the shoulder. A systematic review and meta-analysis

J Bone Joint Surg Am

A discussion of item response theory and its applications in health status assessment

Med Care

Dynamic assessment of health outcomes: time to let the CAT out of the bag?

Health Serv Res

Psychological testing: principles and applications

Item response theory, computerized adaptive testing, and PROMIS: assessment of physical function

J Rheumatol

Basic Science
Psychometric evaluation of the PROMIS Physical Function Computerized Adaptive Test in comparison to the American Shoulder and Elbow Surgeons score and Simple Shoulder Test in patients with rotator cuff disease