Abstract
Purpose
Sociodemographic characteristics may influence responses on self-reported measures. Differential item functioning (DIF) is when individuals expected to have the same ability level on a construct of interest have a different probability of endorsing an item on an item response theory (IRT) scale due to population characteristics. The goal of this study was to identify DIF for items in an outcome instrument by sociodemographic factors and, one controlling for DIF, assess true differences in function by those same factors.
Methods
The Work Disability Functional Assessment Battery 2.0 (WD-FAB 2.0) is an IRT-based self-reported measure of activity limitations relevant to work. Two samples from WD-FAB developed were used: 3793 SSA disability claimants randomly drawn from a pool of 16,500 claimants and a general sample if 2100 working age adults. We used a two-step IRT-based DIF method for three pairs of respondent characteristics: age, gender, and race/ethnicity, and calculated the weighted absolute difference between item characteristic curves. Independent two-group T-tests assessed differences in scores across groups.
Results
Seventeen items displayed DIF. Men had higher scores than women on two physical and two mental function scales. Older respondents had lower physical and higher mental function scores. The lower education group had lower mental function scores.
Conclusion
DIF impacts function measurement and is important when assessing psychometric characteristics of instruments. Self-report measures should include diverse samples to conduct similar analyses. WD-FAB 2.0 scores are now reflections of function with reduced bias related to gender, race/ethnicity, or age.
Similar content being viewed by others
References
Social Security Administration. (2017). Annual statistical report on the social security disability insurance program.
Brandt, D. E., Houtenville, A. J., Huynh, M. T., Chan, L., & Rasch, E. K. (2011). Connecting contemporary paradigms to the social security administration’s disability evaluation process. Journal of Disability Policy Studies. https://doi.org/10.1177/1044207310396509.
Lord, F. M. (1980). Applications of item response theory to practical testing problems. Abingdon: Routledge.
Rivers D. (2006). Sample matching: Representative sampling from internet panels. Polimetrix White Paper Series.
Reeve, B. B., Hays, R. D., Bjorner, J. B., et al. (2007). Psychometric evaluation and calibration of health-related quality of life item banks: Plans for the patient-reported outcomes measurement information system (PROMIS). Medical Care, 45(5 Suppl 1), S22-31. https://doi.org/10.1097/01.mlr.0000250483.85507.04.
Norris, J. M. (2001). Computer-adaptive testing: A primer. Language Learning & Technology., 5(2), 23–27.
Warm, T. A. (1989). Weighted likelihood estimation of ability in item response theory. Psychometrika., 54(3), 427–450.
Wang, S., & Wang, T. (2001). Precision of warm’s weighted likelihood estimates for a polytomous model in computerized adaptive testing. Applied Psychological Measurement, 25(4), 317–331.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika monograph supplement.
Hambleton, R., & Pitoniak, M. (2002). Testing and measurement: Advances in item response theory and selected testing practices.
Meterko, M., Marfeo, E. E., McDonough, C. M., et al. (2014). The work disability functional assessment battery (WD-FAB): Feasibility and psychometric properties. Archives of Physical Medicine and Rehabilitation. https://doi.org/10.1016/j.apmr.2014.11.025.
Ni, P., McDonough, C. M., Jette, A. M., et al. (2013). Development of a computer-adaptive physical function instrument for social security administration disability determination. Arch Phys Med Rehabil., 94(9), 1661–1669.
Marfeo, E. E., Ni, P., Haley, S. M., et al. (2013). Development of an instrument to measure behavioral health function for work disability: Item pool construction and factor analysis. Archives of Physical Medicine and Rehabilitation, 94(9), 1670–1678.
McDonough, C. M., Ni, P., Peterik, K., Marfeo, E. E., Marino, M. E., Meterko, M., … & Chan, L. (2017). Improving measures of work-related physical functioning. Quality of life research, 26(3), 789–798.
Marfeo, E. E., Ni, P., McDonough, C., Peterik, K., Marino, M., Meterko, M., … & Jette, A. M. (2018). Improving assessment of work related mental health function using the work disability functional assessment battery (WD-FAB). Journal of Occupational Rehabilitation, 28(1), 190–199.
Meterko, M., Marino, M., Ni, P., Marfeo, E., McDonough, C. M., Jette, A., … & Chan, L. (2019). Psychometric evaluation of the improved work-disability functional assessment battery. Archives of Physical Medicine and Rehabilitation, 100(8), 1442–1449.
Jette, A. M., Ni, P., Rasch, E., Marfeo, E., McDonough, C., Brandt, D., … & Chan, L. (2019). The Work Disability Functional Assessment Battery (WD-FAB). Physical Medicine and Rehabilitation Clinics.
Coe, N. B., Haverstick, K., Munnell, A. H., & Webb, A. (2011). What explains state variation in SSDI application rates? Center for Retirement Research at Boston College Working Paper. (2011-23).
Coe, N. B., Haverstick, K., Munnell, A. H., & Webb, A. (2012). Why do state disability application rates vary over time? Center for Retirement Research at Boston College Working Paper. 2012(12-2).
Von Wachter, T., Song, J., & Manchester, J. (2011). Trends in employment and earnings of allowed and rejected applicants to the social security disability insurance program. American Economic Review, 107, 3308–3329.
Wilson, K. B. (2000). Predicting vocational rehabilitation acceptance based on race, education, work status, and source of support at application. Rehabilitation Counseling Bulletin, 43(2), 97–105.
Langer, M. M. (2008). A reexamination of Lord's Wald test for differential item functioning using item response theory and modern error estimation.
Woods, C. M., Cai, L., & Wang, M. (2013). The langer-improved wald test for DIF testing with multiple groups evaluation and comparison to two-group IRT. Educational and Psychological Measurement, 73(3), 532–547.
Edelen, M. O., Stucky, B. D., & Chandra, A. (2013). Quantifying ‘problematic’ DIF within an IRT framework: Application to a cancer stigma index. Quality of Life Research, 24, 1–9.
Oshima, T., Kushubar, S., Scott, J., & Raju, N. (2009). DFIT8 for window user’s manual: Differential functioning of items and tests. St. Paul, MN: Assessment Systems Corporation.
Rice, M. E., & Harris, G. T. (2005). Comparing effect sizes in follow-up studies: ROC area, Cohen’s d, and r. Law and Human Behavior, 29(5), 615.
Abel, J. R., Gabe, T. M., & Stolarick, K. (2012). Workforce skills across the urban-rural hierarchy. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2010646.
Autor, D. (2010). The polarization of job opportunities in the US labor market: Implications for employment and earnings. Center for American Progress and The Hamilton Project.
Lin, S., Beck, A. N., Finch, B. K., Hummer, R. A., & Master, R. K. (2012). Trends in US older adult disability: Exploring age, period, and cohort effects. American Journal of Public Health, 102(11), 2157–2163.
Loprest, P., & Maag, E. (2007). The relationship between early disability onset and education and employment. Journal of Vocational Rehabilitation, 26(1), 49–62.
Hopman, W., Harrison, M., Coo, H., Friedberg, E., Buchanan, M., & Van Den Kerkhof, E. (2009). Associations between chronic disease, age and physical and mental health status. Chronic Diseases in Canada, 29(3), 108–116.
Ward, B. W. (2013). Prevalence of multiple chronic conditions among US adults: Estimates from the national health interview survey, 2010. Preventing Chronic Disease. https://doi.org/10.5888/pcd10.120203.
Haveman, R., & Wolfe, B. (2000). The economics of disability and disability policy. Handbook of Health Economics, 1, 995–1051.
Duggan, M, & Imberman, S. A. (2009). Why are the disability rolls skyrocketing? The contribution of population characteristics, economic conditions, and program generosity. In: Health at older ages: The causes and consequences of declining disability among the elderly. Chicago: University of Chicago Press, pp. 337–379.
Hendley, A. A., & Bilimoria, N. F. (1999). Minorities and social security: An analysis of ethnic differences in the current program. Social Security Bulletin, 62, 59.
Kington, R. S., & Smith, J. P. (1997). Socioeconomic status and racial and ethnic differences in functional status associated with chronic diseases. American Journal of Public Health, 87(5), 805–810.
Williams, D. R. (1999). Race, socioeconomic status, and health the added effects of racism and discrimination. Annals of the New York Academy Sciences, 896(1), 173–188.
Nichols, A. & Simms, M. (2012). Racial and ethnic differences in receipt of unemployment insurance benefits during the great recession. Unemployment and Recovery Project Brief.
Kao, G., & Thompson, J. S. (2003). Racial and ethnic stratification in educational achievement and attainment. Annual Review of Sociology, 29(1), 417–442.
DeBose, C. E. (1992). Codeswitching: Black English and standard English in the African-American linguistic repertoire. Journal of Multilingual & Multicultural Development, 13(1–2), 157–167.
Fleishman, J. A., Spector, W. D., & Altman, B. M. (2002). Impact of differential item functioning on age and gender differences in functional disability. The Journals of Gerontology, Series B: Psychological Sciences and Social Sciences, 57(5), S275-84.
Sullivan, O. (1997). Time waits for no (wo) man: An investigation of the gendered experience of domestic time. Sociology, 31(2), 221–239.
Teresi, J. A., & Fleishman, J. A. (2007). Differential item functioning and health assessment. Quality of Life Research, 16(1), 33–42.
Teresi, J. A., Ocepek-Welikson, K., Kleinman, M., et al. (2009). Analysis of differential item functioning in the depression item bank from the patient reported outcome measurement information system (PROMIS): An item response theory approach. Psychological Science Quarterly, 51(2), 148–180.
Rose, M., Bjorner, J. B., Becker, J., Fries, J., & Ware, J. (2008). Evaluation of a preliminary physical function item bank supported the expected advantages of the patient-reported outcomes measurement information system (PROMIS). Journal of Clinical Epidemiology, 61(1), 17–33.
Iwata, N., Turner, R. J., & Lloyd, D. A. (2002). Race/ethnicity and depressive symptoms in community-dwelling young adults: A differential item functioning analysis. Psychiatry Research, 110(3), 281–289.
Gao, Y., & Zhu, W. (2011). Differential item functioning analysis of the 2003–04 NHANES physical activity questionnaire. Research Quarterly for Exercise and Sport, 82(3), 381–390.
Hafner-Eaton, C. (1993). Physician utilization disparities between the uninsured and insured: Comparisons of the chronically III, acutely III, and well nonelderly populations. JAMA, 269(6), 787–792.
Williams, D. R., & Collins, C. (1995). US socioeconomic and racial differences in health: Patterns and explanations. Annual Review of Sociology, 21, 349–386.
Willimas, D., Yu, Y., & Jackson, J. (1997). Racial differences in physical and mental health. Journal of Health Psychology, 2, 335–351.
Yee, S. (2011). Health and health care disparities among people with disabilities. Washington, DC: Disability Rights Education & Defense Fund.
Fremstad, S. (2009). Half in ten. Why taking disability into account is essential to reducing income poverty and expanding economic inclusion. Washington, DC: Center for Economic and Policy Research.
Hanmer, J., Lawrence, W. F., Anderson, J. P., Kaplan, R. M., & Fryback, D. G. (2006). Report of nationally representative values for the noninstitutionalized US adult population for 7 health-related quality-of-life scores. Medical Decision Making, 26(4), 391–400.
Milner, A., LaMontagne, A., Aitken, Z., Bentley, R., & Kavanagh, A. (2014). Employment status and mental health among persons with and without a disability: Evidence from an Australian cohort study. Journal of Epidemiology and Community Health. https://doi.org/10.1136/jech-2014-204147.
McFarland, M. J., & Wagner, B. G. (2015). Does a college education reduce depressive symptoms in American young adults? Social Science & Medicine, 146, 75–84.
Kessler, R. C., Amminger, G. P., Aguilar-Gaxiola, S., Alonso, J., Lee, S., & Ustun, T. B. (2007). Age of onset of mental disorders: A review of recent literature. Current Opinion in Psychiatry, 20(4), 359–364. https://doi.org/10.1097/YCO.0b013e32816ebc8c.
Lee, M., & Mather, M. (2008) US labor force trends. Vol 63. Population Reference Bureau.
Bauer, D. J., Belzak, W. C. M., & Cole, V. T. (2020). Simplifying the assessment of measurement invariance over multiple background variables: using regularized moderated nonlinear factor analysis to detect differential item functioning. Structural Equation Modeling, 27(1), 43–55. https://doi.org/10.1080/10705511.2019.1642754.
Bauer, D. J. (2017). A more general model for testing measurement invariance and differential item functioning. Psychological Methods, 22(3), 507–526. https://doi.org/10.1037/met0000077.
Gottfredson, N. C., Cole, V. T., Giordano, M. L., Bauer, D. J., Hussong, A. M., & Ennetta, S. T. (2019). Simplifying the implementation of modern scale scoring methods with an automated R package: Automated moderated nonlinear factor analysis (aMNLFA). Addictive Behaviors, 94, 65–73. https://doi.org/10.1016/j.addbeh.2018.10.031.
Wu, X., Sawatzky, R., Hopman, W., Mayo, N., Sajobi, T. T., Liu, J., et al. (2017). Latent variable mixture models to test for differential item functioning: a population-based analysis. Health and Quality of Life Outcomes, 15, 102. https://doi.org/10.1186/s12955-017-0674-0.
DeMars, C. E., & Lau, A. (2011). Differential item functioning detection with latent classes: How accurately can we detect who is responding differentially? Educational and Psychological Measurement, 71(4), 597–616. https://doi.org/10.1177/0013164411404221.
Ma, S., Huang, J., Zhang, Z., & Liu, M. (2019). Exploration of heterogeneous treatment effects via concave fusion. The International Journal of Biostatistics, 16(1), 20180026. https://doi.org/10.1515/ijb-2018-0026.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Marino, M., Ni, P., Kazis, L. et al. Demographic and functional differences among social security disability claimants. Qual Life Res 30, 1757–1768 (2021). https://doi.org/10.1007/s11136-021-02765-w
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11136-021-02765-w