Skip to main content
SpringerLink
Log in

A sequential Moken scaling approach to evaluate response quality in survey research

  • Published:
Behavior Research Methods Aims and scope Submit manuscript

Abstract

Careless responding, where participants do not fully engage with item content, is pervasive in survey research. Left undetected, carelessness can compromise the interpretation and use of survey results, including information about participant locations on the construct, item difficulty, and the psychometric quality of the instrument. We present and illustrate a sequential procedure for evaluating response quality in survey research using indicators from Mokken scale analysis (MSA). We use a real data illustration and a simulation study to compare a sequential procedure to a standalone procedure. We also consider how identifying and removing responses with evidence of poor measurement properties affects item quality indicators. Results suggest that the sequential procedure was effective in identifying potentially problematic response patterns that may not always be captured by traditional methods for identifying careless responders but was not always sensitive to specific carelessness patterns. We discuss implications for research and practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Bandalos, D. L. (2018). Measurement Theory and Applications for the Social Sciences.

  • Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19. https://doi.org/10.1016/j.jesp.2015.07.006

    Article  Google Scholar 

  • Curran, P. G., & Hauser, K. A. (2019). I’m paid biweekly, just not by leprechauns: Evaluating valid-but-incorrect response rates to attention check items. Journal of Research in Personality, 82, 103849.

    Article  Google Scholar 

  • DeAyala, R. J. (2009). The theory and practice of item response theory. The Guilford Press.

    Google Scholar 

  • Eisele, G., Vachon, H., Lafit, G., Kuppens, P., Houben, M., Myin-Germeys, I., & Viechtbauer, W. (2022). The effects of sampling frequency and questionnaire length on perceived burden, compliance, and careless responding in experience sampling data in a student population. Assessment, 29(2), 136–151.

    Article  PubMed  Google Scholar 

  • Ferrando, P. J. (2007). Factor-analytic procedures for assessing person scalability. Multivariate Behavioral Research, 42, 481–507.

    Article  Google Scholar 

  • Goldammer, P., Annen, H., Stöckli, P. L., & Jonas, K. (2020). Careless responding in questionnaire measures: Detection, impact, and remedies. The Leadership Quarterly, 31(4), 101384. https://doi.org/10.1016/j.leaqua.2020.101384

    Article  Google Scholar 

  • Koopman, L., Zijlstra, B. J. H., & Van Der Ark, L. A. (2020). New advances in Mokken scale analysis Annual Meeting of the Psychometric Society.

    Google Scholar 

  • Meijer, R. R. (1994). The number of Guttman errors as a simple and powerful person-fit statistic. Applied Psychological Measurement, 18(4), 311–314. https://doi.org/10.1177/014662169401800402

    Article  Google Scholar 

  • Meijer, R. R., & Baneke, J. J. (2004). Analyzing psychopathology items: A case for nonparametric item response theory modeling. Psychological Methods, 9(3), 354–368. https://doi.org/10.1037/1082-989X.9.3.354

    Article  PubMed  Google Scholar 

  • Meijer, R. R., Tendeiro, J. N., & Wanders, R. B. K. (2015). The use of nonparametric item response theory to explore data quality. In S. P. Reise & D. A. Revicki (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 85–110). Routledge.

    Google Scholar 

  • Mokken, R. J. (1971). A theory and procedure of scale analysis. De Gruyter.

  • Molenaar, I. W. (1997). Nonparametric models for polytomous responses. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 369–380). Springer.

    Chapter  Google Scholar 

  • Muraki, E., & Muraki, M. (2018). Generalized partial credit model. In W. J. van der Linden (Ed.), Handbook of item response theory (Vol. 1, pp. 127–138). CRC Press.

    Google Scholar 

  • Niessen, A. S. M., Meijer, R. R., & Tendeiro, J. N. (2016). Detecting careless respondents in web-based questionnaires: Which method to use? Journal of Research in Personality, 63, 1–11. https://doi.org/10.1016/j.jrp.2016.04.010

    Article  Google Scholar 

  • Patton, J. M., Cheng, Y., & Diao, Q. (2019). Detection and treatment of careless responses to improve item parameter estimation. Journal of Educational and Behavioral Statistics, 44(3), 309–341. https://doi.org/10.3102/1076998618825116

    Article  Google Scholar 

  • R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

  • Ryan, R. M., & Deci, E. L. (2020). Intrinsic and extrinsic motivation from a self-determination theory perspective: Definitions, theory, practices, and future directions. Contemporary Educational Psychology, 61, 101860. https://doi.org/10.1016/j.cedpsych.2020.101860

    Article  Google Scholar 

  • Silvia, P. J., & Rodriguez, R. M. (2020). Time to renovate the humor styles questionnaire? An item response theory analysis of the HSQ. Behavioral Sciences, 10(11), 173. https://doi.org/10.3390/bs10110173

    Article  PubMed  PubMed Central  Google Scholar 

  • Straat, J. H., van der Ark, L. A., & Sijtsma, K. (2013). Comparing optimization algorithms for item selection in Mokken scale analysis. Journal of Classification, 30(1), 75–99. https://doi.org/10.1007/s00357-013-9122-y

    Article  Google Scholar 

  • Tendeiro, J. N., & Meijer, R. R. (2014). Detection of invalid test scores: The usefulness of simple nonparametric statistics: Detection of invalid test scores. Journal of Educational Measurement, 51(3), 239–259. https://doi.org/10.1111/jedm.12046

    Article  Google Scholar 

  • van Schuur, W. H. (2011). Ordinal Item Response Theory: Mokken Scale Analysis. Sage.

  • Vriesema, C. C., & Gehlbach, H. (2021). Assessing survey satisficing: The impact of unmotivated questionnaire responding on data quality. Educational Researcher, 50(9), 618–627. https://doi.org/10.3102/0013189X211040054

    Article  Google Scholar 

  • Walker, A. A., & Wind, S. A. (2020). Identifying Misfitting achievement estimates in performance assessments: An illustration using Rasch and Mokken scale analyses. International Journal of Testing, 20(3), 231–251. https://doi.org/10.1080/15305058.2019.1673758

    Article  Google Scholar 

  • Wang, Y., & Wind, S. A. (2020). Psychometric evaluation of a new internalization of learning motivation (ILM) scale. Current Psychology. https://doi.org/10.1007/s12144-020-00909-8

  • Wind, S. A., & Wang, Y. (2022). Using Mokken scaling techniques to explore carelessness in survey research. Behavior Research Methods, Online First. https://doi.org/10.3758/s13428-022-01960-y

Download references

Author information

Authors and Affiliations

  1. Educational Measurement, Educational Studies in Psychology, Research Methodology, and Counseling, The University of Alabama, Box 870231, Tuscaloosa, AL, 35487, USA

    Stefanie A. Wind, Benjamin Lugu & Yurou Wang

Authors
  1. Stefanie A. Wind
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Benjamin Lugu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yurou Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefanie A. Wind.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open practices statement

The authors do not have permission to share the real data used in this study.

Code used for the analyses with an example is available at the following URL:

https://alabama.box.com/s/kpwqin2xfqp5uhl40did8v6oij85enxe

This study was not preregistered.

Appendix A

Appendix A

Item number

Item stem

8

Studying is hardly ever exciting

12

I learn in school because my parents say I have to

14

I learn in school because my teachers say I have to

15

I would not study if my teachers did not make me do it

19

If my parents do not push me, I would not push myself to learn in school

37

I study hard to avoid my parents scolding me

38

I study to avoid being criticized by my parents

39

I study to avoid being criticized by my parents

40

If I do not study hard, my parents will punish me

60

I study so I will not look incompetent in front of others

66

I work on homework, so my classmates will think I am smart

68

I study because I want my teacher to think I am smart

83

I study because I would feel bad about myself if got a bad grade

84

I feel guilty if I do not learn something well

89

I feel ashamed if I do not get a good grade on an exam or homework assignment

97

I study in school because I personally value what I learn

102

It is important to me that I study regularly/consistently

107

I am motivated to learn because I find the content meaningful

113

I study because I am passionate about learning

118

I am motivated to learn in school because it teaches me how to solve problems

121

I study because it helps me figure out a purpose in life

123

I study so I can use what I learn to help others

129

Learning in school helps me figure out what careers fit my personality

131

I study because it increases my desire to learn more

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wind, S.A., Lugu, B. & Wang, Y. A sequential Moken scaling approach to evaluate response quality in survey research. Behav Res 56, 2273–2291 (2024). https://doi.org/10.3758/s13428-023-02147-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.3758/s13428-023-02147-9

Keywords

Search

Navigation