- Published:
On the measurement of competency
Empirical Research in Vocational Education and Training volume 2, pages 41–63 (2010)
Abstract
Across multiple societal sectors, demand is growing to measure individual and group competencies. This paper unpacks Hartig et al.’s (2008) competency definition as a complex ability construct closely related to real-life-situation performance to make it amenable to measurement. Unpacked following the assessment triangle (construct, observation, inference), competency measurement is exemplified by research from business, military and education sectors. Generalizability theory, a statistical theory for modeling and evaluating the dependability of competency scores, is applied to several of these examples. The paper then pulls together the threads into a general competency measurement model.
References
Baxter, G.P. & Shavelson, R.J. (1994.) Science performance assessments: Benchmarks and surrogates. International Journal of Educational Research, 21, 279–298.
Brennan, R. L. (2001). Generalizability theory. New York: Springer-Verlag.
Cardinet, J., Johnson, S. & Pini, G. (2009). Applying generalizability theory using EduG. New York: Routledge/Psychology Press.
Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. Cambridge, MA: Cambridge University Press.
Cronbach, L.J., Gleser, G.C., Nanda, H. & Rajaratnam, N. (1972). The dependability of behavioral measurements. New York: Wiley.
Fitzpatrick, R. & Morrison, E. J. (1971). Performance and product evaluation. In R. L. Thorndike (Ed.) Educational measurement. Washington, DC: American Council on Education.
Green, B. & Shavelson, R.J. (1987). Distinguished panel discussion on issues in the joint-service JPM program. In H.G. Baker and G.J. Laabs (Eds.) Proceedings of the Department of Defense/Educational Testing Service Conference on Job Performance Measurement Technologies., Washington, D.C.: Office of the Assistant Secretary of Defense (Force Management and Personnel).
Hartig, J., Klieme, E. & Leutner, D. (Eds.) (2008). Assessment of Competencies in Educational Contexts: State of the Art and Future Prospects. Göttingen: Hogrefe & Huber.
Kane, M. T. (1982). A sampling model for validity. Applied Psychological Measurement, 6, 125–160.
Klein, S. (2007). Characteristics of hand and machine-assigned scores to college students’ answers to open-ended tasks. In D. Nolan & T. Speed (Eds) Probability and Statistics: Essays in Honor of David A. Freedman. IMS Collections, Volume 2. Beachwood, OH: Institute for Mathematical Statistics.
McClelland, D.C. (1973). Testing for competence rather than testing for “intelligence”. American Psychologist, 28(1), 1–14.
National Research Council (2001). Knowing what students know: The science and design of Educational Assessment. Washington, DC: National Academy Press.
Ruiz-Primo, M.A. & Shavelson, R.J. (1996). Rhetoric and reality in science performance assessments: An update. Journal of Research in Science Teaching, 33(10), 1045–1063.
Shavelson, R.J. (1967). Man’s performance of basic maintenance tasks in lunar gravity. Unpublished master’s thesis, San Jose State College.
Shavelson, R.J. (1991). Generalizability of military performance measurements: I. Individual performance. In A.K. Wigdor & B.F. Green, Jr., (Eds.) Performance assessment for the workplace (Vol. II): Technical issues. Washington, D.C.: National Academy Press.
Shavelson, R.J. (2010). Measuring college learning responsibly: Accountability in a new era. Stanford, CA: Stanford University Press.
Shavelson, R.J., Baxter, G.P. & Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30(3), 215–232.
Shavelson, R.J., Klein, S. & Benjamin, R. (2009). The limitations of portfolios. Inside Higher Education. Retrieved August 10, 2009 from on http://www.insidehighered.com/views/2009/10/16/shavelson.
Shavelson, R.J., Roeser, R.W., Kupermintz, H., Lau, S., Ayala, C., Haydel, A., Schultz, S., Quihuis, G. & Gallagher, L. (2002). Richard E. Snow’s remaking of the concept of aptitude and multidimensional test validity: Introduction to the special issue. Educational Assessment, 8(2), 77–100.
Shavelson, R.J., Ruiz-Primo, M.A. & Wiley, E.W. (1999). Note on sources of sampling variability in science performance assessments. Journal of Educational Measurement, 36 (1), 61–71.
Shavelson, R.J. & Seminara, J. (1968). Effect of lunar gravity on man’s performance of basic maintenance tasks. Journal of Applied Psychology, 52(3), 177–183.
Shavelson, R.J. & Webb, N.M. (1981). Generalizability theory: 1973–1980. British Journal of Mathematical and Statistical Psychology, 34, 133–166.
Webb, N.M., Shavelson, R.J., Kim, K-S & Chen, Z. (1989). Reliability (generalizability) of job performance measurements: Navy machinists mates. Military Psychology, 1(2), 91–110.
Wigdor, A.K. & Green, B.F. Jr., (Eds.) (1991). Performance assessment for the workplace (Vol. I). Washington, D.C.: National Academy Press.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Shavelson, R.J. On the measurement of competency. Empirical Res Voc Ed Train 2, 41–63 (2010). https://doi.org/10.1007/BF03546488
Published:
Issue Date:
DOI: https://doi.org/10.1007/BF03546488