Skip to main content
SpringerLink
Log in

Potential sources of criterion bias in supervisor ratings used for test validation

  • Full Articles
  • Published:
Journal of Business and Psychology Aims and scope Submit manuscript

Abstract

Four possible sources ofcriterion contamination were investigated in the supervisory performance ratings used for a predictive criterion-related validation study. Supervisors' liking for subordinates had a very large association with their performance ratings independent of the effects of employee ability. Also as hypothesized, expectations of employee qualifications were correlated significantly with initial (1-mo.) performance ratings but not with ratings made after 5 mos. Ethnicity was not associated with 1-mo. performance ratings, but after five months supervisors gave significantly higher ratings to subordinates of the same ethnic group as themselves. No evidence was found of sex bias in the ratings. Estimates of test validity were reduced substantially when the potential sources of criterion bias were controlled statistically. The data are interpreted in the contexts of construct relevance for ratings criteria, possible spurious inflation of employment test validities, and the developmental processes by which supervisor-subordinate relationships are established in the first few months of employment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Alexander, E.R. & Wilkins, R.D. (1982). Performance rating validity: The relationship of objective and subjective measures of performance.Group and Organizational Studies, 7, 485–496.

    Google Scholar 

  • Austin, J. T. & Villanova, P. (1992). The criterion problem: 1917–1992.Journal of Applied Psychology, 77(6), 836–874.

    Google Scholar 

  • Bazerman, M.H., Beekun, R.I. & Schoorman, F.D. (1982). Performance evaluation in a dynamic context. A laboratory study of the impact of a prior commitment to the ratee.Journal of Applied Psychology, 67, 873–876.

    Google Scholar 

  • Bernardin, H.J. & Villanova, P. (1986). Performance appraisal. In E.A. Locke (Ed.),Generalizing from laboratory to field settings, (pp. 43–62). Lexington, MA: Lexington Books.

    Google Scholar 

  • Blaney, P.H. (1986). Affect and memory: A review.Psychological Bulletin, 99, 229–246.

    PubMed  Google Scholar 

  • Dansereau, F., Graen, G. & Haga, W.J. (1975). A vertical dyad linkage approach to leadership in formal organizations.Organizational Behavior and Human Decision Processes, 13, 46–78.

    Google Scholar 

  • Deaux, K. (1984). From individual differences to social categories: An analysis of a decade's research on gender.American Psychologist, 39, 105–116.

    Google Scholar 

  • Deaux, K. & Major, B. (1987). Putting gender into context: An interactive model of gender-related behavior.Psychological Review, 94, 369–389.

    Google Scholar 

  • Deaux, K. & Wrightsman, L. S. (1988).Social Psychology. Pacific Grove, Ca.: Brooks/Cole Publishing Co.

    Google Scholar 

  • DeNisi, A., Cafferty, T.P., and Meglino, B.M. (1984). A cognitive view of the performance appraisal process: A model and research propositions.Organizational Behavior and Human Performance, 33, 360–396.

    Google Scholar 

  • Dipboye, R.L. (1985). Some neglected variables in research on discrimination in appraisals.Academy of Management Review, 10, 116–127.

    Google Scholar 

  • Dobbins, G.H., Cardy, R.L. and Truxillo, D.M. (1988). The effects of purpose of appraisal and individual differences in stereotypes of women on sex differences in performance ratings: A laboratory and field study.Journal of Applied Psychology, 73, 551–558.

    Google Scholar 

  • Dobbins, G.H. & Russell, J.M. (1986). The biasing effects of subordinate likableness on leaders' responses to poor performers: A laboratory and a field study.Personnel Psychology, 39, 759–777.

    Google Scholar 

  • Feldman, J.M. (1981). Beyond attribution theory: Cognitive processes in performance appraisal.Journal of Applied Psychology, 66, 127–148.

    Google Scholar 

  • Feldman, J.M. (1991). Chair, Symposium: Affect as Cause and Consequence of Behavior in Organizations. Annual Conference of the Society for Industrial and Organizational Psychology, St. Louis, Mo.

  • Gandy, J. A. & Mann, W.G. (1991). Job performance appraisals: Some systematic differences in administrative and research ratings. Paper presented at the annual conference of the Society for Industrial and Organizational Psychology, St. Louis, Mo., April 26.

  • Greenhaus, J.H., Parasuraman, S., & Wormley, W.M. (1990). Effects of race on organizational experiences, job performance evaluations, and career outcomes.Academy of Management Journal, 33, 64–86.

    Google Scholar 

  • Griffin, R.W. and Bedeian, A.G. (1989). Employee performance evaluations: Effects of ratee age, rater age, and ratee gender.Journal of Organizational Behavior, 10, 81–90.

    Google Scholar 

  • Guion, R. (1965).Personnel Testing. New York, NY: McGraw-Hill.

    Google Scholar 

  • Guion, R. (1983). “Comments on Hunter.” In F. Landy, S. Zedeck, & J. Cleveland (Eds.).Performance Measurement and Theory. Hillsdale, N.J.: Lawrence Erlbaum Associates.

    Google Scholar 

  • Gutek, B.A. (1988). Sex segregation and women at work: A selective review.Applied Psychology: An International Review, 37, 103–120.

    Google Scholar 

  • Haberfield, Y. (1992). Employment discrimination: An organizational model.Academy of Management Journal, 35, 161–180.

    Google Scholar 

  • Hanges, P.J., Braverman, E.P., & Rentsch, J.R. (1991). Changes in raters' perceptions of subordinates: A catastrophe model.Journal of Applied Psychology, 76, 878–888.

    Google Scholar 

  • Hanges, P.J., Holke, J.A. & Cox, J. (1992). Effects of busyness and prior information on ratings: A catastrophe model. Paper presented at the Seventh Annual Conference of the Society for Industrial and Organizational Psychology. Montreal, Quebec, May 1–3, 1992.

  • Hartigan, J. A. (1990). Adjustments of minority group test scores used in Employment referrals.Chance: New Directions for Statistics and Computing, 3(3), 38–44.

    Google Scholar 

  • Hartigan, J. A. & Wigdor, A. K. (Eds.) (1989).Fairness in Employment Testing: Validity Generalization, Minority Issues, and the General Aptitude Test Battery. Washington: National Academy Press.

    Google Scholar 

  • Heilman, M.E., and Stopeck, M.N. (1985). Being attractive, advantage or disadvantage: Performance based evaluations and recommended personnel actions as a function of appearance, sex, and job type.Organizational Behavior and Human Decision Processes, 35, 202–215.

    Google Scholar 

  • Hogan, E.H. (1987). Effects of prior expectations on performance ratings: A longitudinal study.Academy of Management Journal, 30, 354–368.

    Google Scholar 

  • Hunter, J.E. (1983). A causal analysis of cognitive ability, job knowledge, job performance, and supervisor ratings. In F. Landy, S. Zedeck, & J. Cleveland. (Eds.)Performance Measurement and Theory. Hillsdale, N.J.: Lawrence Erlbaum Associates.

    Google Scholar 

  • Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance.Psychological Bulletin, 96, 72–98.

    Google Scholar 

  • Ibarra, H. (1993). Personal networks of women and minorities in management: A conceptual framework.Academy of Management Review, 18, 56–87.

    Google Scholar 

  • Ilgen, D.R. & Feldman, J.M. (1983). Performance appraisal: A process focus.Research in Organizational Behavior, 5, 141–197.

    Google Scholar 

  • Isen, A.M. (1987). Positive affect, cognitive processes, and social behavior.Advances in Experimental Social Psychology, 20, 203–253.

    Google Scholar 

  • Isen, A.M., & Baron, R.A. (1991). Positive affect as a factor in organizational behavior.Research in Organizational Behavior, 13, 1–53.

    Google Scholar 

  • Isen, A.M. & Daubman, K.A. (1984). The influence of affect on categorization.Journal of Personality and Social Psychology, 47, 1206–1217.

    Google Scholar 

  • Isen, A.M., Johnson, M.S., Mertz, E. & Robinson, G.F. (1985). The influence of positive affect on the unusualness of word associations.Journal of Personality and Social Psychology, 48, 1413–1426.

    PubMed  Google Scholar 

  • Kingstrom, P.O. and Mainstone, L.E. (1985). An investigation of the rater-ratee acquaintance and rater bias.Academy of Management Journal, 28, 641–653.

    Google Scholar 

  • Kraiger, K., and Ford, J.K. (1985). A meta-analysis of ratee race effects in performance ratings.Journal of Applied Psychology, 70, 56–65.

    Google Scholar 

  • Landy, F.J., and Farr, J.L. (1980). Performance rating.Psychological Bulletin, 87, 72–107.

    Google Scholar 

  • Lefkowitz, J. (1994). Race as a factor in job placement: Serendipitous findings of “Ethnic Drift.”Personnel Psychology, 47, 497–513.

    Google Scholar 

  • Lefkowitz, J. & Battista, M. (1994). The impact of supervisors' liking for subordinates on their performance ratings: A literature review and proposed causal model. Manuscript submitted for publication.

  • Lefkowitz, J. & Fraser, A.W. (1980). Assessment of achievement and power motivation of blacks and whites, using a black and white TAT, with black and white administrators.Journal of Applied Psychology, 65, 685–696.

    Google Scholar 

  • McClelland, L. (1974). Effects of interviewer-respondent interactions on household interview measures of motivation and intelligence.Journal of Personality & Social Psychology, 29, 392–397.

    Google Scholar 

  • Merton, R.K. (1948). The self-fulfilling prophecy.Antioch Review, 8, 193–210.

    Google Scholar 

  • Messick, S. (1980). Test validity and the ethics of assessment.American Psychologist, 35(11), 1012–1027.

    Google Scholar 

  • Murphy, K.R., Balzer, W.K., Lockhart, M.C. & Eisenman, E.J. (1985). Effects of previous performance on evaluations of present performance.Journal of Applied Psychology, 70, 72–84.

    Google Scholar 

  • Northcraft, G.B., Huber, V. and Neale, M.A. (1988). Sex effects in performance related judgments.Human Performance, 1, 161–175.

    Google Scholar 

  • Oppler, S.H., Campbell, J.P., Pulakos, E.D., & Borman, W.C. (1992). Three approaches to the investigation of subgroup bias in performance measurement: Review, results, and conclusions.Journal of Applied Psychology Monograph, 77, 201–217.

    Google Scholar 

  • Pulakos, E.D., and Wexley, K.N. (1983). Actual similarity, sex, and performance ratings in manager-subordinate dyads.Academy of Management Journal, 26, 129–139.

    PubMed  Google Scholar 

  • Pulakos, E.D., White, L.A., Oppler, S.H. & Borman, W. (1989). Examination of race and sex effects on performance ratings.Journal of Applied Psychology, 74, 770–780.

    Google Scholar 

  • Rosenthal, R. (1966).Experimenter Effects in Behavioral Research. New York: Appleton Century Crofts.

    Google Scholar 

  • Sackett, P.R. & DuBois, C.L.Z. (1991). Rater-ratee race effects on performance evaluation: Challenging meta-analytic conclusions. Paper presented at the Sixth Annual Conference of the Society for Industrial and Organizational Psychology, St. Louis, MO. (a)

  • Sackett, P.R. & DuBois, C.L.Z. (1991). Rater-ratee race effects on performance evaluation: Challenging meta-analytic conclusions.Journal of Applied Psychology, 76, 873–877. (b)

    Google Scholar 

  • Sackett, P.R., DuBois, C.L.Z., & Wiggins Noe, A. (1991). Tokenism in performance evaluation: The effects of work group representation on male-female and white-black differences in performance ratings.Journal of Applied Psychology, 76, 263–267.

    Google Scholar 

  • Schmidt, F.L., Hunter, J.E. & Outerbridge, A.N. (1986). Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of job performance.Journal of Applied Psychology, 71, 432–439.

    Google Scholar 

  • Schmitt, N., and Lappin, M. (1980). Race and sex as determinants of the mean and variance of performance ratings.Journal of Applied Psychology, 65, 428–435.

    Google Scholar 

  • Schmitt, N., Gooding, R. Z., Noe, R. A., & Kirsch, M. (1984). Metaanalyses of validity studies published between 1964 and 1982 and the investigation of study characteristics.Journal of Applied Psychology, 37, 407–422.

    Google Scholar 

  • Schoorman, F.D. (1988). Escalation bias in performance appraisals: An unintended consequence of supervisor participation in hiring decisions.Journal of Applied Psychology, 73, 58–62.

    Google Scholar 

  • Shepard, L. A. (1982). “Definitions of bias.” In Berk, R.A. (Ed.)Handbook of Methods for Detecting Test Bias. Baltimore, MD: Johns Hopkins University Press.

    Google Scholar 

  • Shore, L.M. and Thornton, G.C. (1986). Effects of gender on self and supervisory ratings.Academy of Management Journal, 1, 115–129.

    Google Scholar 

  • Smither, J.W., Reilly, R.R. & Buda, R. (1988). Effect of prior performance information on ratings of present performance: Contrast versus assimilation revisited.Journal of Applied Psychology, 73, 487–496.

    Google Scholar 

  • Society for Industrial & Organizational Psychology, Inc. (1987).Principles for the Validation and Use of Personnel Selection Procedures, 3rd Ed. College Park, MD: Author.

    Google Scholar 

  • Steiger, J.H. (1980). Tests for comparing elements of a correlation matrix.Psychological Bulletin, 87(2), 245–251.

    Google Scholar 

  • Turban, D.B., and Jones, A.P. (1988). Supervisor-subordinate similarity: Types, effects, and mechanisms.Journal of Applied Psychology, 73, 228–234.

    PubMed  Google Scholar 

  • Turban, D.B., Jones, A.P., and Rozelle, R.M. (1990). Influences of supervisor affect towards the subordinate on interactions with and evaluations of that subordinate. Paper presented at the Fifth Annual Conference of the Society for Industrial and Organizational Psychology, Miami, FL.

  • Waldman, D.A. & Avolio, B.J. (1991). Race effects in performance evaluations: Controlling for ability, education, and experience.Journal of Applied Psychology, 76, 897–901.

    Google Scholar 

  • Wayne, S.J. & Ferris, G.R. (1990). Influence tactics, affect, and exchange quality in supervisor-subordinate interactions: A laboratory experiment and field study.Journal of Applied Psychology, 75, 487–499.

    Google Scholar 

  • Williams, K.J., DeNisi, A., Blencoe, A.G., and Cafferty, T. (1985). The role of appraisal purpose: Effects of purpose on information acquisition and utilization.Organizational Behavior and Human Decision Processes, 35, 314–339.

    Google Scholar 

  • Williams, R.S. and Walker, J. (1985). Sex differences in performance rating: A research note.Journal of Occupational Psychology, 58, 331–337.

    Google Scholar 

  • Zajonc, R.B. (1980). Feeling and thinking: Preferences need no inferences.American Psychologist, 35, 151–175.

    Google Scholar 

  • Zedeck, S., and Cascio, W. (1982). Performance appraisal decisions as a function of rater training and purpose of the appraisal.Journal of Applied Psychology, 67, 752–758.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Psychology Department, Baruch College, City University of New York, 17 Lexington Avenue, Box 512, 10010, New York City, N.Y.

    Joel Lefkowitz & Mariangela Battista

Authors
  1. Joel Lefkowitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mariangela Battista
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

The authors are greatly indebted to Linda Iorizzo, Jim Benton, and Lorren Oliver for administration of the test battery, and to Ms. Iorizzo also for portions of the data analyses. The manuscript has benefitted greatly from helpful suggestions made by Bob Cardy and Angelo DeNisi.

Earlier versions of this research were presented at The International Academic Symposium on Psychological Measurement, Nanjing Normal University, The People's Republic of China, December 2–5, 1991, and at the Seventh Annual Conference of the Society for Industrial & Organizational Psychology, Montreal, Canada, May 1–3, 1992.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lefkowitz, J., Battista, M. Potential sources of criterion bias in supervisor ratings used for test validation. J Bus Psychol 9, 389–414 (1995). https://doi.org/10.1007/BF02230978

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02230978

Keywords

Search

Navigation