Predicting Holistic Ratings of Written Performance Assessments from Analytic Scoring

Abstract

The use of experts to judge performance assessments is desirable because ratings of performances, carried out by experts in the content domain of the examination, are often considered to be the ``gold standard.'' However, one drawback of using experts to rate performances is the high cost involved. A more economic alternative for scoring performance assessments entails using analytic scoring, which typically involves assigning points to individual traits present in the performance, and summing to arrive at a single score. This strategy is less costly, but may lack the richness of holistic scoring. This study investigates the use of regression-based techniques to predict expert judgments on a written performance task from a combination of analytic scores. Potentially, this will result in scores that approximate the richness of holistic ratings while maintaining the cost-effectiveness of analytic scoring. Results show that a substantial proportion of variance in expert judgments can be explained by the analytic scores, but that decisions based on actual expert judgments and the predicted expert judgments were not sufficiently consistent to warrant the substitution of one score for the other.