A Hierarchical Model for Accuracy and Choice on Standardized Tests

“A people however, who are possessed of the spirit of commerce, who see, and who will pursue their advantages, may achieve almost anything.”

-George Washington, 1784, Letter to Benjamin Harrison.

Abstract

This paper assesses the psychometric value of allowing test-takers choice in standardized testing. New theoretical results examine the conditions where allowing choice improves score precision. A hierarchical framework is presented for jointly modeling the accuracy of cognitive responses and item choices. The statistical methodology is disseminated in the ‘cIRT’ R package. An ‘answer two, choose one’ (A2C1) test administration design is introduced to avoid challenges associated with nonignorable missing data. Experimental results suggest that the A2C1 design and payout structure encouraged subjects to choose items consistent with their cognitive trait levels. Substantively, the experimental data suggest that item choices yielded comparable information and discrimination ability as cognitive items. Given there are no clear guidelines for writing more or less discriminating items, one practical implication is that choice can serve as a mechanism to improve score precision.

Appendix

1.1 Parameter Recovery Monte Carlo Simulation

This section reports results of a Monte Carlo simulation designed to assess the ability of the proposed algorithm to recover item parameters. Specifically, the estimated model parameters reported in Table 3 were used as population values and data for 252 subjects were simulated to assess bias and root mean squared error (RMSE). Furthermore, the Monte Carlo simulation employed the experimental fixed-effects and random-effects design matrices $\mathbf {X}$ and $\mathbf {W}$ to generate data from the model.

Figures 9 and 10 report parameter bias and RMSE based upon 1000 replications. Figure 9 provides evidence of minimal bias for a small sample size of 252 participants. Figure 10 plots RMSE for the IRT, Thurstone, and hierarchical model parameters. In particular, RMSE for the structural coefficients (i.e., $\varvec{\beta }$) and random-effect variances (i.e., $\text {diag}\left( \varvec{\Sigma }_{\varvec{\zeta }}\right)$ was generally smaller than the RMSE for the item slopes and thresholds. Furthermore, the RMSE for the payout condition fixed-effects (i.e., the first six fixed-effects for $\varvec{\gamma }$) was smaller than for the remaining 27 item evaluations. The difference in RMSE for the payout condition main-effects and item evaluations is expected given that a subset of the paired comparisons were collected and items received fewer evaluations.