A Mixture Fluency model using responses and response times with cognitive diagnosis model framework

Abstract

Random guessing behaviors are frequently observed in low-stakes assessments, often attributed to factors such as test-takers lacking motivation or experiencing time constraints and fatigue. Existing research suggests that responses stemming from random guessing behaviors introduce biases into the constructs and relationships of interest. This is particularly problematic when estimating the relationship between speed and ability. This study introduces a Mixture Fluency model designed to account for random guessing behaviors while utilizing valid response accuracy and response time to uncover students' latent attribute profiles. The model directly addresses a limitation present in the Fluency cognitive diagnostic model (Wang & Chen, Psychometrika, 85, 600–629, (2020), which assumes that test-takers consistently employ solution behaviors when answering questions. To investigate the effectiveness of the proposed Mixture Fluency model, we conducted a simulation study encompassing various simulation conditions. Results from this study not only confirm the model's ability to detect potential random guessing behaviors but also demonstrate its capacity to enhance the inference of targeted latent constructs within the assessment. Additionally, we showcase the practical utility of the proposed model through an application to real data.

Appendix

Priors and posteriors of parameters.

Table 16 Priors of parameters

Table 17 Posterior distributions

Gibbs algorithm

The Gibbs sampling method was used to sample model parameters based on the posterior distributions as listed above. The specific update procedures are listed as follows.

Supplement to simulation results when attribute profiles were generated from the discrete uniform distribution and ${\phi }_{i}\sim \left({\mathrm{1,0.6}}^{2}\right){1}_{\left\{{\phi }_{i}>1\right)}$

Table 18 Results of ${\varvec{\tau}}$ when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim \left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$ and $\omega =0.25$

Table 19 Results of ${\varvec{\tau}}$ when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim \left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$ and $\omega =0.5$

Table 20 Results of ${\varvec{\tau}}$ when attribute profiles were generated from the discrete uniform distribution ${\phi }_{i}\sim \left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$ and $\omega =0.75$

Table 21 Results of item parameters when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim \left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$ and $\omega =0.25$

Table 22 Results of item parameters when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim \left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$ and $\omega =0.25$

Table 23 Results of response time parameters when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim \left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$ and $\omega =0.25$

Table 24 Results of item parameters when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim \left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$ and $\omega =0.5$

Table 25 Results of item parameters when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim \left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$ and $\omega =0.5$

Table 26 Results of response time parameters when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim \left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$ and $\omega =0.5$

Supplement to simulation results when attribute profiles were generated from the discrete uniform distribution and ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\left\{{\phi }_{i}>0.4\right)}$

Table 27 Results of assessment accuracy and ${\phi }_{i}$ when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =1$

Table 28 Results of ${\tau }_{i}$ when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =1$

Table 29 Results of attribute and ${\varvec{\phi}}$ when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =0.25$

Table 30 Results of attribute and ${\varvec{\phi}}$ when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =0.5$

Table 31 Results of attribute and ${\varvec{\phi}}$ when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =0.75$

Table 32 Results of ${\varvec{\tau}}$ when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =0.25$

Table 33 Results of ${\varvec{\tau}}$ when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =0.5$

Table 34 Results of ${\varvec{\tau}}$ when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =0.75$

Table 35 Results of item parameters when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =0.25$

Table 36 Results of item parameters when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =0.5$

Table 37 Results of item parameters when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =0.75$

Table 38 Results of response time parameters when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =0.25$

Table 39 Results of response time parameters when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =0.5$

Table 40 Results of response time parameters when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =0.75$

Table 41 Results of item parameters when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =0.75$

Table 42 Results of item parameters when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =0.5$

Table 43 Results of item parameters when attribute profiles were generated from the discrete uniform distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =0.25$

Simulation results when attribute profiles were generated from truncated normal distribution

Table 44 Results of assessment accuracy and ${\phi }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$ and $\omega =1$

Table 45 Results of assessment accuracy and ${\phi }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$ and $\omega =1$

Table 46 Results of ${\tau }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$ and $\omega =1$

Table 47 Results of ${\tau }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$ and $\omega =1$

Table 48 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$ and $\omega =1$

Table 49 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$ and $\omega =1$

Table 50 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =1$

Table 51 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =1$

Table 52 Results of response time parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =1$

Table 53 Results of response time parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$ and $\omega =1$

Table 54 Results of assessment accuracy and ${\phi }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$, $\omega =0.75$

Table 55 Results of assessment accuracy and ${\phi }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$, $\omega =0.75$

Table 56 Results of ${\tau }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$, $\omega =0.75$

Table 57 Results of ${\tau }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$, $\omega =0.75$

Table 58 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$, $\omega =0.75$

Table 59 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$, $\omega =0.75$

Table 60 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$, $\omega =0.75$

Table 61 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$, $\omega =0.75$

Table 62 Results of response time parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$, $\omega =0.75$

Table 63 Results of response time parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$, $\omega =0.75$

Table 64 Results of assessment accuracy and ${\phi }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$, $\omega =0.5$

Table 65 Results of assessment accuracy and ${\phi }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$, $\omega =0.5$

Table 66 Results of ${\tau }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$, $\omega =0.5$

Table 67 Results of ${\tau }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$, $\omega =0.5$

Table 68 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$, $\omega =0.5$

Table 69 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$, $\omega =0.5$

Table 70 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$, $\omega =0.5$

Table 71 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$, $\omega =0.5$

Table 72 Results of response time parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$, $\omega =0.5$

Table 73 Results of response time parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$, $\omega =0.5$

Table 74 Results of assessment accuracy and ${\phi }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$, $\omega =0.25$

Table 75 Results of assessment accuracy and ${\phi }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$, $\omega =0.25$

Table 76 Results of ${\tau }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$, $\omega =0.25$

Table 77 Results of ${\tau }_{i}$ when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$, $\omega =0.25$

Table 78 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$, $\omega =0.25$

Table 79 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$, $\omega =0.25$

Table 80 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$, $\omega =0.25$

Table 81 Results of item parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$, $\omega =0.25$

Table 82 Results of response time parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{1,0.6}}^{2}\right){1}_{\{{\phi }_{i}>1\}}$, $\omega =0.25$

Table 83 Results of response time parameters when attribute profiles were generated from truncated normal distribution, ${\phi }_{i}\sim N\left({\mathrm{0.6,0.3}}^{2}\right){1}_{\{{\phi }_{i}>0.4\}}$, $\omega =0.25$

Table 84 Results of random guessing parameters

Fig. 6

Distributions of log response times