Beyond alpha and omega: The accuracy of single-test reliability estimators in unidimensional continuous data

Abstract

Coefficient alpha is commonly used as a reliability estimator. However, several estimators are believed to be more accurate than alpha, with factor analysis (FA) estimators being the most commonly recommended. Furthermore, unstandardized estimators are considered more accurate than standardized estimators. In other words, the existing literature suggests that unstandardized FA estimators are the most accurate regardless of data characteristics. To test whether this conventional knowledge is appropriate, this study examines the accuracy of 12 estimators using a Monte Carlo simulation. The results show that several estimators are more accurate than alpha, including both FA and non-FA estimators. The most accurate on average is a standardized FA estimator. Unstandardized estimators (e.g., alpha) are less accurate on average than the corresponding standardized estimators (e.g., standardized alpha). However, the accuracy of estimators is affected to varying degrees by data characteristics (e.g., sample size, number of items, outliers). For example, standardized estimators are more accurate than unstandardized estimators with a small sample size and many outliers, and vice versa. The greatest lower bound is the most accurate when the number of items is 3 but severely overestimates reliability when the number of items is more than 3. In conclusion, estimators have their advantageous data characteristics, and no estimator is the most accurate for all data characteristics.

Appendix: The condition for \({{{\lambda}}}_{4(\mathrm{m}\mathrm{a}\mathrm{x})}\) to equal reliability at the population level when \({{k}}\) = 3

Let us denote the three items as ${X}_{1}$, ${X}_{2}$, and ${X}_{3}$, where ${X}_{i}={a}_{i}+{b}_{i}F+{e}_{i}$, and denote the two split-halves as $A$ and $B$, where $A$ has one item and $B$ has two items. For simplicity, we assume that ${X}_{1}$ and ${X}_{2}$ have the smallest covariance, and ${X}_{2}$ and ${X}_{3}$ have the greatest covariance (i.e., $Cov({X}_{1},{X}_{2})\le Cov({X}_{1},{X}_{3})\le Cov({X}_{2},{X}_{3})$). In this case, ${\lambda }_{4({\text{max}})}$ = 4 ${\sigma }_{AB}/{\sigma }_{X}^{2}$ = 4Cov $({X}_{3}, {X}_{1}+{X}_{2})/{\sigma }_{X}^{2}$ = $4({b}_{1}{b}_{3}+{b}_{2}{b}_{3})/{\sigma }_{X}^{2}$. Reliability (${\rho }_{X{X}{\prime}}$) is = ${({b}_{1}+{b}_{2}+{b}_{3})}^{2}/{\sigma }_{X}^{2}$. Therefore, ${\lambda }_{4({\text{max}})}$ is less than ${\rho }_{X{X}{\prime}}$ if the following formula is negative: $4\left({b}_{1}{b}_{3}+{b}_{2}{b}_{3}\right)-{({b}_{1}+{b}_{2}+{b}_{3})}^{2}$. Rearranging this formula leads to the following formula: ${-({b}_{1}+{b}_{2}-{b}_{3})}^{2}$. This formula is zero if ${b}_{3}= {b}_{1}+{b}_{2}$, and negative otherwise.