Controlling for occasion-specific effects when assessing the test–retest reliability of self-report health questionnaires

Abstract

Objective

This study proposes a method for self-report health questionnaires to adjust test–retest reliability for changes during the test–retest interval based on an external measure, and to distinguish such changes from random response errors.

Methods

In our application, eighty participants completed the Symptoms of Illness Checklist (SIC) on two occasions, two weeks apart, immediately before interviews given on each occasion by one of two physicians in a crossover design. The physician interview scores served as external measures, and structural equation modeling was used to estimate the parameters of a model that corrected for the occasion-specific effect of participants’ responses using information from the interviews.

Results

Correcting for changes in symptoms during the test–retest interval increased SIC test–retest reliability from .744 to .804 and significantly improved model fit (χ² _diff (1) = 30.78, p < .001).

Conclusions

The results suggest methods that can improve the evaluation of self-report health questionnaire test–retest reliability by identifying changes using an external measure, and distinguishing these from random response errors; these increased the estimated SIC test–retest reliability and indicated that the SIC was indeed able to measure changes over the studied time interval. This method can be applied across a broad range of questionnaires.

Appendix: Model specification

Letting bold face characters indicate vectors or matrices, denote ${\mathbf{Y}} = \varvec{\upalpha} + {\mathbf{\Lambda \eta }}$, $Cov(\varvec{\upeta}) = \varvec{\uppsi}$. Then

$$ {\left[ {\begin{array}{*{20}c} {{y^{{(Q)}}_{{j1}} }} \\ {{y^{{(Q)}}_{{j2}} }} \\ {{y^{{(I)}}_{{j1}} }} \\ {{y^{{(I)}}_{{j2}} }} \\ \end{array} } \right]} = {\left[ {\begin{array}{*{20}c} {{a^{{(Q)}}_{1} }} \\ {{a^{{(Q)}}_{2} }} \\ {{a^{{(I)}}_{1} }} \\ {{a^{{(I)}}_{2} }} \\ \end{array} } \right]} + {\left[ {\begin{array}{*{20}c} {1} & {0} & {c} & {0} & {1} & {0} \\ {1} & {0} & {0} & {c} & {0} & {1} \\ {0} & {1} & {1} & {0} & {0} & {0} \\ {0} & {1} & {0} & {1} & {0} & {0} \\ \end{array} } \right]}{\left[ {\begin{array}{*{20}c} {{T^{{(Q)}}_{j} }} \\ {{T^{{(I)}}_{j} }} \\ {{d^{{(I)}}_{{j1}} }} \\ {{d^{{(I)}}_{{j2}} }} \\ {{e^{{(Q)}}_{{j1}} }} \\ {{e^{{(Q)}}_{{j2}} }} \\ \end{array} } \right]},\;\varvec{\uppsi} = {\left[ {\begin{array}{*{20}c} {{\sigma ^{2}_{Q} }} & {{\sigma _{{QI}} }} & {0} & {0} & {0} & {0} \\ {{\sigma _{{QI}} }} & {{\sigma ^{2}_{I} }} & {0} & {0} & {0} & {0} \\ {0} & {0} & {{\sigma ^{2}_{d} }} & {0} & {0} & {0} \\ {0} & {0} & {0} & {{\sigma ^{2}_{d} }} & {0} & {0} \\ {0} & {0} & {0} & {0} & {{\sigma ^{2}_{e} }} & {0} \\ {0} & {0} & {0} & {0} & {0} & {{\sigma ^{2}_{e} }} \\ \end{array} } \right]} $$

Here $\sigma ^{2}_{Q}$, $\sigma ^{2}_{I}$, $\sigma ^{2}_{d}$ and $\sigma ^{2}_{e}$ are the variances of the factors T ^(Q), T ^(I), d ^(I), and e ^(Q), $\sigma _{{QI}}$ is the covariance between T ^(Q) and T ^(I), and c is the correlation coefficient obtained from regressing $d^{{(Q)}}_{{jk}}$ on $d^{{(I)}}_{{jk}}$. Because the mean structure of the model is saturated due to the estimation of separate intercepts for each of the observed variables, it is possible to give a simplified expression for the model-implied covariance matrix as ${\mathbf{\hat{\Sigma }}} = {\mathbf{\Lambda \Psi {\Lambda }\ifmmode{'}\else$'$\fi }}$. The maximum likelihood fitting function which is used to index the discrepancy between this model-implied matrix and the original sample covariance matrix (S) can be given as: $F_{{ML}} = \log {\left| {\hat{\Sigma }} \right|} + trace[{\mathbf{S\hat{\Sigma }}}^{{ - 1}} ] - \log {\left| {\mathbf{S}} \right|} - t,$ where t is the total number of study variables. In the present study, S is the covariance matrix for the four-variate multinormal distribution of the study variables $y^{{(I)}}_{{j1}}$, $y^{{(I)}}_{{j2}}$, $y^{{(Q)}}_{{j1}}$, and $y^{{(Q)}}_{{j2}}$.

With sample size N, F _ML can be rescaled to approximate a chi-square variate for purposes of model goodness-of-fit testing: (N–1)F _ML ∼ χ². Under standard conditions, this provides a chi-square test of model fit with degrees of freedom equal to the difference between the number of estimated model parameters and the total number of means, variances, and covariances.