Multi-state model

We consider m manifest variables Y_it, with i = 1,…,m, which are assessed at n different measurement occasions t = 1,…,n. Following LST-R, each manifest variable Y_it is decomposed into its latent state variable τ_it and its measurement error variable ε_it, (1) All variables can take on different values for different persons. The latent state variable τ_it is the conditional expectation of the manifest variable Y_it given U_t, the person variable at time t, and S_t, the situation variable at time t, (2) The measurement error variable ε_it is the difference between the manifest variable Y_it and its latent state variable τ_it, (3) For identifying the scale of a common latent state variable η_t for different manifest variables at time t, we consider a specific reference, for instance, the first manifest variable, such that, (4) Based on the theoretic variables, a multi-state measurement model can be defined. This first requires an equivalence assumption, like the assumption of η_t-congenericity, specifying each item-specific latent state variable τ_it as a linear function of the time point specific common latent state . For assuring an equal measurement model for the common latent state variable at different measurement occasions, measurement invariance for all measurement occasions t is required. For strong measurement invariance, the equivalence assumption for each τ_it is specified with the same intercept and the same factor loading at each measurement occasion, such that the time index t can be omitted from these parameters, (5) Furthermore, the multi-state model requires that all measurement error variables of different items and time points are uncorrelated. Latent state variables and measurement error variables must be uncorrelated at all time points, too. Some of the correlations are zero by definition, but for others model assumptions are required. These are (6) within a time point t all measurement error variables are uncorrelated, and (7) measurement error variables are uncorrelated with future states. All other correlations (i.e., between measurement error variables of different time points, as well as between measurement error variables and states of the same or earlier time points) are zero by definition as derived in the revised LST theory [17].

As implied by the model assumptions, the responses of a person that are represented by the i = 1,…,m manifest variables Y_it, differ only due to random measurement error and item parameters that are constant across all persons. Thus, all systematic variation in the item responses is represented in the common latent states η_t, which describe an attribute of the persons in a specific situation at time t that is measured on an identical scale across all measurement occasions (i.e., on the scale of the reference item).

Extension with item-effect variables

The assumptions of a common state variable for different manifest variables and uncorrelated measurement errors may not hold in all applications, as additional systematic variation in the observed scores can be present due to method effects. Different strategies to account for method effects in longitudinal data have been proposed [see e.g., 4 for an overview]. In line with LST-R theory, method effects can be defined as regression residuals [see e.g., 21] or as latent differences [see e.g., 16]. Both approaches do not alter the meaning of the formerly well-defined state variables, while extending the multi-state model by the inclusion of item effects [see e.g., 4]. Here, in line with Pohl et al. [16], we define method effects as latent difference variables. We consider the most fine-grained level for method effects, which is the level of individual items (respectively individual manifest variables, if these do not represent items but combined scores like item parcels). As such, we build on the work of Thielemann et al. [14], who define latent item-effect variables for dichotomous items in a probit multi-state model. We use a similar definition for latent item-effect variables δ_it but for continuous manifest variables. A latent item-effect variable δ_it for a manifest variable Y_it is the difference between the respective latent state variable τ_it and the common latent state variabl η_t, (8) We defined η_t as the latent state variable of the reference item, thus, the number of item-effect variables is one less than the number of items. With this definition of item-effect variables, the latent state variable τ_it of each item can be decomposed into the common latent state variable η_t and the item-effect variable δ_it of the item i: (9) One item-effect variable is zero for defining a reference item, for instance, of the first item Y_1t. In addition, all factor loadings are set to 1 and all intercepts to 0. As such, item-effect variables describe all systematic differences between the items and represent individual differences. This specification just separates different sources of systematic variation in item-responses without any assumptions, as it can be seen when inserting Eq 8 in Eq 9, .

The identification of item-effect variables δ_it requires at least three repeated measurements of the same manifest variable and the assumption of identical item-effect variables for each manifest variable across the measurement occasions, that is, (10) This assumption of identic item-effect variables across measurement occasions is equivalent to measurement invariance at the individual level. Meaning that the differences between individual (person-specific) item effects are stable over time (δ_i is constant within each person).

In this model, the common latent states η_t represent an attribute of the persons in a specific situation at time t that is measured on an identical scale across all measurement occasions (i.e., on the scale of the reference item); but while accounting for item-specific method effects. Systematic variation in the item responses is represented in the common latent states η_t and the item effects δ_i. Thereby, modeling item effects as stable latent differences allows for estimating the expected values and variances of the item-effect variables across all persons under investigation, as well as all correlations among latent variables.

Previous applications with item-specific method effects

The inclusion of item-specific method effects relaxes the assumptions of the multi-state model and facilitates the identification of latent states while accounting for systematic item-specific variance in the response process. These benefits of modeling item-specific method effects have already been shown for specific constructs (i.e., cognitive components of Alzheimer’s disease, children’s inattention symptoms, subjective happiness, life satisfaction) in the empirical studies of Cogo-Moreira et al. [11], Geiser et al. [12], Holtmann et al. [13], and Thielemann et al. [14]. We provide an overview on the applications in Table A1 in S1 Appendix A in S1 Appendix. All applications showed that item-specific effects were present, but in the study of Geiser et al. [12] the item effects were relatively small, most likely because the indicators were constructed as rather homogenous parcels. While the applications provide valuable insights on the model implementation and the relevance of item-specific method effects in practice, they rarely investigated the item effects themselves. An exception is the analysis by Thielemann et al. [14], who included explanatory variables (i.e., gender, the highest educational degree of a person, and self-awareness) for predicting item-effect variables. However, the predictors could only partly explain variance of some item-effect variables (up to: 9% by gender, 1% by educational degree, 25% by self-awareness). Thus, item-specific method effects can be related to other person characteristics, but further investigations are warranted for their explanation.

Different causes for item-specific method effects

When item-specific method effects are prevalent, this indicates that items are understood or dealt with differently by different persons. In the simplest case, the items differ just by a constant intercept and loading from the reference item (i.e., a congeneric model holds), then the parameters of the item-effect variables can directly be transformed to the parameters of a unidimensional model (see S1 Appendix B in S1 Appendix for the description of the special cases). Yet, when a unidimensional model does not hold, and differences between items vary between persons, then a variety of reasons can be an explanation for the item-effect variables. Items may show semantic multidimensionality, meaning that different items of a scale do not measure one unidimensional construct, but capture different aspects of a construct that are in part distinct. The idea of semantic multidimensionality is closely related to individual difference research focusing on so-called personality nuances [e.g., 22,23], where item effects can be considered secondary traits that are closely related to the focal construct but reflect unique domain content not shared with the other items. Accordingly, Holtmann et al. [13] and Thielemann et al. [14] referred to the item formulations for investigating item-specific method effects and described semantic differences.

Another explanation for item-specific method effects can be different sources of method variance, that is not conceptually related to the focal construct [see e.g., 24 for an overview], meaning that the respondents systematically differ in their use of the response scale (e.g., different response styles) or their understanding of the items (e.g., violation of measurement invariance between respondents). Such effects would be captured as item-effect variables, if they are not constant for all items, but interact with item characteristics or content (e.g., item-length, comprehensibility, as well as positive or negative wording). For instance, Holtmann et al. [13] modeled different rater groups (i.e., self, parent, peer rating), in addition to item-specific method effects, in order to control for this method variance. Their results indicated that rater effects were not present in the application, but item-specific method effects occurred for other reasons.

A more general explanation for item-specific method effects refers to them as “systematic error” [see 25,26]. This summarizes all unique causes for an item that are stable over time, in distinction to shared causes of the items that form the states. With this explanation, different factor loadings, semantic multidimensionality, as well as method variance are included as possible causes of item-specific method effects, but other stable person characteristics may affect the responses to specific items, too. Such more general person characteristics that may interact with the item characteristics can be the familiarity of respondents with the specific item content or their motivation to answer specific items. While the term “systematic error” may lead to an interpretation that item-specific method effects are a form of error that introduces bias in relations among latent constructs [e.g., 24], they can also be referred to as “systematic item-specific variance”. Especially in case of semantic multidimensionality the item-effect variables may be useful not only on psychometric grounds but for substantive analyses [see e.g., 15].

Correlations in multi-construct multi-state models with item effects

In search of explanations for item-specific method effects, one fundamental question is, whether the effects are similar for different items. We investigate this based on the correlation among the item-effect variables and their relation to other constructs. Especially when the proposed multi-state model with item-effect variables is estimated for two (or more) different constructs, the correlations between all latent variables can be evaluated. This allows for investigations on possible explanations for item-effect variables as well as for identifying, whether item-effect variables are construct-specific or more global. Fig 1 shows an exemplary multi-state model for two constructs at three measurement occasions with all latent variables and their correlations. For this model, Table 1 specifies the elements of a correlation pattern.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. A multi-state model with item-effect variables for two constructs and three measurement occasions.

Path diagram for two constructs A and B that are measured with m manifest items and at three measurement occasions t. For each construct, three latent states respectively are modeled, and m−1 item-effect variables respectively for each item i, except for the reference (i.e., here the first item). Correlations between all latent variables within and between the constructs are included.

https://doi.org/10.1371/journal.pone.0288711.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Exemplary correlation-matrix of an exemplary multi-state model with item-effect variables of two constructs A and B.

https://doi.org/10.1371/journal.pone.0288711.t001

In addition, we study the correlations of the item-effect variables. While a latent state is defined with a reference item (i.e., with no item effect), item-effect variables describe the deviation of the latent state when using another item than the reference item. Correlations among item-effect variables indicate “similarities” [14, p. 276] and can be used to identify whether items share systematic effects. When correlations with item-effect variables are zero, then no systematic item-specific variance is present or can be explained by other variables. Perfect correlations (i.e., 1 or -1) occur, when the variables perfectly explain each other. For instance, if an item differs just by a constant loading from the latent state variable, then the latent state variable itself is a perfect explanation for the occurrence of the item effect (i.e., in this case an item-effect variable is not necessary). Also, if specific items differ from the reference item due to the same reason, like only negative formulations of specific items cause differences to a positively formulated reference item, then these item-effect variables can perfectly explain each other (i.e., in this case a common method effect for the negatively formulated items would be sufficient). S1 Appendix B in S1 Appendix provides more details on the scenarios for zero and perfect correlations.

We investigate the correlation of the item-effect variables with regard to their size and systematic structure across different items and constructs. When high correlations of item-effect variables occur only within a construct ^(b), this indicates that item effects are construct-specific. For instance, they may be due to differences in factor loadings, the specific item-content, response styles, or common factors that affect the responses to the specific items. High correlations of item-effect variables between constructs ^(e) indicate that item effects can occur due to more global item and/or person characteristics, for instance, item-length, or common factors that affect the responses across constructs. Correlations of the item-effect variables with the latent states, within ^(c) as well as between ^(f) constructs, can provide first explanations for the occurrence of item effects by this construct. Hereby, high correlations of item-effect variables and latent states within a construct ^(c) indicate that the latent state is relevant for explaining item effects and vice versa, for instance, because the items differ just in their loadings from the states. When high correlations of item-effect variables and latent states occur between constructs ^(f), this indicates that the other construct can partly explain the systematic item-specific variance.

The present application

The described multi-construct multi-state model with item-effect variables is applied to a sample of Dutch respondents providing measures of life satisfaction and positive affect at five measurement occasions. First, we investigate the relevance of item-effect variables for both constructs considering a) model fit and b) the variance of item effects. For this we a) compare the fit of a congeneric model without item-effect variables to a model with item-effect variables and b) investigate the size and significance of the variance of each item-effect variable. Note, non-neglectable variances of the item-effect variables can occur just due to differences in factor loadings. Thus, the model comparison is essential for judging the relevance of item-effect variables. Only if constant factor loadings are not sufficient for modeling differences between the items (i.e., the congeneric model does not fit the data), it is reasonable to use the more complex model with item-effect variables that entangles different sources of systematic differences between items. Then, we use the model with item-effect variables for studying the correlation pattern (i.e., the correlations of item-effect variables with each other and with the latent state variables for life satisfaction and positive affect). This provides information about the systematic structure of item effects within and between the constructs and can indicate possible explanations for item-effect variables.

Sample

We use data of the Dutch Longitudinal Internet Studies for the Social Sciences (LISS) panel administered by CentERdata (Tilburg University, The Netherlands), following a representative sample of the Dutch population since 2008 [27,28]. Initially 8,722 individuals were invited to participate in the panel and 6,808 did so in the first assessment wave. In total 2,596 of the participants took part in the subsequent measurements in 2009, 2011, 2013, and 2014. For our analyses, this subsample of 2,529 respondents (1,315 female) was selected, who completed at least one item on both scales of interest at each measurement occasion. In August 2008 their mean age was 49.91 years (Min = 16, Max = 88, SD = 14.85), and 740 (about 29%) of them had finished higher vocational education (e.g., college or university).

Instruments

Life satisfaction was measured with the Satisfaction with Life Scale [29] including five items with seven-point response scales from “strongly disagree” (1) to “strongly agree” (7). The five SWLS-items were presented in Dutch as items 014–018 in the panel questionnaire (see S1 Appendix C in S1 Appendix for a detailed description of the scale). All items address the focal construct life satisfaction in a global manner. Yet, the wording and reference frame of the items differs. Item 016 assesses life satisfaction most directly and was used as the reference for defining the latent states. As opposed, item 014 and 015 are more specific by asking for ideal and excellent life conditions. This is also the case for item 017 and 018, but these items include a retro perspective component, too. Especially item 018 seems more difficult to agree with, in comparison to the other items, as it states that one would change almost nothing in life.

Positive affect was measured with ten items of the Positive and Negative Affect Schedule [30] asking to what extent a person feels a certain mood in the present moment. The response scales contained seven steps from “not at all” (1) to “extremely” (7). The items were presented in Dutch as items 146–165 in the panel questionnaire (see S1 Appendix C in S1 Appendix for a detailed description of the scale). All items are formulated exactly the same way, except for the specific feeling to be rated. We used item 162 (“attentive”) as the reference item for specifying positive affect in a calm and broad way. While “interested” is similar to this reference, other items are less calm, like “excited”, “enthusiastic”, “alert”, and “active” or are more specific, like “strong”, “proud”, “inspired”, “determined”.

Analysis strategy

First, the life satisfaction and positive affect scales were examined at each measurement occasion to evaluate whether unidimensional measurement models can be applied at each time point. Then, we used all five measurement occasions and estimated (a) a multi-state model (see Fig D.1. in S1 Appendix) and (b) a multi-state model with item-effect variables (see Fig D.2. in S1 Appendix) to investigate the relevance of item-effect variables in this application. Finally, we combined the longitudinal measurement models of the two constructs in (c) a multi-construct multi-state model with item-effect variables (see Fig D.3. in S1 Appendix) to study the correlation pattern. Path diagrams of the multi-state models are available in S1 Appendix D in S1 Appendix. All models were analyzed with the R package lavaan version 0.6–5 [31] using maximum likelihood estimation (R version 3.6.0). The R code for the analyzed models is given in the electronic supplementary materials. The fit of each model was examined in the large sample with the Root Mean Square Error of Approximation (RMSEA), Standardized Root Mean Square Residual (SRMR), and Comparative Fit Index, (CFI). We considered for a good/ acceptable model fit RMSEA≤.05/.08, SRMR≤.05/.10, CFI≥.97/.95 according to Schermelleh-Engel et al. [32]. We first evaluated whether a congeneric model can be used in the data and then we investigated the variance of the item-effect variables. A neglectable variance indicates that an item-effect variable reduces to a constant that is equal across all persons (see S1 Appendix B in S1 Appendix). To identify whether specific item-effect variables are neglectable, we used the test statistic of the estimated variance divided by its standard error for testing the parameter against zero. In addition, we obtained a standardized measure to describe the size of the variance. For this, we estimated the measurement models with indicators that are standardized in their variance (i.e., each indicator was divided by its standard deviation). As such, the standard deviation of each item-effect variable can be described in standard deviation units of the respective indicator. An additive variance decomposition of the latent state and item-specific variance is not possible with this approach, as the variables can correlate.

Single-construct models and relevance of item effects

Unidimensional models for each construct and measurement occasion reached only insufficient fits (see S1 Appendix E in S1 Appendix). Accordingly, also the multi-state models (see Fig D.1. in S1 Appendix) did not adequately describe the data, neither for life satisfaction nor for positive affect (see Table 2). In comparison to a congeneric model with unidimensional state variables, the presented model with correlated item-effect variables (see Fig D.2. in S1 Appendix) substantially improved model fit and resulted in an acceptable fit for both constructs with respect to the model fit indices. Thus, we examined parameter estimates of the multistate model with item-effect variables for each construct. In the following, the numbers in brackets describe the range of the estimated model parameters across measurement occasions t for the state variables (i.e., ) and across items i for the item-effect variables (i.e., or ).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Model fit parameters for the multi-state model with and without item effects for Life Satisfaction and Positive Affect across five measurement occasions.

https://doi.org/10.1371/journal.pone.0288711.t002

Multi-construct model and correlation pattern

The multi-construct model (see Fig D.3. in S1 Appendix) allows for investigating the correlations among all latent variables within and between the two constructs life satisfaction and positive affect. We observed an acceptable fit for the multi-construct model with respect to the fit indices (see Table 2). However, the multi-construct model required a further restriction to avoid non-convergence. That is, the variance and covariances of the item-effect variable , which stands out because of its small variance, were restricted to zero. All estimated means and variances for the other latent variables were nearly identical to the parameter estimates in the previously reported single-construct models, that included no restrictions on the item-effect variable . All parameter estimates are summarized in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Estimated means, standard deviations and correlations of latent variables in the multi-construct multi-state model.

https://doi.org/10.1371/journal.pone.0288711.t003

Empirical insights

In our application using measures of life satisfaction and positive affect across five measurement occasions, we demonstrated the importance of acknowledging item-effect variables. A congeneric model could not describe the item responses, but the inclusion of item-effect variables substantially improved model fit. In addition, all but one of the item-effect variables showed a non-negligible variance. Thus, differences between items were not constant across the respondents, but modeling individual differences could describe the response process.

Our attempt in identifying systematic associations between the item-effect variables within and between constructs was less successful in the specific setting, because the correlation pattern was complex and various explanations for item-effect variables are possible. All correlations between the constructs were at most weak, indicating only slight similarities between the constructs and no common causes for the observed item-effect variables. Within the constructs, the correlation pattern differed. For life satisfaction, few item-effect variables exhibited high correlations among each other, but not with the latent states. Where high correlations occurred, they were plausible in relation to the item content and depict rather distinct aspects from general life satisfaction measured with the states. A possible explanation for this pattern can be that the item-effect variables reflect unique domain content (i.e., semantic multidimensionality), like in individual difference research focusing on so-called personality nuances [e.g., 22,23]. Within positive affect, the latent states were specified as ‘attentive’, this construct partly explained the item-effect variables. The substantial correlation can indicate differences in factor loading between items, but the pattern is not clear and a congeneric model did not hold in the application. Given that a state measure of positive affect was administered, one might also speculate that some item-effect variables are a consequence of individual differences in attention (or lack thereof) when responding to the items. Yet, this association did not generalize to all items of the scale and, importantly, did not transfer to the second construct. Thus, the observed item-effect variables seemed to have different causes and were rather item-specific. This might be different for other instruments with different item content. For instance, when item-effect variables generalize more across constructs, this may be caused by common method variance like a negative wording effect [e.g., 24]. As such, other applications might exhibit a clearer correlation pattern of item-effect variables. Such a pattern could be explained by other explanatory variables, such as psychological constructs (e.g., self-awareness like in the study of Thielemann et al. [14]), but also group membership, test behavior (e.g., response style), or attitudes towards psychological tests.

Strengths and limitations

The presented multi-state multi-construct model with item-effect variables defines all latent variables based on LST theory. Item-effect variables are modelled as stable differences between an item and the reference item, and we only use necessary model assumptions that specify measurement invariance across time points. Thus, all latent correlations can be further investigated. However, in this flexible model, various explanations for item-effect variables are possible. We provide first insights for investigating similarities and differences between item-effect variables within and between constructs based on the correlation pattern. This can help to identify whether common causes for the systematic item-specific variation are plausible. Especially if this is the case for different constructs, the nature of item-effect variables is more global and less related to the measurement of a specific construct. This was not the case in our application, but it illustrates the interesting information provided in the correlation pattern.

Based on this, item-effect variables in different scales can be further investigated in subsequent research. For instance, in psychometric research on scale construction, our model can be used for studying the dimensionality in multi-construct scales and for identifying, whether more strict measurement assumptions are plausible (e.g., for modelling a common method effect for specific items). Also, methodological research can evaluate the occurrence of item-effect variables under different conditions (e.g., in relation to the item format, different response scales or item characteristics) and can investigate whether item effects generalize across different constructs. Especially substantive research (e.g., on the contribution of item-effect variables for predictive validity) would gain from a better understanding on similarities and possible explanations of item-effect variables [e.g., 15]. As this can help for judging whether the item-effect variables represent semantic or method variance in relation to the item content, and respectively may be more or less useful for substantive analysis.

For investigations on the item-effect variables, it is very important to keep in mind that we used a specific modelling approach relying on a reference coding scheme. Thus, the interpretation of all latent variables refers to this specification and can be different when using another identification scheme or modelling approach with different assumptions.

Alternative modeling approaches

For using the reference coding scheme, the selection of the reference item should be plausible for a specific instrument; we relied on substantive theory and considered items that captured the corresponding construct most directly, although this was less clear for the positive affect scale. Using another reference item will not affect model fit, but the interpretation of the latent variables. Similar, an alternative parametrization using an effect coding scheme can be applied for specifying average states across all items [see 14,33]. The item-effect variables will always represent the difference to the latent states, but the interpretation of the item-effect variables depends on the meaning of the latent states. Thus, analyzing the same data with another identification scheme will enable different interpretations in subsequent analyses. Our approach of using a global refence item might support the occurrence of item-effect variables that capture more distinct aspects of other items in the sense of semantic multi-dimensionality. Instead using a more specific reference item may results in more similarities between the item-effect variables (i.e., they share variation from a specific item). The specification of common latent states would entangle content of different items, thus, comparisons between specific items would be limited.

The presented approach poses no restrictions on the correlation structure of item-effect variables and states, but entangles differences in factor loadings with multidimensionality between items. Modeling item effects as residuals typically can include different factor loadings and, thus, can help to disentangle the variance components [see also e.g., 7,12]. However, typically the correlations of item effects and states of the same construct are restricted to zero in these models for additive variance decomposition (i.e., for investigations on the consistency and method-specificity of items). This assumption can be violated in practice. Geiser and Lockhart [4] provide a detailed treatment of similarities and differences between the latent residual and our difference variable approach for defining item-specific method effects. Another modeling approach would be the specification of item-specific traits [e.g., 7–10] that disentangles the situation specificity from the measures. This approach would directly depict a multidimensional construct specification with different items. However, it does not allow for investigations on the item-specific effects itself, because common and item-specific components are entangled. Depending on the causes of item-specific method effects, different modeling approaches can be reasonable, and our approach can help to gain some insights on possible explanations

When choosing a modeling approach, the scale level of the indicators must be considered. In our application, we relied on the assumption that the items can be treated as continuous, that is the rating scales are rated with sufficient gradations [e.g., 34]. For dichotomous data, the same modeling approach in the tradition of item-response theory can be considered [see 14]. Strategies for modeling item-effect variables with ordinal data are currently under investigation and require additional assumptions on the invariance of threshold parameters across measurement occasions. Yet, already Holtmann et al. [13] presented an approach of modeling item effects for ordinal responses as residuals.

Conclusion

The present study presented an approach for modeling item-effect variables in longitudinal data using multi-construct multi-state models. Model fit and the variances of the item-effect variables provide the basis to check whether item-specific interindividual differences are present when measuring a common latent variable with different items. In addition, the correlation pattern within and between constructs can be examined for systematically studying item-effect variables in different scales and settings. For this we provide first empirical insights, but further applications would be beneficial.