Assessing agreement with intraclass correlation coefficient and concordance correlation coefficient for data with repeated measures

Abstract

The intraclass correlation coefficient and the concordance correlation coefficient are two popular scaled indices for assessing the closeness between observers who make measurements for quantitative responses. These two indices are usually based on subject and observer effects only, and therefore we cannot use these indices if the observer produces repeated measurements rather than replicated readings. In this paper, we consider not only subject and observer effects, but also time effects for data with repeated measurements since it is difficult to obtain the true replications in practice. We compare these two agreement indices for different combinations of random or fixed effects of observer and time. Finally, we use image data of 2D-echocardiograms to illustrate the proposed methodology and the comparison of these two indices. If there is a need to choose between these two indices for repeated measurements, we recommend to use the new concordance correlation coefficient since it does not need ANOVA assumptions.

Introduction

The intraclass correlation coefficient ($\mathrm{ICC}$) and the concordance correlation coefficient ($\mathrm{CCC}$) are two popular indices for assessing agreement between quantitative measurements taken from different observers. $\mathrm{ICC}$ and $\mathrm{CCC}$ are usually used for data without and with replications and the comparison between these two indices based on this data structure under a general model is reported by Chen and Barnhart (2008). However, we cannot use this methodology if repeated measurements rather than replications are collected. Several authors have studied the agreement indices of $\mathrm{ICC}$ and $\mathrm{CCC}$ for data with repeated measurements. Vangeneugden et al. (2005) investigated the $\mathrm{ICC}$ by linear mixed models with serial correlation for inter-observer, intra-observer, and absolute agreement, where both observer and time are treated as random effects. King et al. (2007) proposed a $\mathrm{CCC}$ for assessing inter-observer agreement for a response with repeated measurement, where both observer and time are treated as fixed effects. Chen and Barnhart (2011) also proposed a new $\mathrm{CCC}$ for assessing inter-observer, intra-observer, and absolute agreement for data with repeated measurements where observers and times are treated as random since researchers may assess agreement among many observers who take measurements at different time points.

In addition to treating both observer and time as random effects in defining $\mathrm{CCC}$, there are situations where researchers may be interested in the case of random observer and fixed time, or fixed observer and random time. For example, a study is designed to assess the agreement among subjects’ measurements taken by observers (e.g. nurse) for two shifts. Researchers can conduct and assess an agreement study by randomly selecting several nurses from the nurse population and obtaining measurements from the chosen nurses at two time points. In this study, nurse is random and time is fixed because different nurses may produce different measurements at the specific time. Another example is to assess agreement among a fixed number of observers for image study. Patients may have no scheduled visits to take measurements by these observers. In this example, observer is fixed and time is random because the same observers may produce different measurements at different times. Therefore, any combinations of fixed or random effects for observer and time for an agreement study of $\mathrm{CCC}$ or $\mathrm{ICC}$ can happen depending on the goals of the researchers.

In this paper, we propose new $\mathrm{CCC}$s and $\mathrm{ICC}$s for the remaining combinations of random or fixed effects for the observer and time. We summarize and compare $\mathrm{CCC}$s and $\mathrm{ICC}$s between combinations of random or fixed effects for data with repeated measurements and illustrate the methodology with an example from image study. Section 2 studies four combinations for random or fixed effects of observer and time for the two indices, and introduces the new $\mathrm{CCC}$s and $\mathrm{ICC}$s for the remaining combinations for different agreement assessments. Section 3 presents the estimation and inference for the methods introduced in Section 2. Section 4 demonstrates the performance of the new $\mathrm{CCC}$ for the case of random observer and fixed time by a simulation study. Section 5 illustrates four combinations for $\mathrm{CCC}$ and $\mathrm{ICC}$ by image data. Finally, Section 6 discusses the comparison between these two indices.