Does the burden of the experience sampling method undermine data quality in state body image research?

Highlights

• Increasingly, the experience sampling methodology (ESM) is being used to evaluate state body image.

• There is concern that the burden of ESM may undermine data quality.

• Our findings suggest that compliance and score variability declines over testing period.

• Recommendations are made for screening and correcting reactivity in ESM designs.

Abstract

Despite growing popularity of experience sampling methodology (ESM) for evaluations of state-based components of body image, there have been concerns that the frequent repeated measurement might encourage problematic responding resulting in low data quantity and/or quality. Using a sample of 105 women (mean age = 24.84), this study used multilevel modelling to investigate whether (a) there were changes in compliance or response variability across a 7-day period, and (b) whether such changes are explained by participant characteristics. Present findings suggest that demands of ESM protocol undermine quantity more so than quality of obtained data. Decline in procedural compliance across the testing period correlated with BMI and body shame, whereas reduced variability in state-based assessments did not adversely impact the strength of association between state body satisfaction ratings and other variables in the dataset. The authors make several recommendations for ensuring the quality of ESM-based data in future studies.

Introduction

Body image refers to an individual's perception of, and attitudes towards, his or her body and appearance (Cash, Fleming, Alindogan, Steadman, & Whitehead, 2002). Body dissatisfaction (hereafter BD) is one aspect of body image, relating to an individual's degree of dissatisfaction with particular parts of the body (Cook-Cottone & Phelps, 2003). The importance of investigating BD is indicated by its high prevalence (especially among women – Franko and George, 2009, McCabe et al., 2011) and its correlation with adverse health consequences such as depression, eating disorders, obesity, and reduced psychological wellbeing (Darby, Hay, Mond, Rodgers, & Owen, 2007).

BD can be further broken down into trait and state components (Cash et al., 2002, Thompson, 2004, Vocks et al., 2009). Trait BD is characterized by its stability over time and across a range of contexts, and is usually measured in terms of (dis)satisfaction with either specific body parts (e.g., hips, arms, legs) or global aspects of appearance (weight, shape, and size). In contrast, state BD is characterized by fluctuations over time. Importantly, these fluctuations have been associated with contextual factors (such as present company, mood, exercise, and recent consumption of food) and individual difference factors (e.g., trait body dissatisfaction, trait affect, and appearance-comparison tendencies; Colautti et al., 2011, Fuller-Tyszkiewicz et al., 2013, Lattimore and Hutchinson, 2010, LePage and Crowther, 2010). Furthermore, state BD has been shown to associate with key body image outcomes (such as eating pathology) over and above trait BD, suggesting that both constructs play an important role in mental health (Melnyk, Cash, & Janda, 2004).

State BD is typically measured using measures such as the Body Image States Scale (BISS; e.g., Colautti et al., 2011, Fuller-Tyszkiewicz et al., 2013, Melnyk et al., 2004, Rudiger et al., 2007), the appearance subscale of the State Self-Esteem Scale (Leahey et al., 2011, LePage and Crowther, 2010), or the modified Body-Image Ideals Questionnaire (Heron & Smyth, 2013) that are sufficiently brief to be suitable to accommodate experience sampling methodology (ESM; Stone & Shiffman, 1994). ESM, also known as Ecological Momentary Assessment, involves the repeated measurement of a variable within the context of an individual's daily life. In a typical ESM study, an individual carries a device that sounds an alarm at random and/or fixed intervals throughout the day. When the alarm sounds, the individual fills out questionnaires that relate to her/his momentary experiences. Since the time lag between the signal and the response is very short, data are not contaminated by memory distortions or judgemental biases (see Robinson & Clore, 2002 for a longer discussion). ESM also has the added benefit of increased ecological validity, as it is able to capture the representation of experience (such as mood state or body satisfaction) at, or close to, its occurrence and within the context of a person's everyday life (Connor Christensen, Feldman Barrett, Bliss-Morea, Lebo, & Kaschub, 2003). In addition, using ESM allows the variable of interest to be studied across a variety of environmental, social, and psychological conditions, as well as allowing an examination of the fluctuations in that variable across conditions and time (Stone, Shiffman, & DeVries, 1999).

Despite the potential benefits of ESM, some researchers have raised concern that the added burden of repeated assessment may undermine the quality of ESM data (Hufford, 2007, Shiffman et al., 2008). As, on average, most ESM studies involve between 56 and 168 momentary assessments for studies running 1–2 weeks (Connor Christensen et al., 2003), ESM could be particularly sensitive to one aspect of overall data quality, measurement reactivity, which refers to instances where the act of measurement undermines the accuracy of data obtained (French & Sutton, 2010).

Previous research investigating measurement reactivity in ESM has found little evidence of systematic changes in the variables measured using ESM versus those same variables measured using different methodologies (such as retrospective reports; see Hufford et al., 2002, Shiffman et al., 2008, Stone et al., 2003). In the body image literature, studies that enable assessments of measurement reactivity typically show no systematic changes in key variables when ESM is used (Heron and Smyth, 2013, Leahey et al., 2007, LeGrange et al., 2002, Munsch et al., 2009, Stein and Corte, 2003). For example, Leahey et al. (2007) used a control group to determine reactivity associated with daily recordings of appearance-related social comparisons. The authors found no significant differences in affect, body satisfaction, self-esteem, depressive symptomatology, and weight-related cognitions for those in the ESM group versus those in a control group who did not make ESM assessments. Notably, the indicators of reactivity in this study were changes in predominantly trait measures from pre-sampling to post-sampling (e.g., trait affect, body satisfaction, and self-esteem). Leahey et al. (2011) obtained a similar pattern of findings for state-based measures; while the frequency of appearance-related comparisons increased over the testing period, non-significant differences were found for all other state-based measures (state mood, body dissatisfaction, dieting and exercise behaviours and cognitions).

In a study to assess the utility of ESM for self-monitoring of eating behaviour in comparison to standard monitoring using food diaries, LeGrange et al. (2002) compared two groups: one receiving CBT for binge eating disorder (BED), and another receiving cognitive-behavioural therapy (CBT) with an ESM component. Again, there were no significant differences between groups on binge eating episodes, eating pathology, self-esteem, or depression. Munsch et al. (2009) also investigated measurement reactivity in individuals with BED participating in short-term CBT. The authors found no significant differences between binge eating using ESM and binge eating using self-report methods. Lastly, Stein and Corte (2003) found no differences in the frequency distributions of five eating disordered behaviours over a 1-month period using ESM. These studies, taken together, suggest that measurement reactivity is minimal in body image/ED research using ESM.

However, measurement reactivity is just one aspect to consider when evaluating overall data quality. There is a wide range of ways that data can be corrupted. For example, as signalling in an ESM study necessarily disrupts whatever activity the individual is engaged in, it is possible that individuals may find participation in an ESM study irritating (Shiffman et al., 2008). This potential irritation could contaminate the collection of mood-based data (for example, body satisfaction measures). In addition, the task of filling out up to 12 assessments per day for 1–2 weeks could be perceived as onerous (Napa Scollon, Kim-Prieto, & Diener, 2003). Perceptions of burden or irritation thus could lead to reduced quality and quantity of ESM data through reduced compliance (reduced sampling density) or the use of answering heuristics to expedite response times, such as reliance upon the same response or random responding at each time point. For the former strategy (same response across time points), one may anticipate reductions in variance estimates across the testing period, although mean scores may remain relatively stable. In contrast, random responding may result in increased variance across the testing period and potentially more variable mean scores.

Whilst previous research in the body image literature suggests that ESM is not sensitive to measurement reactivity, these studies have focused primarily on trait body image and other trait variables (self-esteem, trait affect, etc.; Leahey et al., 2007, LeGrange et al., 2002). It is likely that should reactivity occur, it will be manifested in differences on state variables, rather than on trait variables (by definition, considered stable across time). Moreover, it is not enough to examine changes in mean state scores; bias/reactivity may also be expressed in other ways, such as reduced variance due to the use of answering heuristics and/or reduced compliance with study protocols throughout the testing period. In addition, it is not known whether: (1) scores on state measures (such as ED symptoms) change systematically as a function of time, (2) non-compliance changes over the course of an ESM sampling protocol, or (3) individual difference factors can predict such changes in compliance and variability in momentary assessment scores. A more comprehensive examination of overall data quantity and quality in an ESM study of body (dis)satisfaction is required to address the gaps in existing literature.

The overall aim of the current study was to address these gaps by investigating the effects of ESM on overall data quantity and quality in a sample of young adult women. Specifically, the following questions were addressed:

• Do procedural non-compliance and/or variance in state BD scores vary across the testing period? If so, do these changes associate with the predictive utility of state BD?

• Does the impact of time on indices of data quantity and quality (compliance and variability, respectively) differ across individuals?

• And, if so, can trait-level body image variables (BMI, body shame, appearance comparison tendencies, body self-monitoring, and internalization of appearance standards) predict this variation?

Under the assumption of burden due to ESM design, we would anticipate increased non-compliance over the course of the week. Furthermore, if individuals were using heuristics to more quickly complete the momentary assessments, we would expect to see reduced variability in state body satisfaction scores over the course of the testing period. If this increased reliance on heuristics undermines validity of the measurement, it should associate with a decline in the predictive utility of state BD. Finally, it has been shown that incomplete (missing) data in body image surveys are related to BMI and key body image variables (such as trait body dissatisfaction and internalization of appearance standards) (Fonseca et al., 2008, Tiggemann, 2006). Given this, it is important to examine whether these trait variables influence the measures of response bias used in the present study.