The assessment of school engagement: Examining dimensionality and measurement invariance by gender and race/ethnicity

https://doi.org/10.1016/j.jsp.2011.04.001Get rights and content

Abstract

The construct of school engagement has attracted growing interest as a way to ameliorate the decline in academic achievement and increase in dropout rates. The current study tested the fit of a second-order multidimensional factor model of school engagement, using large-scale representative data on 1103 students in middle school. In order to make valid model comparisons by group, we evaluated the extent to which the measurement structure of this model was invariant by gender and by race/ethnicity (European-American vs. African-American students). Finally, we examined differences in latent factor means by these same groups. From our confirmatory factor analyses, we concluded that school engagement was a multidimensional construct, with evidence to support the hypothesized second-order engagement factor structure with behavioral, emotional, and cognitive dimensions. In this sample, boys and girls did not substantially differ, nor did European-American and African-American students, in terms of the underlying constructs of engagement and the composition of these constructs. Finally, there were substantial differences in behavioral and emotional engagement by gender and by racial/ethnic groups in terms of second-order factor mean differences.

Introduction

Active engagement in school is important for student learning and academic success. Previous studies have found that students who are more engaged in school have better academic performance (Csikszentmihalyi and Schneider, 2000, Newmann et al., 1992). Students who attend school regularly, concentrate on learning, and adhere to the rules of the school, generally receive higher grades and perform better on standardized tests (Bandura et al., 1996, Caraway et al., 2003, Wang and Holcombe, 2010). In contrast, lack of school engagement can have serious consequences for students, including underachievement, engagement in deviant behaviors, and increased risk of dropping out of school (Finn & Rock, 1997). Interest in school engagement has intensified recently due to the increasing proportion of adolescents who report feeling alienated, particularly as they progress from elementary to middle and then from middle to high school (e.g., Finn, 1989, Finn and Voelkl, 1993, Marks, 2000, McDermott et al., 2001). The National Research Council and Institute of Medicine (2004) reported that 30% to 50% of middle school students are disengaged from school. These findings lead researchers and educators to underscore the need to enhance students' school engagement.

According to Fredricks, Blumenfeld, and Paris (2004), school engagement is a multidimensional construct that is made up of three components: behavior, emotion, and cognition. Behavioral engagement refers to the actions and practices that students direct toward school and learning; it includes positive conduct (e.g., attending class and completing schoolwork), involvement in learning and academic tasks (e.g., effort and concentration), and participation in extracurricular activities (e.g., athletics). Emotional engagement represents a student's affective reactions and sense of connectedness to school (Finn, 1989, Skinner and Belmont, 1993). Finn (1989) conceptualizes it as students' sense of belonging to and valuing of their school. Cognitive engagement refers to a student's self-regulated and strategic approach to learning in which students use metacognitive strategies to plan, monitor, and evaluate their cognition (Connell and Wellborn, 1991, Zimmerman, 1989).

These engagement components do not operate in isolation, but rather, they are understood as interwoven and dynamic. Fredricks et al. (2004) proposed that patterns of engagement across these three dimensions have long-term effects on students' academic achievement. Emotional identification with school interacts with behavioral engagement and cognitive involvement in school learning. This interaction is of concern as low levels of each dimension have been shown to lead to unsuccessful school outcomes. In addition, a lack of behavioral participation and cognitive engagement is also related to emotional withdrawal from school-related activities, resulting in even less academic success. Over time, behavioral participation, emotional identification, and cognitive engagement exert reciprocal influence. Ultimately, the degree to which students engage in school behaviorally, emotionally, and cognitively influences their academic success, which in turn, may influence changes in all three aspects of school engagement.

Although empirical researchers have developed scales for measuring engagement, until quite recently many have measured school engagement as either a uni-dimensional combination of these various indicators (e.g., Daly et al., 2009, Perry et al., 2010, Simons-Morton and Chen, 2009, You and Sharkey, 2009) or, at most, as two of the three types (e.g., Appleton et al., 2006, Connell and Wellborn, 1991, Skinner et al., 2008). The various measurement models for school engagement are reflective of differences that exist in how researchers define the construct. For instance, some researchers have developed scales for measuring engagement as a global construct, such as the Research Assessment Package for Schools (Institute for Research and Reform in Education, 1998), the High School Survey of Student Engagement (Yazzie-Mintz, 2007), the Rochester Assessment Package for Schools (Wellborn & Connell, 1987), and the engagement scale of the School Success Profile (Bowen & Richman, 1995). While these global instruments of engagement allow the researcher to reflect a holistic picture of engagement, a major weakness of this type of measurement model is that it may not provide sufficient numbers of items to understand individual scales or new understandings about the multidimensionality of the construct. If school engagement is indeed a multidimensional construct, this practice of measuring it as a single construct limits examining differences among the various types of engagement and understanding their possible antecedents and consequences (Fredricks et al., 2004, Guthrie and Wigfield, 2000, Wang and Holcombe, 2010).

Some researchers adopting a two-component model frequently include a behavioral and an emotional or affective component in their engagement instruments (Finn, 1989, Marks, 2000, Skinner et al., 2008). For instance, Marks (2000) measured engagement with a student-report survey designed to assess students' effort, attentiveness, lack of boredom, and the extent to which they completed assignments. Although there is evidence to suggest the importance of cognitive engagement to school performance, few instruments include separate cognitive components (see Appleton et al., 2006, Jimerson et al., 2003 for a review). To address this gap, Appleton et al. (2006) designed the Student Engagement Instrument to measure two subtypes of student engagement with school: psychological and cognitive engagement. Martin (2007) developed the Motivation and Engagement Scale to assess behavioral and cognitive dimensions of school engagement. He proposed a 4 second-order factor model, including adaptive and maladaptive behavioral dimensions and adaptive and maladaptive cognitive dimensions. Most recent studies of engagement have proposed a tripartite conceptualization that includes behavioral, emotional, and cognitive components (Fredricks et al., 2004, Jimerson et al., 2003). However, few existing instruments are adequate for addressing this model. In summary, within the varied approaches, studying school engagement globally and studying it in terms of its component parts, there is a lack of consistency in the specific measurement tools that researchers use.

We have sought to develop an instrument that captures all three of aspects of school engagement and to study them simultaneously given that they are likely to be interrelated. Thus our instrument development was guided explicitly by the theoretical framework proposed by Fredricks et al. (2004). Like other recent papers in this field, we have aimed to create a theoretically-grounded instrument that captures multiple aspects of school engagement (see Appleton et al., 2006, Martin, 2007).

Another potential weakness in common practices of measuring engagement is the lack of accounting adequately for the error present in its measurement (e.g., Bowen and Richman, 1995, Connell and Wellborn, 1991, Jimerson et al., 2003, Skinner et al., 2008). Across studies, researchers often apply different measurement strategies and design different indicators to measure engagement. However, because each of these items or indicators represents that student's level of engagement imperfectly, analyses that use such an index are vulnerable to the negative consequences of measurement error (Bollen, 1989). Thus, we use confirmatory factor analysis (CFA) to take measurement error into account by estimating the measurement error variances associated with each indicator directly.

Prior research indicates that students' level of school engagement and academic performance may differ profoundly by gender and race/ethnicity (Brooks-Gunn & Duncan, 1997). Generally, throughout schooling, girls report higher levels of school engagement than boys regardless of what types of engagement are considered (see Johnson et al., 2001, Martin, 2004, Martin, 2007, Zimmerman and Martinez-Pons, 1990). Research on engagement has produced mixed evidence on racial–ethnic differences, despite most studies indicating that African American students are less likely to engage in their schools than European American students (Ainsworth-Darnell & Downey, 1998). For instance, Johnson et al. (2001) found that African-American students reported lower levels of school attachment but were more likely to pay attention and complete homework, whereas Voelkl (1997) found that African American students had higher levels of school identification than European American students. However, most of these studies have not considered the implications of measurement invariance when making group comparisons (Martin, 2007). Measurement invariance generally refers to the extent to which the content of each item is being perceived and interpreted in the same way across samples (Byrne & Watkins, 2003). In fact, if measures of engagement operate differently across gender and ethnic groups and these variations are not taken into account in the measurement, it is inappropriate to compare levels of engagement or its effects across groups (Glanville & Wildhagen, 2007).

It is important to investigate whether the measurement of engagement functions similarly across groups of students. By doing so, levels of engagement can be compared appropriately from group to group or common scores of engagement can be used to predict school achievement for students with different demographic characteristics (e.g., Byrne and Watkins, 2003, Reise et al., 1993, Wicherts et al., 2005). To confirm measurement invariance, we must test the equivalence of a measured construct across two or more independent groups to assure that the construct is being assessed similarly in each group (Chen, Sousa, & West, 2005). Meaningful comparisons of statistics – such as means and regression coefficients – can only be made then. Without assuring measurement invariance, group comparisons described by researchers are likely to be substantively misleading and potentially artifactual (e.g., Byrne, 1998, Byrne and Stewart, 2006, Thompson and Green, 2006, Van de Vijver and Leung, 1997).

Testing for measurement invariance includes a series of hierarchical steps, beginning with the establishment of a baseline model in each group, followed by tests for equivalence across groups at several increasingly more restricted levels. These steps involve configural, factor loading, and intercept levels based on techniques developed by Meredith, 1993, Widaman and Resie, 1997. The first step (configural invariance) is to test whether each group has the same number of dimensions and patterns of fixed and free parameters (Bollen, 1989). If the fit of this baseline model is acceptable, higher levels of invariance could be examined. The second step is to assess whether the factor loadings for the latent variables are invariant across groups. If this condition holds, one can test whether the intercepts are invariant (Widaman & Resie, 1997).

The main goal, in this study, was threefold. First, we propose and fit a multidimensional factor model of school engagement. Second, we examine the extent to which this model demonstrates measurement invariance by gender and race/ethnicity. Third, we compare latent variable mean differences across groups if an adequate level of measurement invariance is present. In doing so, we address the following research questions:

  • 1.

    Does the construct of school engagement display a second-order multidimensional factor structure in which each of three hypothesized second-order factors subsumes two specified first-order factors?

  • 2.

    Does the multidimensional factor model of school engagement demonstrate measurement invariance separately by gender and race/ethnicity?

  • 3.

    Are there gender and race/ethnic group differences in school engagement and is the pattern of difference the same across all types of school engagement?

We present our hypothesized measurement models in Fig. 1. We hypothesize that the indicators that we used to measure school engagement will display a first-order factor structure that contains six factors, with each of the six factors composed of multiple items, as indicated. These first-order factors are as follows: (a) attentiveness, (b) school compliance, (c) school belonging, (d) valuing of school education, (e) self-regulated learning, and (f) cognitive strategy use. We further hypothesize that these six first-order factors will display a tri-dimensional second-order factor structure with each of the three second-order dimensions subsuming two first-order factors: (a) the behavioral dimension includes school attentiveness and compliance, (b) the emotional dimension includes school belonging and valuing of school education, and (c) cognitive dimension includes self-regulated learning and cognitive strategy use. In addition, we hypothesize that girls will display greater behavioral, emotional, and cognitive engagement than boys. Given the mix of findings on race/ethnic group differences, we make no specific hypotheses regarding these differences.

Section snippets

Dataset

The data are drawn from the Maryland Adolescent Development in Context Study (MADICS), an ongoing longitudinal study of more than 1000 adolescents, their families, and their teachers. The MADICS was conducted for two main purposes: to examine how social context influences psychological determinants of behavioral choices and to examine various developmental trajectories during adolescence and into adulthood. This dataset contains rich descriptors of the home, school, and peer group that may

Assessing the measurement model

Confirmatory factor analysis (CFA) of the students' responses to the 23 items of school engagement verified that the hypothesized factors were measured by discrete, single latent variables. In Fig. 2, we present the factor models with all of the parameter estimates displayed. The six factors were allowed to inter-correlate simultaneously to specify the measurement model, which represents a first-order model. The standardized loadings ranged from .50 to .79 and were all statistically significant

Discussion

In the present study, we tested the multidimensional theoretical conceptualization of school engagement proposed by Fredricks et al. (2004). To do this, we examined the psychometric properties and measurement invariance of a series of questionnaire items that were designed to measure middle-school students' levels of school engagement. Specifically, based on Fredricks et al. (2004), we hypothesized that there was a second-order factor structure for the school engagement, with six first-order

Acknowledgments

The article is adapted from a doctoral dissertation by Ming-Te Wang submitted to the Graduate School of Education at Harvard University. This research was supported by dissertation grants from the American Psychological Association (Division 15) and Harvard University awarded to Ming-Te Wang. The first author would like to thank Robert Selman and Stephanie Jones, members of his dissertation committee, and Terry Tivnan for their helpful feedback.

References (68)

  • J.J. Appleton et al.

    Measuring cognitive and psychological engagement: Validation of the student engagement instrument

    Journal of School Psychology

    (2006)
  • A.N. Baraldi et al.

    An introduction to modern missing data analyses

    Journal of School Psychology

    (2010)
  • L.S. Aiken et al.

    Structural equation analyses of clinical subpopulation differences and comparative treatment outcomes: Characterizing the daily lives of drug addicts

    Journal of Consulting and Clinical Psychology

    (1994)
  • J.W. Ainsworth-Darnell et al.

    Assessing the oppositional culture explanation for racial/ethnic differences in school performance

    American Sociological Review

    (1998)
  • J.C. Anderson et al.

    Structural equation modeling in practice: A review and recommended two-step approach

    Psychological Bulletin

    (1988)
  • R.P. Bagozzi et al.

    On the evaluation of structural equation models

    Journal of the Academy of Marketing Science

    (1988)
  • A. Bandura et al.

    Multifaceted impact of self-efficacy beliefs on academic functioning

    Child Development

    (1996)
  • P.M. Bentler

    Comparative fit indices in structural equation models

    Psychological Bulletin

    (1990)
  • P.M. Bentler

    EQS 6.1: Structural equations program manual

    (2005)
  • K.A. Bollen

    Structural equations with latent variables

    (1989)
  • G.L. Bowen et al.

    The school success profile

    (1995)
  • J. Brooks-Gunn et al.

    The effects of poverty on children and youth

    The Future of Children

    (1997)
  • B.M. Byrne

    Structural Equation Modeling with Lisrel, Prelis, and Simplis: Basic concepts, applications, and programming

    (1998)
  • B.M. Byrne

    Structural equation modeling with AMOS: Basic concepts, applications, and programming

    (2001)
  • B.M. Byrne et al.

    Teacher's corner: The MACS approach to testing for multigroup invariance of s second-order structure: A walk through the process

    Structural Equation Modeling: A Multidisciplinary Journal

    (2006)
  • B.M. Byrne et al.

    The issue of measurement invariance revisited

    Journal of Cross-Cultural Psychology

    (2003)
  • K. Caraway et al.

    Self-efficacy, goal orientation, and fear of failure as predictors of school engagement in high school students

    Psychology in the Schools

    (2003)
  • F.F. Chen et al.

    Testing measurement invariance of second-order factor models

    Structural Equation Modeling

    (2005)
  • G.W. Cheung et al.

    Evaluating goodness-of-fit indexes for testing measurement invariance

    Structural Equation Modeling

    (2002)
  • J.P. Connell et al.

    Competence, autonomy, and relatedness: A motivational analysis of self-system processes

  • T.D. Cook et al.

    Some ways in which neighborhoods, nuclear families, friendship groups, and schools jointly affect changes in early adolescent development

    Child Development

    (2002)
  • M. Csikszentmihalyi et al.

    Becoming adult: How teenagers prepare for the world

    (2000)
  • B.P. Daly et al.

    School engagement among urban adolescents of color: Does perceptions of social support and neighborhood safety really matter?

    Journal of Youth and Adolescence

    (2009)
  • J.S. Eccles et al.

    Development during adolescence: The impact of stage/environment fit

    The American Psychologist

    (1993)
  • X. Fan et al.

    The effects of sample size, estimation methods, and model specification on SEM fit indexes

    Structural Equation Modeling: A Multidisciplinary Journal

    (1999)
  • J.D. Finn

    Withdrawing from school

    Review of Educational Research

    (1989)
  • J.D. Finn et al.

    Academic success among students at risk for school failure

    The Journal of Applied Psychology

    (1997)
  • J.D. Finn et al.

    School characteristics related to school engagement

    Journal of Negro Education

    (1993)
  • J.A. Fredricks et al.

    School engagement: Potential of the concept, state of the evidence

    Review of Educational Research

    (2004)
  • J.L. Glanville et al.

    The measurement of school engagement: Assessing dimensionality and measurement invariance across race and ethnicity

    Educational and Psychological Measurement

    (2007)
  • J.T. Guthrie et al.

    Engagement and motivation in reading

  • L.-T. Hu et al.

    Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification

    Psychological Methods

    (1998)
  • L.-T. Hu et al.

    Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives

    Structural Equation Modeling: A Multidisciplinary Journal

    (1999)
  • L.-T. Hu et al.

    Can test statistics in covariance structure analysis be trusted?

    Psychological Bulletin

    (1992)
  • Cited by (273)

    View all citing articles on Scopus
    View full text