Elsevier

Journal of Public Economics

Volume 152, August 2017, Pages 154-169
Journal of Public Economics

The effects of test-based retention on student outcomes over time: Regression discontinuity evidence from Florida

https://doi.org/10.1016/j.jpubeco.2017.06.004Get rights and content

Highlights

  • Test-based third-grade retention increases short-run reading and math achievement.

  • The achievement gains from retention fade out entirely within five years.

  • Test-based retention improves course grades in high school.

  • Test-based retention has no effect on high school graduation rates.

Abstract

Many American states require that students lacking basic reading proficiency after third grade be retained and remediated. We exploit a discontinuity in retention probabilities under Florida's test-based promotion policy to study its effects on student outcomes through high school. We find large positive effects on achievement that fade out entirely when retained students are compared to their same-age peers, but remain substantial through grade 10 when compared to students in the same grade. Being retained in third grade due to missing the promotion standard increases students' grade point averages and leads them to take fewer remedial courses in high school but has no effect on their probability of graduating.

Introduction

Sixteen states and the District of Columbia have recently enacted policies requiring that students who do not demonstrate basic reading proficiency at the end of third grade be retained and provided with remedial services (Workman, 2014). Similar policies are under debate in states and school districts across the nation. Although these policies aim to provide incentives for educators and parents to ensure that students meet performance expectations, they can also be expected to increase the incidence of retention in the early grades. Their enactment has therefore renewed a longstanding debate about retention's consequences for low-achieving students.

Roughly 10% of American students are retained at least once between kindergarten and eighth grade, with the incidence of retention concentrated among low-income students and traditionally disadvantaged minorities (Planty et al., 2009). Retaining students in the same grade is costly in terms of additional per pupil spending and foregone earnings, if students (as intended) spend an additional year in full-time public education as a result of being held back. Yet consensus is lacking as to whether retention yields benefits for students that could offset these costs and, if so, under what conditions.

Proponents of policies encouraging the retention of low-performing students contend that these students stand to benefit from an improved match of their ability to that of their peers, from the opportunity for additional instruction before confronting more challenging material, and from any additional services provided to students during the retention year. Critics, meanwhile, warn that retained students may be harmed by stigmatization, reduced expectations for their academic performance on the part of teachers and parents, and the challenges of adjusting to a new peer group. In fact, a large literature in educational psychology confirms that retained students achieve at lower levels, complete fewer years of school, and have worse social-emotional outcomes than observably similar students who are promoted.1 Because the decision to retain a student is typically made based on characteristics unobserved by the researcher, however, even studies that match retained and promoted students based on prior academic achievement are likely to suffer from selection bias. Consistent with this, more recent research in economics exploiting credibly exogenous variation in retention probabilities has found less negative and, in some cases, positive effects on student outcomes (Jacob and Lefgren, 2004, Jacob and Lefgren, 2009, Greene and Winters, 2007).

In this paper, we use statewide administrative data covering all students in Florida public schools to study the causal effect of third grade retention and remediation on student outcomes through high school. The primary outcomes we examine include test scores for eight years following potential third grade retention in reading and six years in math, subsequent retention rates, and high school grade point average (GPA), coursetaking patterns, and graduation outcomes. The Florida database has four key advantages for studying the consequences of grade retention.

First, since 2003 Florida has required that schools retain third grade students who do not demonstrate basic proficiency on the state reading test, unless the student is eligible for one of a specified set of exemptions. This test-based promotion policy generates a discontinuity in the probability of retention at the test score cutoff used to determine reading proficiency. We can therefore employ a standard regression discontinuity design to overcome the selection issues plaguing most existing research on this topic (Jacob and Lefgren, 2004, Jacob and Lefgren, 2009, Greene and Winters, 2007, Winters and Greene, 2012).

Second, the Florida database contains vertically scaled test scores in reading and math that make it possible to compare the achievement of students tested in different grades during the same year. Making this comparison is essential because the counterfactual condition for students who are retained is to have been immediately promoted to the next grade. While often the sole focus of studies of retention, same-grade comparisons conflate any effect of retention with the effect of being a year older and having an additional year of schooling at the time the relevant test is administered.

Third, the availability of annual test scores for up to eight years after the retention decision makes it possible to determine the extent to which any changes over time in the magnitude of the estimated effect of retention are driven by grade-specific effects on achievement. The average amount students learn varies across grades for reasons including differences in teacher quality, the alignment of curricula with test content, and the share of students making school transitions. Because estimates of retention effects based on same-age comparisons capture these grade-specific effects along with the isolated effect of being retained, studies examining the outcomes of retained students after only two years (e.g., Jacob and Lefgren, 2004, Greene and Winters, 2007) are unable to determine whether any short-term effects of retention persist, fade out, or grow larger over time.

Finally, the availability of high school transcript and graduation data through 2014 makes it possible to study the effects of test-based retention on students' course-taking patterns and performance in high school and on the probability that they graduate. For the first cohort of students affected by the policy, we are also able to provide a preliminary analysis of effects on enrollment in a Florida college.

It is important to note that the Florida policy requires that retained students be given the opportunity to attend a summer reading program prior to the next school year and that they be assigned to a “high-performing” teacher and receive intensive reading interventions during that year. Our estimates of the policy's impact will therefore capture the combined effect of retention and these additional measures and may not be directly comparable to those of some previous studies of retention. Requirements that retained students receive remedial interventions are typical of test-based promotion policies in use and under consideration in other settings, however, giving our results considerable policy relevance.

Due to the availability of exemptions for students scoring below the promotion cutoff, as well as to the voluntary retention of some higher-scoring students, our regression discontinuity design is fuzzy and yields estimates local to students who are retained as a result of the policy but would otherwise have been promoted (i.e., compliers). From a policy perspective, this local average treatment effect is arguably the most relevant parameter. Teachers granting a low-scoring student an exemption or recommending that a student with higher test scores be retained presumably do so because they have strong views as to whether retention would be beneficial for the student in question. In the case of compliers, in contrast, the fact that retention occurs only as a result of the test-based promotion policy implies that local educators are uncertain about whether retention is desirable. Moreover, because the retention policy is based on reading scores alone, we can exploit variation in compliers' math achievement to provide suggestive evidence that our estimates are generalizable to a broader population in terms of third grade achievement.

Our analysis confirms that students retained in third grade under Florida's test-based promotion policy experience substantial short-term gains in both math and reading achievement. On average over the first three years after being held back, retained students outperform their same-age peers who were promoted by 0.31 standard deviations in reading and by 0.23 standard deviations in math. These positive effects fade out over time, becoming statistically insignificant in both subjects within five years, but retained students continue to outperform their promoted peers when tested in the same grade through grade eight in math and grade ten in reading. Consistent with this evidence of improved performance against grade-level expectations, we find that being retained in third grade as a result of missing the promotion standard improves students' grade point averages (GPAs) and leads them to take fewer remedial courses in high school. Test-based retention delays students' graduation from high school by 0.63 years and leads them to complete slightly fewer overall course credits, but has no effect on their overall probability of graduating or their probability of receiving a regular diploma.

These findings contribute to an emerging literature using quasi-experimental research designs to study the effects of retention policies.2 In prior studies of the Florida policy, Greene and Winters (2007) find that third grade retention improved student achievement after two years, and Winters and Greene (2012) present evidence based on same-grade comparisons that these gains persisted through eighth grade. Looking at behavioral outcomes, Ozek (2015) finds that students retained under the Florida policy were disciplined and suspended more frequently in the first two years after being retained, but that these effects dissipated entirely after two years. Jacob and Lefgren (Jacob and Lefgren, 2004, Jacob and Lefgren, 2009) study the impact of retention in third, sixth, and eighth grade on achievement and high school completion in Chicago. They find that retention and mandatory summer school had a small positive short-term effect on achievement for third graders but not for sixth graders. They also find that retention reduced high school graduation rates for eighth graders but not for sixth graders. In a comparative setting, Manacorda (2012) finds that retention in junior high school increases dropout rates for Uruguayan students.

Taken as a whole, this evidence suggests that retention in higher grade levels may have detrimental effects on future student outcomes, but that early grade retention may be more beneficial. We confirm that test-based retention in third grade in Florida improves students' achievement in the short run but show that these initial academic benefits fade out over time. At the same time, test-based retention leads students to perform better academically and need less remediation while enrolled in high school and has no effect on their probability of graduating.

Our evidence that test-based retention in third grade reduces the probability of retention in subsequent grades highlights an additional consequence of policies that increase retention rates in early grades. Specifically, we show that many of the students retained as third graders as a result of Florida's test-based promotion policy would otherwise have been retained in a subsequent grade. After five years, students retained in third grade are, on average, only 0.73 grade levels behind their promoted peers. To the extent that later grade retention is in fact less beneficial, students who are retained earlier rather than later may particularly benefit from the policy.

The paper proceeds as follows. In Section 2 we describe the Florida policy and our data and discuss measurement issues. Section 3 presents our identification strategy and graphical evidence supporting its validity, while Section 4 presents our findings concerning the effects of third grade retention on student outcomes over time, demonstrates their robustness, and examines potential mechanisms. Section 5 concludes.

Section snippets

Test-based retention policy in Florida

In 2002, Florida's legislature mandated that third grade students scoring below level two (of five performance levels) on the Florida Comprehensive Assessment Test (FCAT) reading test be retained and provided with remedial services unless they qualify for one of six “good cause exemptions.”3 The Florida policy's exclusive focus on third grade reading

Empirical strategy

Empirical strategies that rely on a selection-on-observables assumption will fail to provide unbiased estimates of the effect of early grade retention on future student outcomes if students are selected for retention based on factors unobserved by the researcher that influence educational outcomes. We address this concern by taking advantage of Florida's test-based promotion policy, which leads to a discontinuous relationship between third grade reading test scores and the probability of grade

Results

Table 3 reports results from estimating the first-stage model in Eq. (1) for each cohort of students separately and for the pooled sample. For purposes of comparison, we also present results for the two cohorts of students in our data that were not impacted by the policy. Note that all estimations are based on our preferred discontinuity sample within a 10 test-score-point bandwidth around the cutoff. Despite this narrow bandwidth, we still have between 9981 and 15,687 students in each

Conclusion

Our analysis exploits a discontinuity in the probability of grade retention under Florida's test-based promotion policy to study the policy's long-run effects on students retained in the third grade. Based on same-age comparisons, we find evidence of substantial short-term gains in both math and reading achievement. However, these positive effects fade out over time and become statistically insignificant within five years. We also find that test-based retention (and remediation) in third grade

References (33)

  • R. Chetty et al.

    How does your kindergarten classroom affect your earnings? Evidence from project star

    Q. J. Econ.

    (2011)
  • M.M. Chingos et al.

    Do more effective teachers earn more outside of education?

    Educ. Finance Policy

    (2012)
  • B.R. Foorman et al.

    Evaluating the screening accuracy of the Florida assessments for instruction in reading (FAIR) (REL 2013008)

    (2013)
  • J.P. Greene et al.

    Revisiting grade retention: an evaluation of Florida's test-based promotion policy

    Educ. Finance Policy

    (2007)
  • E.A. Hanushek et al.

    Generalizations about using value-added measures of teacher quality

    Am. Econ. Rev.

    (2010)
  • R. Hoffman et al.

    Florida comprehensive assessment test: technical report on vertical scaling for reading and mathematics

  • Cited by (0)

    We are grateful to the Florida Department of Education for providing the primary dataset for this study. We thank Stefan Bauernschuster, Matthew Chingos, Andrew Ho, Paul Peterson, Ludger Woessmann, and the seminar participants at the National Bureau of Economic Research, Harvard University, the Ifo Institute, Mathematica Policy Research, Stanford University, the European Economic Association Meeting in Gothenburg, the European Association of Labour Economists Meeting in Turin and the Swedish Institute for Social Research for helpful comments. The Helios Education Foundation provided financial support for this research. The views contained herein are not necessarily those of the Helios Education Foundation. Any errors are our own.

    View full text