Carrot and stick?: Impact of a low-stakes school accountability program on student achievement

doi:10.1016/j.econlet.2015.10.007

Economics Letters

Volume 137, December 2015, Pages 195-199

https://doi.org/10.1016/j.econlet.2015.10.007 Get rights and content

Highlights

•
This study examines a school accountability program providing financial resources.
•
This study uses the regression discontinuity design for the causal inference.
•
The low-stake accountability program significantly improves student performance.

Abstract

A key concern in the design of education policies relates to the structure of incentives in accountability systems. This paper examines a school accountability program that provides financial support to low-performing schools but has no direct punishment scheme for recipients who do not exhibit improvement. Although the program does not include high-stakes consequences, our estimates indicate that the program reduced the share of underperforming students by 18%. This paper’s results suggest that to improve student achievement, a school accountability program does not need to set high-stakes consequences that potentially induce unwanted strategic behaviors on the part of school workers.

Introduction

One of the main concerns in education policy-making is how to properly structure incentives in school accountability programs (e.g., Figlio and Loeb, 2011 and Neal, 2011). In the US, such accountability programs are implemented by the federal government (e.g., the No Child Left Behind Act of 2001) as well as by states and localities (e.g., the 2007 New York City accountability system that established school progress reports).¹ These accountability programs set performance goals that schools are required to meet and impose consequences on schools deemed to be low performing. Rarely do such programs provide low-performing schools with additional resources to meet these goals. Furthermore, whether or not performance goals are met can significantly affect school employees in terms of their monetary compensation, employment, and promotions.

While this high-stakes “carrot and stick” approach provides a significant incentive for school workers to meet performance goals, it may also lead to counterproductive actions that could jeopardize the educational system (e.g., manipulation of test scores and shifting of resources to test subjects from non-test subjects).² Therefore, it is important to examine whether an accountability program that imposes weaker financial incentives–and thus imposes a lower risk of strategic behaviors–can improve school performance as much as the current high-stakes programs. The results of this paper suggest that it may be possible to improve student achievement without relying on heavy financial incentives.

We study a national accountability program called the “School For Improvement” (SFI herein) program that was introduced in South Korea in 2010. Similar to accountability programs in the US, the SFI program classifies schools according to student achievement and aims to help failing schools. Unlike the US programs, the SFI program provides all failing schools with substantial financial resources that they can use to increase human resources by hiring additional teachers’ aides, acquiring consulting services for teaching skills, and offering additional classes after school. The SFI program does not provide financial incentives to school workers based on post-period student performance; rather, it uses indirect pressure from peers and parents by publicly announcing schools’ performance.

The SFI program identifies a school as failing when at least 20% of its students are classified as “underperforming” based on the baseline assessment in 2009. Due to this selection rule, we use a regression discontinuity (RD) design to estimate the impact of the SFI program on student achievement. We use a nationwide administrative dataset containing reliable measures of student performance in 2009 and 2010. We find that the SFI program leads to a significant 3.7 percentage point (18%) reduction in the percentage of students classified as underperforming and significantly improves student achievement in all test subjects (Reading, Math, English, Science, and Social Studies). These results are robust to various specifications and alternative empirical models.

Our results suggest that a well-defined low-stakes accountability program that provides failing schools with additional resources may be able to improve student achievement, just like a high-stakes accountability program, while imposing a lower risk of inducing unintended behavior on the part of school workers. Therefore, this paper calls for further studies on designing the proper level of incentives for school improvement.

Section snippets

Institutional background and data

Since 2009, the South Korean government has conducted a national test every year to evaluate all students enrolling in grade 6 (the final grade of primary school), grade 9 (the final grade of middle school) or grade 11 (the second year of high school).³

Econometric framework and identification strategy

We employ the fuzzy regression discontinuity design in the spirit of previous work on the causal impact of school accountability policies on student achievement (e.g., Chiang, 2009, Rockoff and Turner, 2010, Dee, 2012 and Rouse et al., 2013). We use the discontinuous relationship between a school’s treatment status and its average share of underperforming students in the baseline year to identify the causal impact of SFI eligibility on student achievement. Specifically, we estimate local OLS

Results

We first present graphical evidence of the positive effect of the SFI on school performance. In Fig. 1 (Panel B), we plot the average share of underperforming students in 2010–the year after the program was implemented–for each bin of baseline performance. For easy comparison, we include two solid lines in the graph that are the estimated quadratic polynomial for both sides of the threshold.⁵

Conclusion

This paper examines the impact of a school accountability program that provides financial support to failing schools but has no direct punishment scheme for schools that do not exhibit improvement. Although the program does not provide explicit economic incentives for school personnel, we find that it had sizable positive impacts on students’ achievement the year after it was launched. Our results imply that it may be possible to “nudge” schools, without employing any aggressive incentive

References (15)

Hanley Chiang
How accountability pressure on failing schools affects student achievement
J. Public Econ.
(2009)
Kristof De Witte et al.
Public expenditures, educational outcomes and grade inflation: Theory and evidence from a policy intervention in the Netherlands?
Econ. Educ. Rev.
(2014)
Derek Neal
The design of performance pay in education
Carneiro, Pedro, Koussihouede, Oswald, Lahire, Nathalie, Meghir, Costas, Mommarts, Corina, 2015. Decentralizing...
Dee, Thomas, 2012. School turnarounds: Evidence from the 2009 stimulus. NBER Working Paper No....
David Figlio et al.
School accountability
Gelman, Andrew, Imbens, Guido, 2014. Why high-order polynomials should not be used in regression continuity designs....

There are more references available in the full text version of this article.

Cited by (11)

Higher education responses to accountability
2024, Economics of Education Review
This paper estimates the impact of accountability scores on Brazilian higher education. We implement a regression discontinuity design to explore a natural experiment resulting from the federal rules for assigning quality levels to undergraduate programs. We test whether program quality is sensitive to negative reinforcement, such as punishments imposed when a minimum threshold is not attained. The findings indicate that program administrators exhibit a positive response to the prospect of punishment by enhancing program quality in the subsequent evaluation cycle. The primary drivers of this advancement – infrastructure, teaching and learning evaluations, faculty dedication, and the proportion of faculty with a Ph.D. degree – are largely under the administrators’ direct control. However, quality indicators less subject to administrative manipulation, such as student performance and value-added measures, exhibit minimal change.
Does funding targeted at improving the management of schools increase school performance over the long term?
2021, Economics Letters
There is a large literature that links school expenditures to student outcomes. However much of this research focuses on general funding increases rather on the purposes to which additional expenditures are allocated. This research considers the effect of a policy in England that ran from 2003–2006 that provided additional funding to low performing schools specifically targeted at improving the middle and senior management capabilities of the school. Using a sharp regression discontinuity design, this study finds that the funding did increase school performance, suggesting that targeted, time limited funding to improve school management can increase school performance over the longer term. The analysis also shows that it took 4 years from the start of the programme for these positive effects to arise (one year after the funding ended). Thus underlining the importance of allowing school improvement programmes sufficient time to demonstrate effectiveness.
Low-stakes standardized tests in British Columbia, Canada: system accountability and/or individual feedback?
2023, Education Economics
Predicting secondary school students’ academic achievement from their elementary school performance and learning behaviours: A longitudinal study based on South Korea’s national assessment of educational achievement
2023, Asia Pacific Journal of Education
“We Want to Make our Customers Happy”: How Principals of Zoned Elementary Schools Navigate Inequity and Marketing Across Neighborhood Contexts
2022, Urban Education
Taking Education to the Next Level: What Can Be Learned from Benchmarking Education across Economies?
2020, SSRN

View all citing articles on Scopus

^☆: We thank Lesley Turner and Ben Malin for detailed suggestions. All errors are our own. Woo acknowledges the financial support by the 2013 Research Fund of Myongji University. The Online Appendix is available at www.soohyunglee.com/research.

View full text

Carrot and stick?: Impact of a low-stakes school accountability program on student achievement☆