Comparing opportunity to learn and student achievement gains in southern African primary schools: A new approach
Highlights
► We estimate the impact of OTL on gains using a classroom fixed effects approach. ► We found very different results for South African and Botswana classroom samples. ► Botswana students averaged greater mathematics gains than South African students. ► In South African classrooms more lessons on test items have a positive impact. ► In Botswana classrooms increased content emphasis has a negative effect on gains.
Introduction
A popular explanation for low student achievement in many developing countries’ primary schools is that students have relatively little opportunity to learn the skills needed for academic success (Benavot and Amadio, 2004, Abadzi, 2009). However logical the explanation may be, surprisingly little empirical evidence has been presented to support it. We address this gap by estimating the effect of opportunity to learn (OTL) on students’ academic achievement using rich data we gathered on the teaching process in a large number of South African and Botswana classrooms.
There is a long history of OTL as a key issue in student achievement. In its earlier incarnation in developed countries, OTL referred to the amount of time required for students with different abilities to learn the same curriculum (see, for example, Carroll, 1963, Berliner, 1990). In developing countries, OTL is defined more broadly as the amount of time that students are exposed to the curriculum in a given academic year. The educational effectiveness literature further breaks down OTL into “content coverage,” “content exposure” and “content emphasis.” The term “content coverage” is used to refer to the topics and sub-topics taught (Thompson and Senk, 2001, Porter and Smithson, 2001, Schmidt et al., 2001, Stigler and Hiebert, 2004). “Content exposure” refers to the amount of time devoted to a given subject overall (Rosenshine and Berliner, 1978, Floden, 2003, Lee, 1982, Wang, 1998, Porter and Smithson, 2001, Schmidt et al., 2001). “Content emphasis” refers to the relative amount of time students spend on the various topics (Wang, 1998).
Most empirical research on OTL in developing countries has focused on measuring content coverage and content exposure (Gillies and Quijada, 2008, DeStefano et al., 2006, Moore et al., 2006). For example, the scheduled instructional time in developing country primary schools is often low compared to the minimum 850 h annually that experts claim is needed for a “quality education” (Benavot and Amadio, 2004). Schools in many developing countries are characterized by considerable teacher absenteeism (Chaudhury et al., 2005, Beteille, 2009) and student absenteeism may in part depend on teacher absenteeism (Marshall, 2003). Beyond absences, teachers are often present at school but not in their classrooms teaching, or are in their classrooms “teaching,” but are not teaching the required curriculum or are often not engaging the students in a meaningful way, especially on parts of the curriculum they feel uncomfortable teaching (Stallings, 1980).
This growing literature has identified the generally poor and highly unpredictable delivery of the official curriculum to learners as a potentially important problem, particularly in low and middle-income societies. Yet, analysts have done little to link variation in OTL to variation in student learning outcomes, providing limited evidence that increasing any of the usual measures of OTL improves student performance. There are a few exceptions. International cross section data from the Third International Mathematics and Science Survey (TIMSS) suggested that mathematics curriculum exposure has a positive effect on 8th grade student mathematics scores (Schmidt et al., 2001). Marshall in Guatemala (2003) and Reeves in the Western Cape Province of South Africa (2005), use student pre- and post-tests and classroom observations to estimate that teacher time use in primary school classrooms (Marshall, 2003) and the coverage of the required curriculum over a number of school years (Reeves, 2005) are associated with greater student learning gains.
In this paper, we attempt to provide further empirical evidence for whether OTL contributes to student learning. The study is innovative in two ways.
- •
It compares the impact of OTL on the academic performance of very linguistically and socio-culturally similar students in the border regions of two educational systems—South Africa (North West Province) and Botswana—with different development histories but almost identical national curricula. Students in both systems are relatively low scoring, and low OTL has been identified as a problem, particularly in South Africa (Fleisch, 1999, Taylor et al., 2003). This allows us to test whether variation in OTL may have different impacts on student learning in different educational contexts.
- •
We use a classroom fixed effects approach to estimate the impact of OTL on students’ mathematics achievement gains in Grade 6 classrooms in a large sample of schools on either side of the border. We were able to get very detailed data on curriculum coverage (content coverage, content exposure, and content emphasis) from student notebooks, since in both countries teachers require students to record activities from all lessons in their individual notebooks. Using this detailed information on what took place in classrooms during the academic year, we were able to relate frequency of specific curriculum coverage in each classroom (content emphasis) to student achievement gains item-by-item on a 40-item mathematics test in that same classroom.
The focus of our study in linking OTL to student mathematics performance is on the content (topics and sub-topics) made available in classrooms during the academic school year. In particular, we wanted to establish the extent to which Grade 6 students had the time and opportunity to learn the mathematics topics and sub-topics associated with the content in the items in the test we administered to students in our sample classrooms; that is, the “content emphasis” element of opportunity to learn.
We found statistically significant but very different results for our South Africa and Botswana classroom samples. The discussion of those results in the context of differences in the two school systems provides insights into the importance and limits of OTL as an explainer of student learning in low achievement schools.
Section snippets
The context
The school systems in Botswana and North West Province are imbedded in the same Setswana culture and language and both use English as the main teaching language in upper primary grades. However, Botswana gained independence from Britain in 1966 and developed its education in the context of national goals that were designed to serve the entire population of Botswana equally. South African education was dominated by apartheid policies that separated Africans’ schooling from white and so-called
Data
The data for the present study were collected as part of a larger study in a random sample of 116 public schools drawn from the population of no-fee public schools in both urban and rural areas located in the border region of the North West Province in South Africa and the Southeast region of Botswana (Carnoy et al., 2012). The study measured student learning gains, student family background variables, teacher background variables and mathematics knowledge, and data at the school level for 116
Method
A typical value-added model to estimate the effect of a classroom input on student outcomes relates the post-test score to the input, controlling for students’ pre-test score. Economists argue that this model assumes that students’ knowledge “decays” over the course of a year at a constant rate (an estimate of the decay is 1 minus the pre-test coefficient), with part of the treatment coefficient attributed to replacing the decay in knowledge. The model presents a problem in that the student's
Student mathematics achievement gains
The average mathematics achievement in our sample of more than 3800 students in North West Province who took the initial test was 28.6%, with a standard deviation of 12.2. The average test score in our sample of 1774 Botswana students who took the initial test was 33.6% (including items 27 and 37, which had printing issues in the Botswana version of the test that made them difficult to read) with a standard deviation of 12.0.3
Discussion
Our findings show that in North West Province schools, topic coverage matters for student learning, confirming what others have claimed in more general terms (Schmidt et al., 2001). We cannot argue definitively that lessons per item cause higher test score gains in North West Province, since, as we show elsewhere (Carnoy et al., 2012), the number of lessons given seems to be correlated with teacher mathematics knowledge across classrooms. Nevertheless, our estimates in Table 2a implicitly
Acknowledgements
Thanks go to the schools, teachers, students and fieldworkers who participated in the study. We are grateful to the Spencer Foundation for their generous financial support and to Henry M. Levin for his insightful comments.
References (33)
Instructional time loss in developing countries: concepts, measurement, and implications
World Bank Research Observer
(2009)- Benavot, A, Amadio, M., 2004. A Global Study of Intended Instructional Time and Official School Curricula, 1980–2000....
‘What's all the fuss about instructional time?’
- Beteille, T., 2009. Absenteeism, transfers, and patronage: the political economy of teacher labor markets in India....
A model of school learning
Teachers College Record
(1963)- et al.
Missing in Action: Teacher and Health Worker Absence in Developing Countries
(2005) - et al.
How we should measure change: or should we?
Psychological Bulletin
(1970) - et al.
Reaching the Underserved through Complementary Models of Effective Schooling
(2006) School management and teacher accountability
Education Africa Forum
(1999)