Abstract
Purpose
This research investigates the impact of managerial practices implemented by Italian school principals on students' outcomes, using micro-data provided by the National Evaluation Committee for Education (INVALSI) for 2013/14 school year.
Design/methodology/approach
Employing an educational production function, the authors regress a set of student and school's characteristics, enriched by information from a questionnaire filled by school principals to estimate student's score at grade 8 (last year of junior secondary school), also taking into account student's prior achievement (at grade 6 – first year of junior secondary school).
Findings
Indicators about managerial practices have positive coefficients, but low statistical significance. Stronger associations between management variables and test scores are detected for low-SES schools.
Originality/value
The research presented here is particularly innovative in the Italian context, where little evidence exists about the impact of managerial skills in education, though institutional reforms are leading towards a strengthening of school principal's leadership role. In this paper, the authors move a first step by describing managerial practices and their diffusion in different schools and geographical areas within the country. The authors focus the attention on the role of managerial practices (what principals do) and not on the managerial skills (what principals are able to do).
Keywords
Citation
Agasisti, T., Falzetti, P. and Soncin, M. (2020), "School principals' managerial behaviours and students' achievement: An empirical analysis of Italian middle schools", International Journal of Educational Management, Vol. 34 No. 5, pp. 937-951. https://doi.org/10.1108/IJEM-11-2018-0350
Publisher
:Emerald Publishing Limited
Copyright © 2020, Tommaso Agasisti, Patrizia Falzetti and Mara Soncin
License
Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode
1. Motivation and objectives
For policymakers, being able to measure the impact of management quality in the public sector is a key function. Education is an interesting sub-sector in this context, because of the high number of outcome measures that have to be considered (e.g. test scores and non-cognitive skills learnt by students). The most challenging part of measuring management quality in schools is to identify the effect that principals, as school managers, have on educational outcomes. The principals' effect on learning can be of two different kinds. On one side, there is a “direct” effect, related with the managerial practices to create good educational conditions within the school – such as better processes, more efficient organization, attention to evaluation and quality assurance and so on. However, there is also an “indirect” effect of principals that deals with the work conducted with teachers in improving instructional activities. In this perspective, such effect is mainly mediated by the teachers' classroom work, and the role of teachers in influencing student achievement has been broadly investigated through “value-added” measures (e.g. Hanushek and Rivkin, 2010). Far less empirical evidence exists on the impact of principals, although their role as decision-makers places them at the top of a school's organization, where, through their management practices and attitude, they have a potentially high effect on school productivity and effectiveness (Leithwood and Jantzi, 1999; Quinn, 2002).
The objective of this paper is to understand how school management affects student achievement, once individual and school characteristics have been taken into consideration. In order to pursue this objective, we used data from the Italian National Evaluation Committee for Education (hereafter INVALSI) on standardized tests to assess students' mathematical and reading skills. We specified an educational production function (EPF), estimated econometrically through an OLS regression with standard errors clustered at school level (to properly investigate school effects while considering the hierarchical nature of data, i.e. students clustered into schools). Every year, a random group of schools is chosen for the National Sample (NS), the reference group where assessment is monitored by external inspectors. From the school year (SY) 2013/14, NS school principals were asked to complete a detailed questionnaire about their career and their work in practical school management. Because of this additional information, it was possible to conduct an empirical evaluation into the relationship between management practices and the students' scores in tests taken in the last year of junior secondary school (grade 8, year 9, 13–14 years old), taking into proper account their previous test scores in grade 6 (year 7, 11–12 years old) – thus, we tested the potential association between the principals' management practices and the academic achievement of students. The questionnaire also contained personal facts about the principal, such as his/her experience, degree and so on – and previous works have shown that these features can play a certain part in explaining some of the drivers of principals' effectiveness. Precisely, our research question is:
Is there a relationship between the management practices implemented by the school principal, his/her characteristics and the students' results in a standardized test? We investigate this research question by exploring the “net” impact of the principal's managerial practices, meaning that our empirical model controls for some specific contextual variables and school-specific indicators that are likely to affect the students' performance, such as the students' socio-economic status (SES), the size of the school and the classes and so on. We analyse the effect of the principals' managerial practices on student achievement once these contextual and school-specific variables are taken into account – that is, all these factors being equal.
We are aware that the effect of the principals' action is somehow mediated by the role of the teachers (this is the so-called “indirect effect”) and that, in our analysis of student achievement, we should consider the factor of “teacher quality” as it is likely to exert a major impact (Hanushek and Rivkin, 2006). Unfortunately, we do not have the data to control for this element. Looking at school impact on achievement, the only part that we can investigate is the principal's “direct” effect. We have focused our attention on the role of management practices (what principals do) and not on their management skills (what principals are able to do).
The remainder of the paper is organized as follows. Section 2 gives an introduction to the role of school principals in Italy, Section 3 contains the state of the art on this topic and the theoretical framework. Section 4 describes the data set and the methodology implemented, and results of this are discussed in Section 5. Finally, conclusions and policy implications are presented in Section 6.
2. School principals in Italy
It is worth outlining the role of and the selection process for school principals, as well as their main duties. In Italy, for the academic year 2013/14 (the period relating to our data), there were 8,644 schools, in the wider sense of local school authorities or school districts (known as “istituzioni scolastiche”), and they were divided into 41,483 schools in a stricter sense, the individual public (state) schools (known as “plessi scolastici”). We use the term school in the former sense. There were also 13,847 private schools, 72 per cent of which are kindergartens. In the same year, there were 8,053 public school principals, fewer than the number of public/state schools because of a newish decision (2005) to group together small schools (in mountain areas or the smaller islands) under a single school principal, who is given the title of “reggente” (literally regent). This aggregation system adds a layer of complexity to school management as well as brings up the issue of how much time the school principal spends on each school, with a possible backlash to the quality of management practices within these schools.
It is helpful to run quickly through the main points in the process to select principals, which was in operation until the period under study (2013 – after that the system was substantially reformed). Selection was based on an open competitive exam announced by the Regional School Authority (RSA) (i.e. the regional branch of the Ministry for Education). The exam was open to all the teachers with a master's degree who had been permanent members of teaching staff for at least five years. The fact that school principals were chosen only from among teachers ensured that applicants would have the right teaching experience but not necessarily the right administrative and management skills, so these were tested at various stages during the selection process. The competitive exam consisted of two main phases: (1) evaluation of the candidate's CV in order to screen their competencies and verify whether the teacher had joined in organizational activities and/or had held responsibilities at his/her school, and (2) written and oral tests, which were mainly to check the applicant's knowledge of administrative legislation and principles of instructional leadership. The examination board was appointed by the Regional School Authority (RSA), the body that then published the final ranking. Applicants could only apply for positions within a specific regional authority (“Region”), and the RSA allocated principals to the various schools in that region on the basis of their scores in the exam and additional professional titles.
At the time relating to the data used in this research, the duties of a school principal involved, for example, (1) being responsible for the management of the school's financial and instrumental resources, in the main provided by central government; (2) being the school's legal representative; (3) promoting initiatives to ensure the quality of educational processes and engage with cultural, professional and economical resources in the local area; (4) leading and coordinating the human resources in the school, within the constraints set by central government (schools cannot hire or fire teaching staff, who are hired and allocated through a centralized process based on a competitive exam and directly paid by the Ministry of Treasury).
School autonomy was first introduced with law 275/99 and then reinforced with law 165/01. This is the first step in increasing the school principal's decisional power, with responsibilities being transferred from the central government to schools, albeit with specific limitations. First, instructional autonomy is constrained within the specifications set in the educational offer plan (“Offerta Formativa”), specific to each school and approved year by year by the Board of Teachers. The plan sets out the schools' curricular and extra-curricular projects as well as its organizational guidelines. Second, a school's organizational autonomy is restricted to adjusting or tweaking the school calendar and scheduling the number of hours of lessons per week (in accordance with the total number of hours per year defined nationally). Third, autonomy of research is defined as being able to test and develop instructional tools and models. Finally, financial autonomy simply means using money allocated by central government, giving priority to instructional and formative activities.
School principals in the past were assessed periodically by the Director of the Regional School Authority (this was partially to determine the variable component of their salary). Therefore, the evaluation mechanism was quite unstructured, often took the form of self-evaluation and negotiation and was not anchored to measurable indicators of performance. Data about student achievement never came into the assessment of the principals or of their effectiveness.
3. Previous literature and framework
From the academic literature, we selected two different streams about the effect of school management on student achievement.
The first stream deals with measuring the quantitative impact of the principals on the students' scores. Although there is wide variety of analytical approaches to this research question, the results tend to confirm the fact that the principal's actions have an important impact on different measures of student outcome. The methodological challenge involves being able to disentangle the principal's effect from the effect of other school-level factors that are out of the principal's control or are the current effect of decisions taken before the present principal's tenure. Four interesting studies have explored this point, making use of large data sets with information about the principals' features that have recently been made public. Branch et al. (2012) measured the principal's value-added on the test scores of Texas (USA) students between 1995 and 2001, finding that a principal who is ranked one standard deviation (SD) above the average of the quality distribution leads to an annual gain of 0.05 SDs above average for all the students at the school. As a measure of principal leadership, they focused on teacher turnover, under the assumption that highly rated principals are more successful in retaining highly effective teachers. Dhuey and Smith (2014) used data from North Carolina (USA) to find an effect of 0.13 SD in mathematics and 0.10 in reading, which went up to 0.18 and 0.14 when considering the school fixed effects because of the negative relationship between principal and school effects. This is explained as a compensatory matching, where the best principals are intentionally allocated to less effective schools. The approach used by Coelli and Green (2012) on data from British Columbia is particularly interesting for its use of the dynamic principal effect, according to which there is a cumulative effect of the principal over time, dismissing the assumption of time-invariant effects. Under the assumption that the principal will lead the school long enough for his/her effect to be fully completed (the time needed also depends on the effectiveness of the prior principal), a difference of 1 SD in the effectiveness distribution leads to a 2.5 per cent increase in English exam scores and a 2.6 per cent rise in graduation rates. Grissom et al. (2015) employed three alternative models to catch the impact of the principal's performance on student achievement at public schools in Miami-Dade County (USA) over the period between SY 2003/04 and SY 2010/11. They observed a high variation to the principal's effects, depending on the model used, going from 0.18 SDs in mathematics and 0.12 in reading to 0.05 in mathematics and 0.03 in reading for the same principals.
A second stream of the literature focuses on the characteristics of the educational leader, describing the management practices implemented in the school in order to define archetypes of leadership attitudes and activity. On this point, Leithwood and Jantzi (1999) conducted a survey in 94 Canadian elementary schools looking for the effect of a particular leadership style, known as the “transformational leadership”, on student engagement. In accordance with the literature on this topic, they defined transformational leadership as the school head's ability to “foster capacity development and higher levels of personal commitment of organizational goals” (p. 453). Using a structural equation modelling approach, they showed that the school principal can play a role in this process, operating on the organizational conditions at school level, once a mediating variable – family educational culture – is taken into account. On the contrary, Quinn (2002) investigated the role of the school principal as “instructional leader” in 24 project schools in Missouri (USA), measuring the principal's impact on the teachers' instructional practice and the students' engagement. He found a powerful correlation between the dimensions of instructional leadership (where the principal's role is that of resource provider, instructional resource, communicator and visible presence) and student engagement. Waters et al. (2003) implemented a meta-analysis on 70 studies about the effect of leadership on achievement. They classified 21 principal leadership responsibilities and asserted that an SD increase in all 21 areas corresponds to a gain in the average student achievement of 10 percentage points from the mean. Among the several types of school leadership identified in the literature (e.g. Bush and Glover, 2002, provide an overview), instructional leadership is described as the type that is most strongly connected to student outcome. According to Robinson et al. (2008), who provides a meta-analysis on 27 studies between 1978 and 2006, the effect of instructional leadership on student achievement is three to four times that of transformational leadership.
We move on from this design to inserting the role of the school principal within the economic framework that deals with the determinants of student achievement, and that is based on the concept of “educational production function” (EPF) (e.g. Hanushek and Woessmann, 2011). Focusing on school factors, we have considered the role of school management in terms of both the school principals' personal characteristics (e.g. experience and gender) and the management practices which they implemented. We measured this “direct” effect of principals on students' academic achievement – that is the effect obtained by principals through their managerial practices, such as organizational decisions, work on internal processes, quality assurance and evaluation and so on. As we had no information about the role of teachers in mediating the activity of the principals, we have not measured the “indirect” effect of schools' principals, that is, the one that acts through working with teachers and improving their instructional effectiveness. In our model, we have considered both contextual factors (geographical location, school composition and average SES) and antecedent variables (the school principal's personal characteristics) meaning that we could take complete account of their role in influencing the relationship between the school principal's practices (thus, “net” of their characteristics) and student achievement.
Studies about the school management effects in Italy are still in the early stages. Two interesting works have been published by Bloom et al. (2015) and Di Liberto et al. (2013), both developed within the network of the Word Management Survey (WMS). They elaborated a management index by interviewing school principals about their management practices and attitudes and comparing results obtained internationally. In more detail, Bloom et al. (2015) collected data from over 1,800 secondary schools in eight countries (Brazil, Canada, Germany, India, Italy, Sweden, the United Kingdom and United States). Among the countries observed, Italy was far below the average, which suggests that Italian school principals lack the adequate management skills. Similar results were obtained by Di Liberto et al. (2015), who interviewed 338 secondary school principals in Italy and then compared these results with results collected from the same network in Canada, Germany, Sweden, the United Kingdom and United States. They computed an average management index for Italian principals, which is the lowest among the countries analysed.
4. Data and methods
4.1 The data set
The original data set came from INVALSI, which carries out a yearly assessment of Italian students in reading and mathematics. Tests are taken in given school years (at grades 2, 5, 8 and 10 and until SY 2013/14 also at grade 6) and national level, using a representative random sample of schools selected every year (NS). These tests are monitored by external evaluators. In this paper, we have information about the NS's test results for SY 2013/14 at grade 8, a total of 28,145 students in 1,414 schools. We were also given information about test scores of the same cohort of students two years earlier, when they were in grade 6. Because of this particular aspect of the data set, we could formulate a value-added model (VAM), where we could control for prior achievement (e.g. Todd and Wolpin, 2007).
The reason why this data set is new is that it contains a set of additional information about the school principals, resulting from a questionnaire compiled by school principals themselves (introduced in SY 2013/14) and covering aspects such as their career and work in practical school management. Our final data set is, therefore, the result of a three-step process that involved (1) results from tests assessing mathematical and reading skills of students at grade 8 (SY 2013/14, last year of junior secondary school), (2) results from tests assessing mathematical and reading skills of the same students at grade 6 (SY 2011/12, first year of junior secondary school), (3) school questionnaire completed by school principals.
Administrative problems relating to the procedure led to a series of missing data[1], so that our final sample consisted of 8,946 students in 586 schools. In the Technical Annex (available upon request from the authors), we tested the representativeness of the initial sample of 1,414 schools (NS) against the final sample of 586 schools, finding some slight differences at individual level, but no statistically significant differences among school characteristics, with particular regard to management practices.
School principals completed a questionnaire in which they were asked to answer 32 questions clustered into five main groups: principal's attitude towards standardized tests, stakeholder's engagement in school issues, contextual information about the school, personal information about the principal and management practices. These two last points (sections 4 and 5) in the questionnaire were the main subject for attention in our research. We are aware of the fact that, when exploring the associations between the principal's characteristics and practices and student achievement, both these sets of variables are affected by endogeneity problems, and so we consider all the results presented later in this paper as being mainly descriptive and not necessarily causal.
All the questions considered were multiple-choice and respondents were asked about practices in their school or whether they agreed with a given statement and had to answer on a scale from 1 (“I never use this practice” / “I completely disagree with this statement”) to 4 (“I always use this practice / “I completely agree”). As this was a self-assessment questionnaire, we must be cautious in interpreting the results, as they are more perceptions (which are important in terms of giving an overall impression) than externally measured objective management skills. Moreover, school principals may have moved schools in the period under study, so, in other words, there is no certainty that the principals who answered the questionnaire in 2013/14 were the same as those in charge at the school two years beforehand. To control for this eventuality, we added the information about whether the principal was appointed for less than three years (when the answer is “yes”, we cannot straightforwardly attribute the overall principal effect to the current head).
4.2 Measuring management practices
In order to gain a measure of the management practices implemented by school principals, we followed the literature on this theme. We relied specifically on the framework adopted by Bloom et al. (2015) and Di Liberto et al. (2015) that had been developed by the WMS, an organization that measures and benchmarks management practices in different sectors (with research having been conducted in manufacturing, retail, health care and education) – see https://worldmanagementsurvey.org. Their framework identifies five main areas of management, defined according to the sector analysed. In the case of education, this entails:
Operations, which refer to process standardization, use of best practice and decisions about school curriculum.
Monitoring, which indicates the principals' supervision of school activities and performance and the management of anomalies in school processes.
Target setting, which refers to the principal's ability of setting clear goals for the school and managing resources in order to reach them.
People, which indicates whether the principle ensures the teachers' continuous professional development and is able to retain best teachers in the school.
Leadership, which measures the principals' leadership skills and their ability to clearly define roles and responsibilities within the school.
Following this framework, we classified the questions described earlier into five management areas. However, in all the empirical analyses we considered the impact of a general index of management instead of the five indexes separately[2].
Table I presents the results of this procedure, showing the partition and distribution of the school principals' answers about managerial practices across the groups. Four of the questions were excluded from this part of the analysis as it was not possible to define a clear correlation with one of the management classes. We therefore obtained a set of 25 questions, seven on operations, four on monitoring, four on target setting, five on people and five on leadership.
Every question is formulated so that a low value (with a minimum of 1) means the sporadic use of the management practice, a high value (with a maximum of 4) means a frequent implementation of that practice. Starting from this point, we elaborated a general management index, which is the mean of the 25 questions that make up this part of the analysis, and five specific indexes that refer to the management area of interest. Table II gives the descriptive statistics for the indexes, where the general index has a mean of 3.23, which corresponds to the frequent use of management practices[3].
4.3 Empirical modelling
In the empirical analysis, we have considered an EPF in which the result obtained by a student at grade 8 is predicted by his/her prior achievement (at grade 6). This is determined through a set of individual characteristics and a vector of school-level factors, among which managerial practices play a fundamental role. In detail, we estimated the following regression:
Among school-level characteristics, represented by the vector
5. Results
5.1 Baseline results
The baseline results from the estimation of equation (1) are presented in Table III (while the results from the corresponding multilevel model are given in the Technical Annex). The variables are classified at student level, contextual school level and management school level, and the standard errors are clustered at school level. We have also considered the characteristics of the principals in the same, baseline model. The overall variance of the students' test scores that can be explained by the model is higher in mathematics (42 per cent) than in reading (34 per cent).
5.1.1 Individual-level variables
Although individual-specific features of the students are not the key interest of this research, it is worth commenting their association with test scores.
Individual-level variables show a high degree of statistical significance and coherence with existent studies on the Italian educational system (e.g. Brunello and Checchi, 2005). One noteworthy point is that the effect of the SES is higher when making comparison with prior achievement. In determining a student's achievement at grade 8, it weighs three times more than the result obtained by the student at grade 6 (this finding is in line with results obtained by Agasisti and Falzetti, 2017). Among the other factors, being a female student is associated with better results in reading and lower results in mathematics. Being an immigrant student is associated with lower scores in both subjects, but the size of the coefficient is higher in reading than in mathematics, demonstrating the greater difficulty faced by immigrant students in closing the language gap. A similar result is registered for the variables about students who entered the educational system later than the standard year for their age or repeated one or more school years. This is often associated to immigrant status, as demonstrated by our sample, where 30 per cent of immigrant students had also started schooling later than other children of the same age. On the contrary, the coefficient for the relationship between achievement and early-enrolment status (children who started school one year before the standard cohort age) is not statistically significant, albeit negative.
5.1.2 School-level characteristics
Among the school-level characteristics, we can highlight the negative relationship between the students' scores in mathematics and the average SES in the school. This relationship, however, is positive but non-significant for reading, suggesting that the peer effect driven by the SES may differ between subjects. Moreover, despite the negative impact that being an immigrant student has on achievement, the proportion of immigrant students in the school is positively related to the school-level score in the reading test, while it is still positive but non-significant for mathematics. Such a correlation can reveal the positive effect of a diverse student population, once the SES is accounted for. Attending a public or private school is not associated with statistically different test scores. Coherently with previous research, schools in the South obtain (on average) lower test scores than those in the North, about two or three times higher than for any other variable (Agasisti and Vittadini, 2012; Sulis and Porcu, 2015).
5.1.3 Principal's characteristics
When considering the characteristics of the school principals, we did not observe any relevant correlation with student achievement, with the notable exception of a positive association between holding a scientific degree and test scores in mathematics. This latter finding can be interpreted in the light of the particular attention of principals holding a scientific degree to stimulate high-level teaching in scientific subjects, including mathematics. This aside, the findings reported here suggest that, if the principals produce a positive effect on educational results, this is unlikely to be driven by their individual characteristics per se.
Observing the general index of management practices, we note that it has a positive correlation with the students' score, although without statistical significance. A possible interpretation for our finding is that management practices have a general positive connection with the schools' average test scores (after having controlled for several individual- and school-level factors). Its statistically positive effect does not, however, apply to the “average” school, but instead is heterogeneous for some categories of schools – something that we have explored in the next section[6].
5.2 Heterogeneity of management practices' influence
Moving from the baseline results, our aim was to investigate the existence of some mediating variables which, acting both as antecedents and mediators, influence the relationship between student achievement and management practices. In particular, we focused on the composition of the student body, represented by the average SES of the students.
In order to account for the role exerted by the students' socioeconomic background, we analysed the distribution of the schools’ average index of the students' economic, social and cultural status (ESCS) and defined as low-SES (disadvantaged) schools in the first tertile of the distribution and as high-SES schools in the third tertile. The results from the comparison between low and high-SES schools are reported in the Appendix (in the technical annex). The positive correlation between the management index and test scores is higher for low-SES schools and statistically significant when considering the test scores in mathematics. This is consistent with the idea that the effectiveness of principals – that is, the effectiveness of their managerial practices, in our framework – plays a major role where it is needed most, namely in “troubled” schools (see the analysis of principals' leadership as described in Leithwood et al., 2004). On observing a greater variation in school principal quality in low-income schools in their previous study, Branch et al. (2012) highlighted the beneficial impact that high-quality principals have on this category of schools. In the Italian context, Di Liberto et al. (2015) tested the heterogeneous effects of management practices, finding that there was a larger impact for the disadvantaged group of students, a factor that was interpreted as a possible substitution effect between parental investment and school principal management practices. Another alternative (or complementary) explanation is that, in schools where students are more disadvantaged, they put more effort into the test at grade 8, which for them is high-stake (i.e. it counts towards their final exam score) and less into the test at grade 6. It could be the case that a quota of the principals' management-related activity is to help disadvantaged students work harder towards the grade 8 test, so the two explanations can actually co-exist and interact.
6. Management implications, policy suggestions and concluding remarks
In this exploratory study, our aim was to understand how a new data set on management practices in Italy can be adapted to a research design that explores the relationship between the principals' characteristics and activities and the students' results. Given the consistency between our results at individual level and the existent literature on this topic, we argue that the low level of statistical significance of the management index can relate to the structure of the questionnaire, which fails to catch the key factors of management practices that do have an impact on student achievement.
When considering the managerial practices adopted by the principal in a given school, we cannot ignore the impact that can also been influenced by the organizational conditions of the school itself. For example, if the principal has good collaborators at disposal, implementing more effective managerial practices can actually be easier. The importance of “middle management” cannot be neglected, as the very possibility of implementing managerial practices can be favoured or can encounter obstacles depending upon the presence and quality of principal's team of assistants and collaborators – see the discussion in Harris and Jones (2017). Therefore, the questionnaire developed by INVALSI and used in this research does not explicitly look at this specific factor that then must be left for future investigations in the field.
The results presented in this work are innovative – and claim for some specific and straightforward policy implications – with the aim of reinforcing the margins of freedom to exert the managerial practices and skills of the principals. At the same time, the reader should have in mind two limitations of the present study that despite not being harmful for the overall messages and findings of the paper require a specific attention. These limitations are related to (1) the method of assessment and (2) the structure of the questionnaire itself. With reference to the first point, we have to deal with the internal risks of self-assessment instruments. Podsakoff et al. (2003) have presented a review of the literature in which they identified several sources of bias in behavioural research. Among these, “social desirability” – the tendency to respond to items because of their social acceptability – could be a probable source of bias in our context. With reference to the second point, our proposal is that questions about management practices should evaluate “fully-rounded” and comprehensive management practices. An effort in this direction would tap into the huge potential amount of data collected through an administrative process, according to a framework that is standing out for its ability to catch the main aspects of management practices. More work is needed, operating alongside practitioners (i.e. school principals) and INVALSI analysts, to define the pathways to design and introduce an updated questionnaire for school principals.
Managerial practices, by areas of management
| Areas of management | Questions | Never | Sometimes | Often | Always |
|---|---|---|---|---|---|
| Operations | D11j. I take into account test scores when I make decisions on the school curriculum. | 3.0% | 17.2% | 46.9% | 33.0% |
| D12a. In my job, it is important to make sure that educational strategies, approved by the Ministry, are explained to new teachers and applied by more experienced teachers. | 0.3% | 7.0% | 70.6% | 22.0% | |
| D12i. It is important for the school to avoid mistakes in administrative procedures. | 0.0% | 0.3% | 25.8% | 74.0% | |
| D12j. In my job, it is important to solve timetable problems and/or lesson scheduling problems. | 0.2% | 3.1% | 43.2% | 54.0% | |
| D12m. In this school, we work by objectives and/or on the basis of the formative offer plan. | 0.2% | 1.9% | 64.3% | 34.0% | |
| D12n. I define the objectives to be reached by the school personnel. | 4.0% | 25.1% | 56.8% | 15.0% | |
| D12o. I promote an atmosphere of planning aimed at reaching specific objectives. | 0.0% | 0.3% | 56.3% | 43.0% | |
| Monitoring | D11c. I observe educational activities in the classrooms. | 3.0% | 28.2% | 45.1% | 24.0% |
| D11f. I supervise students' works. | 12.0% | 51.9% | 29.9% | 6.0% | |
| D11m. I deal with bothering behaviours in the classes. | 0.3% | 6.3% | 35.8% | 58.0% | |
| D12h. It is important for the school to verify that rules are respected by everybody. | 0.2% | 0.3% | 37.7% | 62.0% | |
| Targets | D11a. I make sure that teachers' professional development activities are in line with the school's educational objectives. | 1.0% | 10.1% | 42.3% | 46.0% |
| D11b. I make sure that teachers work in conformity with the school's educational objectives. | 0.0% | 3.1% | 40.1% | 57.0% | |
| D11d. I use students' scores to revise educational objectives. | 2.0% | 18.9% | 48.5% | 31.0% | |
| D11i. I make sure that teaching activities in the classrooms are in accordance with our educational objectives. | 1.0% | 18.3% | 46.1% | 35.0% | |
| People | D11e. I provide teachers with suggestions for improving their teaching effectiveness. | 2.0% | 31.6% | 46.6% | 20.0% |
| D11g. When a teacher has a problem in the classroom, I take the initiative to discuss with him/her about it. | 0.0% | 5.8% | 35.8% | 58.0% | |
| D11h. I inform teachers on opportunities of disciplinary and educational update. | 0.0% | 3.2% | 28.7% | 68.0% | |
| D11l. When a teacher raises a problem in the classroom, we face it together. | 0.2% | 3.2% | 34.3% | 62.0% | |
| D12d. In my job, it is important to make sure that teachers' skills are improving continuously. | 0.2% | 0.9% | 57.3% | 42.0% | |
| Leadership | D11k. I make sure that responsibilities on the coordination of the school curriculum are clearly defined. | 0.7% | 10.1% | 44.9% | 44.0% |
| D12e. In my job, it is important to make sure that teachers feel responsible for the achievement of school objectives. | 0.0% | 0.5% | 43.7% | 56.0% | |
| D12f. In my job, it is important to be convincing when presenting new projects to parents. | 2.0% | 14.0% | 58.2% | 26.0% | |
| D12g. I can influence decisions on this school that are made by upper-level administrative positions. | 8.0% | 43.3% | 42.8% | 5.0% | |
| D12k. It is important that I contribute to maintain a peaceful atmosphere in the school. | 3.0% | 29.5% | 51.0% | 16.0% |
Note(s): Reclassification into five areas of management proposed by the authors. Percentages refer to the proportion of school answering from “Never” to “Always” over the total number of schools (N = 586)
Managerial indexes: descriptive statistics
| Index | Mean | Std. Dev. | Min. | Max. | Northern | Central | Southern |
|---|---|---|---|---|---|---|---|
| General index | 3.23 | 0.34 | 1.96 | 4 | 3.16 | 3.26 | 3.31 |
| Operations | 3.29 | 0.34 | 2.29 | 4 | 3.24 | 3.31 | 3.36 |
| Monitoring | 3.08 | 0.48 | 1.5 | 4 | 2.97 | 3.11 | 3.2 |
| Targets | 3.28 | 0.52 | 1.75 | 4 | 3.18 | 3.31 | 3.39 |
| People | 3.4 | 0.39 | 2 | 4 | 3.34 | 3.45 | 3.46 |
| Leadership | 3.05 | 0.41 | 1.6 | 4 | 2.99 | 3.07 | 3.11 |
Note(s): Min = 1, Max = 4. N = 586. The general index is computed as the mean of all the 25 questions related to the managerial practices. The other indexes specifically refer to each of the five areas of management. The labels “Northern”, “Central” and “Southern” refer to the three main geographical areas across Italy
Students' test scores as explained by student/school characteristics, principal's managerial practices and principals' characteristics
| Dep. Variable: student's test score | Baseline model | Full model | |||
|---|---|---|---|---|---|
| Variables | Reading | Mathematics | Reading | Mathematics | |
| coef. | coef. | coef. | coef. | ||
| Student-level characteristics | |||||
| Prior achievement (grade 6) | 0.581*** (0.026) | 0.588*** (0.014) | 0.583*** (0.025) | 0.589*** (0.014) | |
| Socio-economic background (ESCS) | 1.562*** (0.162) | 1.573*** (0.142) | 1.556*** (0.160) | 1.568*** (0.143) | |
| Female student | 1.574*** (0.255) | −1.031*** (0.260) | 1.566*** (0.253) | −1.031*** (0.260) | |
| Immigrant student | −1.768*** (0.519) | −1.294*** (0.477) | −1.725*** (0.517) | −1.263*** (0.475) | |
| Early-enrolled student | −0.573 (1.560) | −0.666 (1.401) | −0.208 (1.531) | −0.598 (1.337) | |
| Late-enrolled student | −2.383*** (0.761) | −1.954*** (0.709) | −2.481*** (0.760) | −1.991*** (0.708) | |
| School-level characteristics | |||||
| School average socio-economic background (ESCS) | 0.711 (0.952) | −1.831** (0.852) | 0.664 (0.905) | −1.773** (0.860) | |
| Proportion of female students | −0.034 (0.032) | 0.03 (0.035) | −0.028 (0.029) | 0.026 (0.034) | |
| Proportion of immigrant students | 0.066** (0.026) | 0.035 (0.024) | 0.070** (0.028) | 0.04 (0.026) | |
| Public school | 1.08 (2.200) | −1.847* (1.068) | 0.098 (2.364) | −1.28 (1.179) | |
| School in Central Italy | −0.603 (0.711) | −1.870** (0.857) | −0.583 (0.670) | −1.604* (0.824) | |
| School in Southern Italy | −4.563*** (1.172) | −3.409*** (0.983) | −4.380*** (1.132) | −3.246*** (0.978) | |
| Managerial practices | |||||
| General index (management practices) | 0.622 (1.172) | 1.094 (1.181) | 0.767 (1.181) | 1.288 (1.254) | |
| School principal's characteristics | |||||
| Age of the school principal | 0.091 (0.068) | −0.016 (0.052) | |||
| Female principal (dummy) | −1.057 (0.845) | −0.153 (0.688) | |||
| Multi-school appointed (dummy) | −2.092 (2.215) | 1.072 (0.893) | |||
| Fixed-term contract (dummy) | −1.951 (2.306) | 0.476 (1.345) | |||
| Degree in humanistic studies | 1.299 (0.954) | 0.648 (0.818) | |||
| Degree in scientific studies | −0.573 (1.536) | 2.184** (1.047) | |||
| Experience as school principal | 0.008 (0.054) | −0.111 (0.069) | |||
| Experience at current school | 0.047 (0.132) | 0.169 (0.105) | |||
| Appointed by less than 3 years at current school (dummy) | 1.593 (1.267) | 1.19 (1.015) | |||
| Constant | 23.785*** (4.680) | 29.497*** (3.781) | 17.971*** (5.819) | 28.066*** (4.536) | |
| R2 | 34% | 42% | 34% | 42% | |
Note(s): Robust standard errors in parenthesis clustered at school level. Number of schools: 586. The baseline model includes student and school-level characteristics referring to the school context and management practices. The full model includes also principal's characteristics. ***p < 0.01, **p < 0.05, *p < 0.1
Notes
The main problem of missing data relates to the lack of data about the students' prior achievement (at grade 6). Every student has an unequivocal code which links to all the relevant data kept about them and which is transmitted to INVALSI by the school and provides the means to trace a student through the years. This procedure was introduced in 2010, but before then, the students' codes were not collected systematically, meaning that there is a high rate of lost information for the initial years (in practice, 47 per cent of the sample was lost in this first step). Moreover, some of the school principals did not answer the questionnaire, and this brought down the size of the sample with complete information to the final data set described in this paper.
An important methodological issue must be discussed here. The WMS does not use self-reported measures of management practices. Its method is to use telephone interviews which are rated by independent researchers. This means that we cannot assume that our measures (derived from a questionnaire with self-reported answers) can be compared with the validity of the WMS. We instead want to follow the theoretical WMS framework and survey the major areas of management practices. Other studies that make use of self-reported data on the management practices adopted by school principals make the effort to classify the various indicators into “areas”. For example, Grissom and Loeb (2011) defined the following tasks: instruction management, internal relations, organization management, administration and external relations. As can be easily acknowledged, these categories mirror the general view about which management practices that can be adopted by principals as part of their duties, so confirming the coherence of our empirical approach.
For example, Di Liberto et al. (2015) confirm this tendency in Italian school principals by comparing distributions of answers using OECD data from the Programme for International Student Assessment PISA (self-reported) with the World Management Survey's answers (externally assessed).
A methodological issue is worth exploring here. In the literature on “school effects”, an important stream deals with the choice of using multilevel (hierarchical) modelling instead of traditional OLS for statistical estimations (see McCaffrey et al., 2004). Multilevel modelling is particularly suited to considering the hierarchical structure of data, especially when individuals are nested within more than one group (students nested within classes, classes nested within schools). In principle, we can also use this approach in the present paper, although our data refer only to one aggregation (i.e. students within schools; we do not have information about the classes). At the same time, the literature that uses OLS indicates that deriving robust standard errors, clustered at the unit of aggregation (in our case, the schools), produces the effect of considering the correlation within groups, while maintaining the other desirable characteristics of OLS estimator (Moulton, 1990). Therefore, as a further test, we have also reported the results from a multilevel model (see the Technical Annex) – and, to anticipate these findings, the results are qualitatively and quantitatively very similar.
As already mentioned in previous sections, a limitation to our empirical model is that we lack information about teacher quality and/or administrative staff that should be included in the vector
A further methodological aspect deserves attention here, namely that we are measuring the principals' management practices and the students' test score for the same year. Ideally, management practices should be measured at time t and correlated with students' scores at a time t+n, allowing for a time lag between the action (practices) and results (scores). Unfortunately, we do not have any longitudinal data on the principals' survey. We must assume that, when the principals answer the survey, they indicate their usual (i.e. time-invariant) management styles, attitudes and practices and not their actions in a given point in time. This appears a reasonable assumption, therefore the results must be interpreted in this light.
Supplementary material is available at: http://www.w3.org/1999/xlink
References
Agasisti, T. and Falzetti, P. (2017), “Between-classes sorting within schools and test scores: an empirical analysis of Italian junior secondary schools”, International Review of Economics, Vol. 64 No. 1, pp. 1-45.
Agasisti, T. and Vittadini, G. (2012), “Regional economic disparities as determinants of students' achievement in Italy”, Research in Applied Economics, Vol. 4 No. 1, pp. 33-54.
Bloom, N., Lemos, R., Sadun, R. and Van Reenen, J. (2015), “Does management matter in schools?”, The Economic Journal, Vol. 125 No. 584, pp. 647-674.
Branch, G.F., Hanushek, E.A. and Rivkin, S.G. (2012), Estimating the Effect of Leaders on Public Sector Productivity: The Case of School Principals. NBER Working Paper No. 17803, National Bureau of Economic Research, Cambridge, MA.
Brunello, G. and Checchi, D. (2005), “School quality and family background in Italy”, Economics of Education Review, Vol. 24 No. 5, pp. 563-577.
Bush, T. and Glover, D. (2002), School Leadership: Concepts and Evidence, National College for School Leadership, Nottingham.
Coelli, M. and Green, D.A. (2012), “Leadership effects: school principals and student outcomes”, Economics of Education Review, Vol. 31 No. 1, pp. 92-109.
Dhuey, E. and Smith, J. (2014), “How school principals influence student learning”, Empirical Economics, pp. 1-32.
Di Liberto, A., Schivardi, F. and Sulis, G. (2015), “Managerial practices and student performance”, Economic Policy, Vol. 30 No. 84, pp. 683-728.
Grissom, J.A. and Loeb, S. (2011), “Triangulating principal effectiveness: how perspectives of parents, teachers, and assistant principals identify the central importance of managerial skills”, American Educational Research Journal, Vol. 48 No. 5, pp. 1091-1123.
Grissom, J.A., Kalogrides, D. and Loeb, S. (2015), “Using student test scores to measure principal performance”, Educational Evaluation and Policy Analysis, Vol. 20 No. 10, pp. 1-26.
Hanushek, E.A. and Rivkin, S.G. (2006), “Teacher quality”, Handbook of the Economics of Education, Vol. 2, pp. 1051-1078.
Hanushek, E.A. and Rivkin, S.G. (2010), “Generalizations about using value-added measures of teacher quality”, The American Economic Review, Vol. 100 No. 2, pp. 267-271.
Hanushek, E.A. and Woessmann, L. (2011), “The economics of International differences in educational achievement”, in Hanushek, E.A., Machin, S. and Woessmann, L. (Eds), Handbooks in Economics, North-Holland, The Netherlands, Vol. 3, pp. 89-200.
Harris, A. and Jones, M. (2017), “Middle leaders matter: reflections, recognition, and renaissance”, School Leadership & Management, Vol. 37 No. 3, pp. 213-216.
Leithwood, K. and Jantzi, D. (1999), “Transformational school leadership effects: a replication”, School Effectiveness and School Improvement, Vol. 10, pp. 451-479.
Leithwood, K., Louis, K.S., Anderson, G. and Wahlstrom, K. (2004), How Leadership Influences Student Learning: A Review of Research for the Learning from Leadership Project, The Wallace Foundation, New York.
McCaffrey, D.F., Lockwood, J.R., Koretz, D., Louis, T.A. and Hamilton, L. (2004), “Models for value-added modeling of teacher effects”, Journal of Educational and Behavioral Statistics, Vol. 29 No. 1, pp. 67-101.
Moulton, B.R. (1990), “An illustration of a pitfall in estimating the effects of aggregate variables on micro unit”, The Review of Economics and Statistics, Vol. 72 No. 2, pp. 334-338.
Podsakoff, P.M., MacKenzie, S.B. and Lee, J.Y. (2003), “Common method biases in behavioral research: a critical review of the literature and recommended remedies”, Journal of Applied Psychology, Vol. 88 No. 5, pp. 879-903.
Quinn, D.M. (2002), “The impact of principal leadership behaviors on instructional practice and student engagement”, Journal of Educational Administration, Vol. 40 No. 5, pp. 447-467.
Robinson, V.M.J., Lloyd, C.A. and Rowe, K.J. (2008), “The impact of leadership on student outcomes: an anlysis of the differential effect of leadership types”, Educational Administration Quarterly, Vol. 44 No. 5, pp. 635-674.
Sulis, I. and Porcu, M. (2015), “Assessing divergences in Mathematics and Reading achievement in Italian primary schools: a proposal of adjusted indicators of school effectiveness”, Social Indicators Research, Vol. 122 No. 2, pp. 607-634.
Todd, P.E. and Wolpin, K.I. (2007), “The production of cognitive achievement in children: home, school and racial test score gaps”, The Journal of Human Capital, Vol. 1 No. 1, pp. 91-136.
Waters, T., Marzano, R.J. and McNulty, B. (2003), Balanced Leadership: What 30 Years of Research Tells Us about the Effect of Leadership on Student Achievement, Aurora, CO: Mid-continent Research for Education and Learning.