The dispersion of research performance within and between universities as a potential indicator of the competitive intensity in higher education systems

Abstract

Higher education systems in competitive environments generally present top universities, that are able to attract top scientists, top students and public and private financing, with notable socio-economic benefits in their region. The same does not hold true for non-competitive systems. In this study we will measure the dispersion of research performance within and between universities in the Italian university system, typically non-competitive. We will also investigate the level of correlation that occurs between performance in research and its dispersion in universities. The findings may represent a first benchmark for similar studies in other nations. Furthermore, they lead to policy indications, questioning the effectiveness of selective funding of universities based on national research assessment exercises. The field of observation is composed of all Italian universities active in the hard sciences. Research performance will be evaluated using a bibliometric approach, through publications indexed in the Web of Science between 2004 and 2008.

Introduction

In every field of human activity there are certain individuals who distinguish themselves by particular competencies, attitudes and interests, generally translating into outstanding levels of performance. Similarly, in every economic sector there are certain organizations, public or private, for-profit or not-for-profit, which consistently demonstrate top performance, in turn leading to a reputation which is likely to become a distinctive competence of the organization. These organizations generally excel at every link in the chain of value, thanks to quality of their personnel and the strategic and organizational capacity of their management. The fundamental key to their long-term success is a capacity to attract, develop and retain talent, meaning the best representatives of the work force in each function. Research organizations (universities and institutions), whether public or private, will not escape this rule as long as they operate in competitive environments. The level of domestic competition in higher education is determined by several contextual elements, from cultural practices to political legitimization of the system. One of the fundamental determinants of competitive intensity is undoubtedly the typology of funding and related incentives, at macro, meso and micro levels. In countries where government funding of universities represents a large share of total budget and is allocated with the intention of ensuring sufficiency of resources, competition among universities will likely be low in intensity. On the contrary, in nations where extra-governmental financing is significant and the public allocation is essentially based on merit, the environment is likely to be more competitive (Geuna & Martin, 2003). Over time, competition should lead to the development of distinctive competencies and to a subsequent competitive advantage of some organizations over all the others, meaning that in competitive environments it is possible to observe more marked performance differences among universities, and a mapping of the higher education industry into strategic groups.

Auranen and Nieminen (2010) classified the higher education systems of such countries as Germany, Sweden and Denmark (the present authors would also add Italy) as non-competitive. In these nations, the distinction of the different levels of excellence among universities is not immediate. On the contrary, in countries such as the United States, United Kingdom and Australia, which fall in the competitive category, few would have difficulty in identifying the best research universities. These universities are in competition with one another to bring in, from at home and abroad, the best researchers and teaching professors, the best technical-administrative personnel, best students, donations and public and private financing for research. Their reputations and competitive advantages are reflected as much in the salary levels and status that they succeed in providing as in the willingness of students to pay tuition fees that are well over average. Local and national governments have every interest in developing and cultivating champion institutions. In fact, the social and economic benefits for the nation and home region of a prestigious university have been soundly demonstrated (Fritsch and Slavtchev, 2006, Pressman et al., 1995, Rosenberg and Nelson, 1994). Governments should thus provide the conditions for the development of competitive environments, leading to continuous improvement in the entire higher education system, and the emergence of top universities.

It is no accident that, in recent years, an increasing number of nations have begun regular national exercises for research evaluation, permitting the allocation of funds on the basis of performance criteria and the stimulation of greater research productivity.

These exercises are generally based on the peer-review method, implying the evaluation of only a share of the entire research output of each university. Rankings for the universities are drawn up on the basis of the average quality of the research products as submitted by the institutions, and not on productivity. Performances of universities are compared at discipline level and not at individual level.

If the objective of government is to stimulate competition among universities, and so lead to the development of top universities that can compete internationally and produce the relevant socio-economic benefits, then one of the indicators to monitor over time, other than performance itself, is the degree of concentration of performance within universities. The less research performance is dispersed within universities then the greater is the probability of having top universities that are able to compete at the international level. This would indicate that the competitive environment has led to the formation of a system of universities with marked differences in performance, and thus the concentration of top scientists in top universities. In the American system of higher education, which appears to be among the most competitive, presenting a significant number of recognized universities able to attract resources and students from around the world, we expect that the average performance of researchers in these top universities would be very high, but that the variability, and thus the concentration, would be low. Universities of average level would be expected to show an average performance, perhaps with points of excellence in one or two disciplines, but again with a quite low concentration. To achieve balanced regional development, an additional policy objective could be that the top universities be distributed quite uniformly throughout the nation.

To date, the literature does not offer detailed analyses of the concentration of performance in university systems. In 1974, Allison and Stewart detected a highly skewed distribution of productivity (publications and citations) among 2172 scientists working in US academic departments. The authors pointed out the presence of accumulative advantage, so that “because of feedback through recognition and resources, highly productive scientists maintain or increase their productivity, while scientists who produce very little produce even less later on”. Daniel and Fisch (1990), in their review on the history of research performance evaluation in German universities, stressed the importance to take into account concentration of performance: in their opinion, when the department is the unit of analysis, the evaluator “should distinguish between those with ‘collective strength’ and those with ‘individual strength’ by computing some measure of concentration”. However, the paper provides no empirical evidence about concentration: the authors make only reference to one German leading psychology department and recall fundamental regularities of the distribution of scientific productivity, already pointed out in pioneering works of Lotka (1926) and De Solla Price (1971). The lack of studies on concentration of research performance is easily understandable: realizing such analysis requires the measurement of performance at the level of single scientists, which can be hardly achieved on large populations. Meanwhile, except for Abramo and D’Angelo (2011), the bibliometric exercises available in the literature have been limited to single institutions or disciplines (Costas et al., 2010, Gonçalves et al., 2009, Kalaitzidakis et al., 2003, Macri and Dipendra, 2006), and have never entered into the comparative analysis of the degree of concentration of performance in the institutions examined.

What we propose for this work is exactly this measurement of the degree of concentration of performance, in the hard sciences for Italian universities. We will subsequently compare the variability of performance within universities to that between universities. Finally, we will examine the level of correlation between degree of concentration and average level of performance. Through the findings of our analyses, we aim at providing policy indications on the risks of assigning public funds to universities on the basis of the outcomes of national research assessment exercises. Furthermore, we hope to stimulate similar analyses in other national contexts in order to demonstrate the assumption that non-competitive higher education systems differ from competitive ones in terms of concentration of performance within and between universities.

The next section of the paper presents a summary description of the Italian university system. Section 3 describes the methodology adopted, the dataset and indicators used. Section 4 presents the results, with discussion. The last section presents the pertinent conclusions from the work, with the authors’ comments.