Discussion paper
Out for the count: some methodological questions in ‘publications counting’ literature

https://doi.org/10.1016/S0278-4319(01)00007-XGet rights and content

Abstract

This paper considers ‘publications counting’ literature, that literature which seeks to establish league tables of the published output of academics. Often justified on the grounds of benchmarking for supporting tenure and appointment decisions, publications counting literature is revealed to be methodologically flawed. This paper focuses on the major limitations of this kind of study, presenting a conceptual critique of existing ‘publications counting’ processes, and argues that the methods of such studies are neither reasonable nor consistent. This is particularly clear in areas such as the choice of journals to be sampled, the time frame established in sampling procedures, and various arithmetic procedures employed in calculating output measures. The paper concludes with discussion of possible alternative means of achieving goals of benchmarking hospitality scholarship while suggesting that publications counting should be abandoned as a sole means of determining research excellence.

Introduction

Measurement of the publications output of academics as a means of eliciting some assessment of individual and institutional productivity and equality is not a new phenomenon, although in the hospitality and tourism context it is thus far principally a US one. A number of studies focusing on the tourism and hospitality areas have appeared in the last dozen years or so (Weaver and McCleary, 1989; Weaver et al., 1990; Rutherford and Samenfink, 1992; Samenfink and Rutherford, 1996). Related studies have been undertaken on faculty selection and review criteria (Sheldon and Collinson, 1990; Schmidgall and Woods, 1992); trends in the authorship and content of published articles in hospitality (Chon et al., 1989; Roberts, 1998); the rating of different kinds of publications output (Schmidgall and Woods, 1993); the rating of journals in the hospitality and tourism fields (Ferreira et al (1994), Ferreira et al (1998)); and perceptions of the value of single, co-authored and multiple authored articles in hospitality (Woods, 1998). The ‘publications counting’ literature has been further extended by citation analysis of published hospitality research (Woods and Schmidgall, 1995; Howey et al., 1999).

Although, ostensibly, the purpose of the research studies described above has been to investigate the most productive and influential scholars in the fields concerned, precise research objectives are noticeable by their absence although there are passing references to the benefits of benchmarking (Rutherford and Samenfink, 1992) and to the usefulness of such research in terms of “tenure, promotion, and salary issues...recruiting purposes and for promotional reasons…for placement or relocation purposes” (Woods and Schmidgall, 1995, p. 33). Perhaps more remarkable has been the lack of apparent critique of the methodologies of these studies or their general tenor. In an early and still important paper Smith (1983), by the way of outlining approaches to the production of academic articles, charted much of the political context and terrain attendant on such activity. More recently, Seaton (1996) has commented upon the claustrophobic nature of the refereeing process in tourism journals and Wood (1995) has examined some of the conceptual and methodological problems raised by the hospitality ‘publications counting’ literature. These contributions, while valuable, are characterised by brevity and leave considerable scope for further reflection on a range of issues, issues that this paper seeks to address.

The rationale for what follows does not simply emanate from the nature of the studies concerned. In many countries higher education is subject to increasing public and political scrutiny in the name of fiscal accountability and cost efficiency. The workings of the contemporary university in particular are constantly subjected to new tests of social, business and community ‘relevance’. In America, where publication has always had a central role in individual and institutional politics in higher education, these and additional pressures are reflected in semi-public critical discourse on the intellectual practices of higher seats of learning (Sykes, 1988; Hughes, 1994; Paglia (1993), Paglia (1994)). Fears about the quality of academic research recur, as is most recently seen in the controversy over Alan Sokal's duping of a social science journal, where the latter published a ‘phoney’ article by the physicist purporting to deconstruct theories of quantum gravity (Sokal and Bricmont, 1998). These and other pressures have led to a measure of defensiveness on the part of the higher education sector. More important, perhaps, are the mechanisms that some governments have introduced to place checks on quality. In the last 10–15 years in the UK both teaching and research have come to be periodically peer-assessed, the latter every five years in a ‘Research Assessment Exercise’ (RAE) with resultant ratings in subject areas being widely published (a similar exercise is undertaken in Hong Kong). The circulation of such information and the competitiveness it creates means that measures such as publications counting and citation analysis, when they come from academic scholars, are particularly important as they might exert influence on external assessments. Thus, the main objective of this paper is to offer a conceptual critique of existing studies of ‘publications counting’ in the hospitality field.

Section snippets

Conceptual critique of ‘publications counting’

Within the hospitality field, publications counting research is, as already intimated, an ethnocentric affair, the majority of published work emanating from the USA. Of the literature cited earlier, Rutherford and Samenfink (1992) remains the seminal template for the field, a point noted by Wood (1995) who illustrates his own discussion with examples from this study. A similar approach is undertaken here, but examples are drawn from a slightly larger number of sources. The main areas for

Some ways forward?

This paper has sought to unravel by means of a critique some of the assumptions underlying ‘publications counting’ literature in hospitality. On the basis of the foregoing discussion it would be tempting to suggest clear recommendations as to future methodological procedures for ‘publications counting’ research. Such a strategy would, however, be flawed. Much of the literature alluded to in this paper evidences, as the critique offered suggests, many methodological flaws and problems, some of

Conclusion

Somewhere between the two approaches described in the previous section lies, probably, a means by which research in hospitality can be benchmarked in a fashion that commands reasonable agreement. Some hope resides in the emergent consensus concerning the ‘best’ journals in the field. For the reasons described earlier, ‘bracketing’ the Cornell Hotel and Restaurant Administration Quarterly leaves Annals of Tourism Research, the Journal of Hospitality and Tourism Research, the International

References (25)

  • C. Paglia

    Sex, Art and American CultureEssays

    (1993)
  • C. Paglia

    Vamps and TrampsNew Essays

    (1994)
  • Cited by (22)

    View all citing articles on Scopus
    View full text