Sir

Methods for assessing the quality of research are increasingly based on impact factors and citation analysis. But there is concern about the accuracy of science citation1,2,3,4,5,6,7, and suspicion has been cast on the impartiality of citation analysis2,3,4,.

As well as conscious bias5,8, it has been suggested that there is a tendency to make more citation errors in names from unfamiliar languages4. We have examined the tendency to make errors when typing strings of familiar and unfamiliar names. These experiments, together with an analysis of the numbers of authors per paper in different journals, show why there is a positive association between journal quality (impact factor) and rate of apparent undercitation4.

To see whether language familiarity and the length of name strings influenced typing accuracy, we asked 36 undergraduate zoology students to type name strings that consisted of one, three or six Finnish (unfamiliar) or English (familiar) names that were projected on a screen for 8, 24 or 48 seconds, respectively. All subjects were native English speakers with no knowledge of Finnish. The error rate in unfamiliar names increased significantly faster with the length of the name string than the error rate in familiar names (Fig. 1).

Figure 1: Number of errors (mean ± 1 s.e.m.) in short, intermediate and long name strings for Finnish and English names.
figure 1

Errors are ln-transformed. There is a significant interaction between language and length of the name string on error rate (repeated measures ANOVA F 1.50,52.33 = 6.44, P = 0.007). (To correct for a violation of the sphericity assumption, the degrees of freedom are Greenhouse-Geisser adjusted.) Short name string was one name totalling 5 to 8 characters; intermediate string, three names totalling 15 to 24 characters; and long string, six names totalling 30 to 48 characters. There was no difference in the total number of characters between Finnish and English name strings (ANOVA F 1,54 = 0.00, P = 0.953). Finnish names with non-English characters (Ä and Ö) were not used. All subjects received all combinations 20 times and each name was used only once.

Full size image

We then calculated the mean number of authors per article for each of 65 journals, and plotted this number against the journal's impact factor. We found a strong positive correlation (Fig. 2). Our experimental results (Fig. 1) show that typing errors occur at a greater rate in long strings of non-English names. Therefore, the positive correlation between journal impact factor and number of authors (Fig. 2) will inflate the effect of mis-citation, and contribute to the positive association between journal impact factor and rate of apparent undercitation4.

Figure 2: Correlation between a journal's impact factor and the mean number of authors per paper.
figure 2

Pearson's r = 0.706, n = 65, P < 0.001. Journals are a random sample of biological/ ecological/evolutionary journals from the Biological Sciences Library of the University of Western Australia. The first 30 papers in 1998 in each journal were analysed. When there were fewer than 30 papers during that year, additional papers from 1997 were included. Impact factors are from the 1997 Science Citation Index.

Full size image

Our results confirm suspicions that citation analysis is biased. Mis-citation of unfamiliar names is serious, because almost half of all scientific publications come from English-speaking countries1, so non-English-speaking countries will suffer more mis-citations. Our results suggest that undercitation3,4 could be an artefact of mis-citation rather than true undercitation, and so cast doubt on the objectivity of citation analysis.