Skip to main content
SpringerLink
Log in

Does the h index for assessing single publications really work? A case study on papers published in chemistry

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Schubert (Scientometrics, 78:559–565, 2009) showed that “a Hirsch-type index can be used for assessing single highly cited publications by calculating the h index of the set of papers citing the work in question” (p. 559). To demonstrate that this single publication h index is a useful yardstick to compare the quality of different publications; the index should be strongly related to the assessment by peers. In a comprehensive research project we investigated the peer review process of the Angewandte Chemie International Edition. The data set contains manuscripts reviewed in the year 2000 and accepted by the journal or rejected but published elsewhere. Single publication h index values were calculated for a total of 1,814 manuscripts. The results show a correlation in the expected direction between peer assessments and single publication h index values: After publication, manuscripts with positive ratings by the journal’s reviewers show on average higher h index values than manuscripts with negative ratings by reviewers (and later published elsewhere). However, our findings do not support Schubert’s (2009) assumption that the additional dimension of indirect citation influence contributes to a more refined picture of the most cited papers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45, 199–245.

    Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2007). What do we know about the h index? Journal of the American Society for Information Science and Technology, 58(9), 1381–1385. doi:10.1002/asi.20609.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2008a). The effectiveness of the peer review process: Inter-referee agreement and predictive validity of manuscript refereeing at Angewandte Chemie. Angewandte Chemie International Edition, 47(38), 7173–7178. doi:10.1002/anie.200800513.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2008b). Selecting manuscripts for a high impact journal through peer review: A citation analysis of Communications that were accepted by Angewandte Chemie International Edition, or rejected but published elsewhere. Journal of the American Society for Information Science and Technology, 59(11), 1841–1852. doi:10.1002/asi.20901.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2009a). Extent of type I and type II errors in editorial decisions: A case study on Angewandte Chemie International Edition. Journal of Informetrics, 3(4), 348–352.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2009b). The luck of the referee draw: The effect of exchanging reviews. Learned Publishing, 22(2), 117–125. doi:10.1087/2009207.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2009c). The state of h index research. Is the h index the ideal way to measure research performance? EMBO Reports, 10(1), 2–6. doi:10.1038/embor.2008.233.

    Article  Google Scholar 

  • Bornmann, L., & Daniel, H.-D. (2010). The manuscript reviewing process: Empirical research on review requests, review sequences and decision rules in peer review. Library and Information Science Research, 32(1), 5–12. doi:10.1016/j.lisr.2009.07.010.

    Article  Google Scholar 

  • Bornmann, L., Marx, W., Schier, H., Rahm, E., Thor, A., & Daniel, H. D. (2009). Convergent validity of bibliometric Google Scholar data in the field of chemistry. Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published elsewhere, using Google Scholar, Science Citation Index, Scopus, and Chemical Abstracts. Journal of Informetrics, 3(1), 27–35. doi:10.1016/j.joi.2008.11.001.

    Article  Google Scholar 

  • Bornmann, L., Mutz, R., & Daniel, H.-D. (2007). The b index as a measure of scientific excellence. A promising supplement to the h index. Cybermetrics, 11(1), paper 6.

    Google Scholar 

  • Bornmann, L., Mutz, R., Marx, W., Schier, H., & Daniel, H.-D. (2011). A multilevel modelling approach to investigating the predictive validity of editorial decisions: Do the editors of a high-profile journal select manuscripts that are highly cited after publication? Journal of the Royal Statistical Society–Series A (Statistics in Society), 174(4). doi:10.1111/j.1467-985X.2011.00689.x.

  • Bornmann, L., Weymuth, C., & Daniel, H.-D. (2010). A content analysis of referees’ comments: How do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ? Scientometrics, 83(2), 493–506.

    Article  Google Scholar 

  • Cole, S. (1989). Citations and the evaluation of individual scientists. Trends in Biochemical Sciences, 14(1), 9–13.

    Article  Google Scholar 

  • Franceschini, F., & Maisano, D. A. (2010). Analysis of the Hirsch index’s operational properties. European Journal of Operational Research, 203(2), 494–504. doi:10.1016/j.ejor.2009.08.001.

    Article  MATH  Google Scholar 

  • Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572. doi:10.1073/pnas.0507655102.

    Article  Google Scholar 

  • Schubert, A. (2009). Using the h-index for assessing single publications. Scientometrics, 78(3), 559–565.

    Article  Google Scholar 

  • Sheskin, D. (2007). Handbook of parametric and nonparametric statistical procedures (4th ed.). Boca Raton: Chapman & Hall/CRC.

    MATH  Google Scholar 

  • StataCorp. (2009). Stata Statistical Software: Release 11. College Station: Stata Corporation.

    Google Scholar 

  • Thorngate, W., Dawes, R. M., & Foddy, M. (2009). Judging merit. New York: Psychology Press.

    Google Scholar 

  • van Eck, N. J., & Waltman, L. (2008). Generalizing the h- and g-indices. Journal of Informetrics, 2(4), 263–271.

    Article  Google Scholar 

Download references

Acknowledgments

The entire research study, which is also investigating quality assurance of open access journals, is supported by a grant from the Max Planck Society. The authors would like to thank Dr. Christophe Weymuth (formerly at the Organic Chemistry Institute of the University of Zurich and now at BIOSYNTH AG, Switzerland) for investigation of the manuscripts rejected by Angewandte Chemie International Edition and published elsewhere.

We thank Dr. Peter Gölitz, Editor-in-Chief of Angewandte Chemie, the Editorial Board of Angewandte Chemie, and the German Chemical Society (GDCh, Frankfurt am Main, Germany) for permission to conduct the evaluation of the selection process of the journal, and are grateful to the members of the editorial office for their generous support during the carrying out of the study.

Author information

Authors and Affiliations

  1. Max Planck Society, Hofgartenstr 8, 80539, Munich, Germany

    Lutz Bornmann

  2. Max Planck Institute for Solid State Research, Heisenbergstraße 1, 70569, Stuttgart, Germany

    Hermann Schier & Werner Marx

  3. ETH Zurich, Professorship for Social Psychology and Research on Higher Education, Zähringerstr 24, 8092, Zurich, Switzerland

    Hans-Dieter Daniel

  4. University of Zurich, Evaluation Office, Mühlegasse 21, 8001, Zurich, Switzerland

    Hans-Dieter Daniel

Authors
  1. Lutz Bornmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hermann Schier
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Werner Marx
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hans-Dieter Daniel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lutz Bornmann.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bornmann, L., Schier, H., Marx, W. et al. Does the h index for assessing single publications really work? A case study on papers published in chemistry. Scientometrics 89, 835–843 (2011). https://doi.org/10.1007/s11192-011-0472-0

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-011-0472-0

Keywords

Search

Navigation