Skip to main content
SpringerLink
Log in

The generalized propensity score methodology for estimating unbiased journal impact factors

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

The journal impact factor (JIF) proposed by Garfield in the year 1955 is one of the most commonly used and prominent citation-based indicators of the performance and significance of a scientific journal. The JIF is simple, reasonable, clearly defined, and comparable over time and, what is more, can be easily calculated from data provided by Thomson Reuters, but at the expense of serious technical and methodological flaws. The paper discusses one of the core problems: The JIF is affected by bias factors (e.g., document type) that have nothing to do with the prestige or quality of a journal. For solving this problem, we suggest using the generalized propensity score methodology based on the Rubin Causal Model. Citation data for papers of all journals in the ISI subject category “Microscopy” (Journal Citation Report) are used to illustrate the proposal.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Allison, P. D. (1999). Logistic regression using SAS: Theory and application. Cary, NC: SAS Institute Inc.

    Google Scholar 

  • Bornmann, L., Mutz, R., Neuhaus, C., & Daniel, H.-D. (2008). Citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics in Science and Environmental Politics, 8, 93–102.

    Article  Google Scholar 

  • Braun, T., Glänzel, W., & Schubert, A. (1989). Some data on the distribution of journal publication types in the Science Citation Index Database. Scientometrics, 15, 325–330.

    Article  Google Scholar 

  • Cochran, W. G. (1968). The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics, 24(2), 295–313.

    Article  MathSciNet  Google Scholar 

  • Fan, X., Felsöváli, Á., Sivo, S. A., & Keenan, S. C. (2001). SAS for monte carlo studies: A guide for quantitative researchers. Cary, NC: SAS Institute Inc.

    Google Scholar 

  • Feng, P., Zhou, X.-H., Zou, Q.-M., Fan, M.-Y., & Li, X.-S. (2011). Generalized propensity score for estimating the average treatment effect of multiple treatments. Statistics in Medicine. doi: 10.1002/sim.4168 (published online February 24, 2011).

  • Garfield, E. (1955). Citation indexes to science: A new dimension in documentation through association of ideas. Science, 122, 108–111.

    Article  Google Scholar 

  • Garfield, E. (1999). Journal impact factor: A brief review. Journal of the Canadian Medical Association, 161(8), 979–980.

    Google Scholar 

  • Garfield, E. (2006). The history and meaning of the Journal Impact Factor. Journal of the American Medical Association, 295(1), 90–93.

    Article  Google Scholar 

  • Glänzel, W., & Moed, H. (2002). Journal impact measures in bibliometric research. Scientometrics, 53(2), 171–193.

    Article  Google Scholar 

  • Guo, S., & Fraser, M. W. (2010). Propensity score analysis—statistical methods and applications. London, UK: Sage.

    Google Scholar 

  • Hirano, K., & Imbens, G. (2004). The propensity score with continuous treatments. In A. Gelman & X.-L. Meng (Eds.), Applied Bayesian modeling and causal inference from incomplete-data perspective (pp. 73–84). London: Wiley.

    Google Scholar 

  • Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945–970.

    MathSciNet  MATH  Google Scholar 

  • Imai, K., & van Dyk, D. A. (2004). Causal inference with general treatment regimes: Generalizing the propensity score. Journal of the American Statistical Association, 99(467), 854–866.

    Article  MathSciNet  MATH  Google Scholar 

  • Imbens, G. (2000). The role of propensity score in estimating dose-response functions. Biometrika, 87(3), 706–710.

    Article  MathSciNet  MATH  Google Scholar 

  • Kluve, J., Schneider, H., Uhlendorff, A., & Zhao, Z. (2012). Evaluating continuous training programmes by using the generalized propensity score. Journal of the Royal Statistical Society A, 175(Part 2), 1–31.

    MathSciNet  Google Scholar 

  • Leydesdorff, L., & Bornmann, L. (2011). How fractional counting of citations affects the impact factor: Normalization in terms of differences in citation potentials among fields of science. Journal of the American Society for Information Science and Technology, 62(2), 217–229.

    Article  Google Scholar 

  • Lu, B., Greevey, R., Xu, X., & Beck, C. (2011). Optimal nonbipartite matching and its statistical applications. American Statistician, 65(1), 21–30.

    Article  MathSciNet  Google Scholar 

  • Moed, H. F., & van Leeuwen, T. N. (1995). Improving the accuracy of the Institute for Scientific Information’s Journal Impact Factor. Journal of the American Society of Information Science, 46, 461–467.

    Article  Google Scholar 

  • Moed, H. F., Van Leeuwen, T. N., & Reeduk, J. (1999). Towards appropriate indicators of journal impact. Scientometrics, 46(3), 575–589.

    Article  Google Scholar 

  • Mutz, R., & Daniel, H.-D. (2012, in press). Skewed citation distributions and bias factors: Solutions to two core problems with the journal impact factor. Journal of Infometrics.

  • Neuhaus, C., Marx, W., & Daniel, H.-D. (2009). The publication and citation impact profile of Angewandte Chemie and the Journal of the American Chemical Society based on the sections of Chemical Abstracts: A case study on the limitations of the Journal Impact Factor. Journal of the American Society for Information Science and Technology, 60(1), 176–183.

    Article  Google Scholar 

  • Rosenbaum, P. R. (2010). Design of observational studies. New York: Springer.

    Book  MATH  Google Scholar 

  • Rosenbaum, P. R., & Rubin, D. B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79(387), 516–524.

    Google Scholar 

  • Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66, 688–701.

    Article  Google Scholar 

  • Rubin, D. B. (1977). Assignment to treatment group on the basis of a covariate. Journal of Educational Statistics, 2(1), 1–26.

    Article  Google Scholar 

  • Rubin, D. B. (2004). Teaching statistical inference for causal effects in experiments and observational studies. Journal of Educational and Behavioral Statistics, 29(3), 343–367.

    Article  Google Scholar 

  • Rubin, D. B. (2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469), 322–331.

    Article  MathSciNet  MATH  Google Scholar 

  • Rubin, D. B. (2006). Matched sampling for causal effects. Cambridge, UK: Cambridge University Press.

    Book  MATH  Google Scholar 

  • Rubin, D. B. (2007). The design versus the analysis of observational studies for causal effects: Parallels with the design of randomized trials. Statistics in Medicine, 26, 20–36.

    Article  MathSciNet  Google Scholar 

  • Rubin, D. B., & Thomas, N. (1996). Matching using estimated propensity scores: Relating theory in practice. Biometrics, 52, 249–264.

    Article  MATH  Google Scholar 

  • SAS Institute Inc. (2009). SAS/STAT 9.2 user’s guide. Cary, NC: SAS Institute Inc.

    Google Scholar 

  • Spreeuwenberg, M. D., Bartak, A., Croon, M. A., Hagenaars, J. A., Bussbach, J. J. V., Andrea, H., et al. (2010). The multiple propensity score as control for bias in the comparison of more than two treatment arms. An introduction from a case study in mental health. Medical Care, 48(2), 166–174.

    Article  Google Scholar 

  • Todorov, R., & Glänzel, W. (1988). Journal citation measures: A concise review. Journal of Information Science, 14, 47–56.

    Article  Google Scholar 

  • Vanclay, J. K. (2012). Impact factor: Outdated artifact or stepping-stone to journal certification? Scientometrics (accepted paper).

  • Wang, J., Donnan, P. T., Steinke, D., & MacDonald, T. M. (2001). The multiple propensity score for analysis of dose-response relationships in drug safety studies. Pharmacoepidemiology and Drug Safety, 10, 105–111.

    Article  Google Scholar 

  • Zanutto, E., Lu, B., & Hornik, R. (2005). Using propensity score subclassification for multiple treatment doses to evaluate a national antidrug media campaign. Journal of Educational and Behavioral Statistics, 30(1), 59–73.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Social Psychology and Research on Higher Education, ETH Zurich, Muehlegasse 21, 8001, Zurich, Switzerland

    Rüdiger Mutz & Hans-Dieter Daniel

  2. Evaluation Office, University of Zurich, Muehlegasse 21, 8001, Zurich, Switzerland

    Hans-Dieter Daniel

Authors
  1. Rüdiger Mutz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hans-Dieter Daniel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rüdiger Mutz.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Mutz, R., Daniel, HD. The generalized propensity score methodology for estimating unbiased journal impact factors. Scientometrics 92, 377–390 (2012). https://doi.org/10.1007/s11192-012-0670-4

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-012-0670-4

Keywords

Search

Navigation