Skip to main content
SpringerLink
Log in

Are bibliometric measures consistent with scientists’ perceptions? The case of interdisciplinarity in research

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Interdisciplinary research (IDR) has become an important component in the conduction of leading-edge science and innovation. From the different approaches available to measuring IDR, bibliometric indicators have experienced the greatest growth. Despite the frequent use of bibliometric measures of IDR in research and policymaking, their adequacy has not been validated against scientists’ perceptions. Using the case of an IDR-oriented research institute in Japan, this study aims to investigate the differences and similarities between the outcomes of common bibliometric measures of IDR and the scientists’ perceptions of IDR. We used a unique dataset combining bibliometric measures with survey data collected from the scientists’ self-assessment of their research. This study also investigates the factors influencing the outcomes of bibliometrics and scientists' perceptions. Moreover, this study explores how IDR qualitative and quantitative measures differ from those that are more intuitive, such as scientific impact. It was observed that there is no “holy grail” measure for interdisciplinarity when compared with scientific impact, for which the impact factor is considered as a key metric by scientists. While bibliometric measures of interdisciplinarity show mild correlations with scientists' perceptions, they display high discriminatory power. The disagreement between qualitative and quantitative evaluations, as well as the significant field-specific nature of interdisciplinarity, calls for the use of multidimensional assessment approaches for assessing IDR, and the building of a consensus about the meaning and measurement of interdisciplinarity among scientists, respectively. The results of this study provide a series of guidelines for a more effective implementation of interdisciplinarity-oriented R&D policies at different organizational levels.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Abramo, G., D’Angelo, C. A., & di Costa, F. (2017). Do interdisciplinary research teams deliver higher gains to science? Scientometrics, 111, 317–336.

    Article  Google Scholar 

  • Abramo, G., D’Angelo, C. A., & Zhang, L. (2018). A comparison of two approaches for measuring interdisciplinary research output: The disciplinary diversity of authors vs the disciplinary diversity of the reference list. Journal Informetrics, 12, 1182–1193.

    Article  Google Scholar 

  • Abramo, G., D’Angelo, C. A., & Costa, F. (2012). Identifying interdisciplinarity through the disciplinary classification of coauthors of scientific publications. Journal American Society Information Science Technology, 63, 2206–2222.

    Article  Google Scholar 

  • Adams, J., Loach, T., & Szomszor, M. (2016). Interdisciplinary research: Methodologies for identification and assessment. Digital Research Reports, Digital Science, London.

  • Amir-Aslani, A., & Mangematin, V. (2010). The future of drug discovery and development: Shifting emphasis towards personalized medicine. Technological Forecasting Social Change, 77, 203–217.

    Article  Google Scholar 

  • Anzai, T., Kusama, R., Kodama, H., & Sengoku, S. (2012). Holistic observation and monitoring of the impact of interdisciplinary academic research projects: An empirical assessment in Japan. Technovation, 32, 345–357.

    Article  Google Scholar 

  • Ávila-Robinson, A., & Miyazaki, K. (2013). Dynamics of scientific knowledge bases as proxies for discerning technological emergence—The case of MEMS/NEMS technologies. Technological Forecasting Social Change, 80, 1071–1084.

    Article  Google Scholar 

  • Avila-Robinson, A., & Sengoku, S. (2017). Multilevel exploration of the realities of interdisciplinary research centers for the management of knowledge integration. Technovation, 62, 22–41.

  • Callon, M., Courtial, J. P., Turner, W. A., & Bauin, S. (1983). From translations to problematic networks: An introduction to co-word analysis. Social Science Information, 22, 191–235.

    Article  Google Scholar 

  • Claudel, M., Massaro, E., Santi, P., Murray, F., & Ratti, C. (2017). An exploration of collaborative scientific production at MIT through spatial organization and institutional affiliation. PLoS ONE, 12, e0179334.

    Article  Google Scholar 

  • Dai, L., & Boos, M. (2017). How much sharing is enough? cognitive patterns in building interdisciplinary collaborations. In X. Fu, J. Luo, & M. Boos (Eds). Social Network Analysis: Interdisciplinary Approaches and Case Studies (pp. 41–70). NW: CRC Press.

  • Hessels, R. S., & Kingstone, A. (2019). Fake collaborations: Interdisciplinary science can undermine research integrity, June 29, 2019. https://doi.org/10.31234/osf.io/rqwea.

  • Huutoniemi, K., Klein, J. T., Bruun, H., & Hukkinen, J. (2010). Analyzing interdisciplinarity: Typology and indicators. Research Policy, 39, 79–88.

    Article  Google Scholar 

  • Jensen, P., & Lutkouskaya, K. (2014). The many dimensions of laboratories’ interdisciplinarity. Scientometrics, 98, 619–631.

    Article  Google Scholar 

  • Katoh, S., Lauto, G., Anzai, T., & Sengoku, S. (2018). Identification of factors to promote interdisciplinary research: A trial at COINS. In Proceedings of the 2018 Portland International Conference on Management of Engineering and Technology (PICMET), Honolulu, HI, USA, 19–23 August 2018, pp. 1–11.

  • Kessler, M. M. (1963). Bibliographic coupling between scientific papers. Journal Association Information Science Technology, 14, 10–25.

    Google Scholar 

  • Klavans, R., & Boyack, K. W. (2017). Which type of citation analysis generates the most accurate taxonomy of scientific and technical knowledge? Journal Association Information Science Technology, 68, 984–998.

    Article  Google Scholar 

  • Kueffer, C., Underwood, E., Hadorn, G. H., Holderegger, R., Lehning, M., Pohl, C., Schirmer, M., Schwarzenbach, R., Stauffacher, M., & Wuelser, G. (2012). Enabling effective problem-oriented research for sustainable development. Ecology Society, 17(4), 8.

  • Larivière, V., & Gingras, Y. (2010). On the relationship between interdisciplinarity and scientific impact. Journal American Society Information Science Technology, 61, 126–131.

    Article  Google Scholar 

  • Lauto, G., & Sengoku, S. (2015). Perceived incentives to transdisciplinarity in a Japanese university research center. Futures, 65, 136–149.

    Article  Google Scholar 

  • Leinster, T., & Cobbold, C. A. (2012). Measuring diversity: the importance of species similarity. Ecology, 93, 477–489.

    Article  Google Scholar 

  • Levitt, J., & Thelwall, M. (2008). The most highly cited Library and Information Science articles: Interdisciplinarity, first authors and citation patterns. Scientometrics, 78, 45–67.

    Article  Google Scholar 

  • Leydesdorff, L. (2007). Betweenness centrality as an indicator of the interdisciplinarity of scientific journals. Journal American Society Information Science Technology, 58, 1303–1319.

    Article  Google Scholar 

  • Leydesdorff, L., & Rafols, I. (2011). Indicators of the interdisciplinarity of journals: Diversity, centrality, and citations. Journal Informetrics, 5, 87–100.

    Article  Google Scholar 

  • Leydesdorff, L., Wagner, C. S., & Bornmann, L. (2018). Betweenness and diversity in journal citation networks as measures of interdisciplinarity—A tribute to Eugene Garfield. Scientometrics, 114, 567–592.

    Article  Google Scholar 

  • Leydesdorff, L., Wagner, C. S., & Bornmann, L. (2019a). Diversity measurement: Steps towards the measurement of interdisciplinarity? Journal Informetrics, 13(3), 904–905.

    Article  Google Scholar 

  • Leydesdorff, L., Wagner, C. S., & Bornmann, L. (2019b). Interdisciplinarity as diversity in citation patterns among journals: Rao-Stirling diversity, relative variety, and the Gini coefficient. Journal Informetrics, 13, 255–269.

    Article  Google Scholar 

  • Littmann, M., Selig, K., Cohen-Lavi, L., Frank, Y., Hönigschmid, P., Kataka, E., & Rost, B. (2020). Validity of machine learning in biology and medicine increased through collaborations across fields of expertise. Nature Machine Intelligence, 2(1), 18–24.

    Article  Google Scholar 

  • Martín-Martín, A., Orduna-Malea, E., Thelwall, M., & López-Cózar, E. D. (2018). Google scholar, web of science, and scopus: A systematic comparison of citations in 252 subject categories. Journal Informetrics, 12, 1160–1177.

    Article  Google Scholar 

  • MIT (2016) Convergence: The future of health. Massachusetts Institute of Technology. Available online at: http://www.convergencerevolution.net/2016-report/ (accessed June 30, 2016).

  • Moed, H. F. (2006). Citation analysis in research evaluation. Springer.

    Google Scholar 

  • Molas-Gallart, J., Rafols, I., & Tang, P. (2014). On the Relationship between Interdisciplinarity and Impact: Different modalities of interdisciplinarity lead to different types of impact. Journal Science Policy Research Management, 29, 69–89.

    Google Scholar 

  • Morillo, F., Bordons, M., & Gómez, I. (2001). An approach to interdisciplinarity through bibliometric indicators. Scientometrics, 51, 203–222.

    Article  Google Scholar 

  • Mugabushaka, A.-M., Kyriakou, A., & Papazoglou, T. (2016). Bibliometric indicators of interdisciplinarity: The potential of the Leinster-Cobbold diversity indices to study disciplinary diversity. Scientometrics, 107, 593–607.

    Article  Google Scholar 

  • Porter, A. L., Cohen, A. S., Roessner, J. D., & Perreault, M. (2007). Measuring researcher interdisciplinarity. Scientometrics, 72, 117–147.

    Article  Google Scholar 

  • Porter, A., & Rafols, I. (2009). Is science becoming more interdisciplinary? Measuring and mapping six research fields over time. Scientometrics, 81, 719–745.

    Article  Google Scholar 

  • Porter, A. L., Roessner, D. J., & Heberger, A. E. (2008). How interdisciplinary is a given body of research? Research Evaluation, 17, 273–282.

    Article  Google Scholar 

  • Rafols, I. (2014). Knowledge Integration and Diffusion: Measures and Mapping of Diversity and Coherence. In Y. Ding, R. Rousseau, & D. Wolfram (Eds.), Measuring Scholarly Impact: Methods and Practice.Switzerland: Springer.

    Google Scholar 

  • Rafols, I., Leydesdorff, L., O’Hare, A., Nightingale, P., & Stirling, A. (2012). How journal rankings can suppress interdisciplinary research: A comparison between innovation studies and business & management. Research Policy, 41, 1262–1282.

    Article  Google Scholar 

  • Rafols, I., & Meyer, M. (2010). Diversity and network coherence as indicators of interdisciplinarity: Case studies in bionanoscience. Scientometrics, 82, 263–287.

    Article  Google Scholar 

  • Rousseau, R. (2019). On the Leydesdorff-Wagner-Bornmann proposal for diversity measurement. Journal Informetrics, 13(3), 906.

    Article  Google Scholar 

  • Ruiz-Castillo, J., & Waltman, L. (2015). Field-normalized citation impact indicators using algorithmically constructed classification systems of science. Journal Informetrics, 9, 102–117.

    Article  Google Scholar 

  • Sanz MenéndezBordonsZulueta, L. M. M. (2001). Interdisciplinarity as a multidimensional concept: its measure in three different research areas. Research Evaluation, 10, 47–58.

    Article  Google Scholar 

  • Schmidt, J. C. (2008). Towards a philosophy of interdisciplinarity. Poiesis Praxis, 5, 53–69.

    Article  Google Scholar 

  • Schummer, J. (2004). Multidisciplinarity, interdisciplinarity, and patterns of research collaboration in nanoscience and nanotechnology. Scientometrics, 59, 425–465.

    Article  Google Scholar 

  • Siedlok, F., & Hibbert, P. (2014). The organization of interdisciplinary research: modes, drivers and barriers. International Journal Management Reviews, 16, 194–210.

    Article  Google Scholar 

  • Silva, F. N., Rodrigues, F. A., & Oliveira, O. N. (2013). Quantifying the interdisciplinarity of scientific journals and fields. Journal Informetrics, 7, 469–477.

    Article  Google Scholar 

  • Soós, S., & Kampis, G. (2011). Towards a typology of research performance diversity: the case of top Hungarian players. Scientometrics, 87, 357–371.

    Article  Google Scholar 

  • Stirling, A. (2007). A general framework for analysing diversity in science, technology and society. Journal Royal Society Interface, 4, 707–719.

    Article  Google Scholar 

  • van Eck, N. J., & Waltman, L. (2011). Text mining and visualization using VOSviewer. ISSI Newsletter, 7, 50–54.

    Google Scholar 

  • Wagner, C. S., Roessner, J. D., Bobb, K., Klein, J. T., Boyack, K. W., Keyton, J., Rafols, I., & Börner, K. (2011). Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature. Journal Informetrics, 5, 14–26.

    Article  Google Scholar 

  • Waltman, L., & van Eck, N. J. (2012). A new methodology for constructing a publication-level classification system of science. Journal American Society Information Science Technology, 63, 2378–2392.

    Article  Google Scholar 

  • Wang, J., Thijs, B., & Glänzel, W. (2015). Interdisciplinarity and impact: Distinct effects of variety, balance, and disparity. PLoS ONE, 10, e0127298.

    Article  Google Scholar 

  • Wang Q, Schneider JW (2019) Consistency and validity of interdisciplinarity measures. Quantitative Science Studies, 1, 239–263.

  • Whitesides, G. (2010). Solving problems. Lab on a Chip, 10, 2317–2318.

    Article  Google Scholar 

  • Yegros, A., D'este Cukierman, P., & Rafols, I. (2015). Does interdisciplinary research lead to higher citation impact? the different effect of proximal and distal interdisciplinarity. PLoS One, 10(8), e0135095. https://doi.org/10.1371/journal.pone.0135095.

    Article  Google Scholar 

  • Zhang, L., Rousseau, R., & Glänzel, W. (2016). Diversity of references as an indicator of the interdisciplinarity of journals: Taking similarity between subject fields into account. Journal Association for Information Science Technology, 67, 1257–1265.

    Article  Google Scholar 

  • Zhang, L., Sun, B., Chinchilla-Rodríguez, Z., Chen, L., & Huang, Y. (2018). Interdisciplinarity and collaboration: On the relationship between disciplinary diversity in departmental affiliations and reference lists. Scientometrics, 117, 271–291.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. EGADE Business School, Tecnologico de Monterrey, 01389, Mexico City, Mexico

    Alfonso Ávila-Robinson

  2. School of Environment and Society, Tokyo Institute of Technology, Tokyo, 108-0023, Japan

    Cristian Mejia & Shintaro Sengoku

Authors
  1. Alfonso Ávila-Robinson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cristian Mejia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shintaro Sengoku
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alfonso Ávila-Robinson.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ávila-Robinson, A., Mejia, C. & Sengoku, S. Are bibliometric measures consistent with scientists’ perceptions? The case of interdisciplinarity in research. Scientometrics 126, 7477–7502 (2021). https://doi.org/10.1007/s11192-021-04048-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-021-04048-0

Keywords

Search

Navigation