Skip to main content
SpringerLink
Log in

High social media attention scores are not reflective of study quality: an altmetrics-based content analysis

  • IM - ORIGINAL
  • Published:
Internal and Emergency Medicine Aims and scope Submit manuscript

Abstract

Recent literature has demonstrated the associations between social media attention, as measured by altmetric attention score (AAS), and higher citation rates across medical disciplines. Despite increasing use of AAS, an understanding of factors associated with higher AAS and social media attention remains lacking. Furthermore, if this increased attention correlates with a higher methodological quality and lower biases has not been determined. Therefore, the purpose of the current study was to determine the relationship between methodological quality, study biases and the AAS in randomized controlled trials (RCTs). All RCTs from 2016 in the New England Journal of Medicine (NEJM), Journal of the American Medical Society (JAMA), and Lancet were extracted and the (1) AAS; (2) Methodological Bias (JADAD Scale); Study Bias (Cochrane Risk-of-Bias tool for RCTs) recorded. A total of 296 RCTs with a median (range) AAS and citation rate per article of 234.0(7–4079) and 165.0(4–3257), respectively, were included. The AAS was positively associated with citation rate (β 0.19, 95% CI 0.10–0.29; P < 0.001). Methodological bias was not associated with the AAS (β − 36.3, 95% CI − 83.5–10.9; P = 0.131), but was negatively associated with higher citation rates (β − 66.4, 95% CI − 106.0 to − 26.9; P = 0.001). The number of study biases was not associated with the AAS (β 43.7, 95% CI − 6.3–93.7;P = 0.086), but was positively associated with a higher citation rate (β 64.5, 95% CI 22.4–106.6; P = 0.003). The online attention of RCTs in medical journals was not necessarily reflective of high methodological quality and minimal study biases, but was associated with higher citation rates. Researchers and clinicians should critically examine each article despite the amount of online attention an article receives as the AAS does not necessarily reflect article quality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Seglen PO (1997) Why the impact factor of journals should not be used for evaluating research. BMJ 314:497

    Article  Google Scholar 

  2. Wang J, Alotaibi NM, Ibrahim GM, Kulkarni AV, Lozano AM (2017) The spectrum of altmetrics in neurosurgery: the top 100 “Trending” articles in neurosurgical journals. World Neurosurg 103:883-895.e881

    Article  Google Scholar 

  3. Pulido CM, Redondo-Sama G, Sordé-Martí T, Flecha R (2018) Social impact in social media: a new method to evaluate the social impact of research. PLoS ONE 13:e0203117

    Article  Google Scholar 

  4. Barakat AF, Nimri N, Shokr M et al (2018) Correlation of altmetric attention score with article citations in cardiovascular research. J Am Coll Cardiol 72:952–953

    Article  Google Scholar 

  5. Punia V, Aggarwal V, Honomichl R, Rayi A (2019) Comparison of attention for neurological research on social media vs academia: an altmetric score analysis. JAMA Neurol. https://doi.org/10.1001/jamaneurol.2019.1791

    Article  PubMed  PubMed Central  Google Scholar 

  6. Parrish JM, Jenkins NW, Brundage TS, Hrynewycz NM, Singh K (2020) The top 100 spine surgery articles on social media: an altmetric study. Spine (Phila Pa 1976). https://doi.org/10.1016/j.spinee.2020.05.458

    Article  Google Scholar 

  7. Kunze KN, Polce EM, Vadhera A et al (2020) What Is the Predictive ability and academic impact of the altmetrics score and social media attention? Am J Sports Med 48:1056–1062

    Article  Google Scholar 

  8. Altmetrics. Attention Sources coverage dates. Last modified Sep 2021. Available at https://help.altmetric.com/support/solutions/articles/6000240455-attention-sources-coverage-dates. Accessed April 21, 2021

  9. Altmetrics. How is the Altmetric Attention Score calculated? Last modified Sep 2021. Available at https://help.altmetric.com/support/solutions/articles/6000233311-how-is-the-altmetric-attention-score-calculated-. Accessed April 21, 2021.21

  10. Chaimani A, Salanti G, Leucht S, Geddes JR, Cipriani A (2017) Common pitfalls and mistakes in the set-up, analysis and interpretation of results in network meta-analysis: what clinicians should look for in a published article. Evid Based Ment Health 20:88–94

    Article  Google Scholar 

  11. Altman DG (2002) Poor-quality medical research: what can journals do? JAMA 287:2765–2767

    Article  Google Scholar 

  12. Boissel JP (1989) Impact of randomized clinical trials on medical practices. Control Clin Trials 10:120S-134S

    Article  CAS  Google Scholar 

  13. Nwachukwu BU, Kahlenberg CA, Lehman JD, Lyman S, Marx RG (2017) Characteristics of orthopedic publications in high-impact general medical journals. Orthopedics 40:e405–e412

    Article  Google Scholar 

  14. Aksnes D, Langfeldt L, Wouters P (2019) Citations, citation indicators, and research quality: an overview of basic concepts and theories. SAGE Open. https://doi.org/10.1177/2158244019829575

    Article  Google Scholar 

  15. Hook D, Porter S, Herzog C (2018) Dimensions: building context for search and evaluation. Front Res Metr Anal. https://doi.org/10.3389/frma.2018.00023

    Article  Google Scholar 

  16. Polce E, Kunze KN, Farivar D, Fu MC, Nwachukwu BU, Nho SJ, Chahla J (2020) Orthopedic randomized controlled trials published in general medical journals are associated with higher altmetric attention scores and social media attention. Arthrosc J Arthrosc Relat Surg. https://doi.org/10.1016/j.arthro.2020.09.015

    Article  Google Scholar 

  17. Jadad AR, Moore RA, Carroll D et al (1996) Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 17:1–12

    Article  CAS  Google Scholar 

  18. Olivo SA, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ (2008) Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther 88:156–175

    Article  Google Scholar 

  19. Higgins JP, Altman DG, Gotzsche PC et al (2011) The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 343:D5928

    Article  Google Scholar 

  20. Farrah K, Young K, Tunis MC, Zhao L (2019) Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols. Syst Rev 8:280

    Article  Google Scholar 

  21. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. Available from www.training.cochrane.org/handbook.

  22. Sathianathen NJ, Lane Iii R, Murphy DG et al (2020) Social media coverage of scientific articles immediately after publication predicts subsequent citations—#SoME_Impact score: observational analysis. J Med Internet Res 22:e12288

    Article  Google Scholar 

  23. Luc JGY, Percy E, Hirji S et al (2020) Predictors of high impact articles in the annals of thoracic surgery. Ann Thorac Surg. https://doi.org/10.1016/j.athoracsur.2020.04.102

    Article  PubMed  PubMed Central  Google Scholar 

  24. Ladeiras-Lopes R, Clarke S, Vidal-Perez R et al (2020) Twitter promotion predicts citation rates of cardiovascular articles: a preliminary analysis from the ESC Journals Randomized Study. Eur Heart J. https://doi.org/10.1093/eurheartj/ehaa211

    Article  PubMed  Google Scholar 

  25. Chang J, Desai N, Gosain A (2019) Correlation between altmetric score and citations in pediatric surgery core journals. J Surg Res 243:52–58

    Article  Google Scholar 

  26. Haustein S, Bowman TD, Holmberg K, Tsou A, Sugimoto CR, Larivière V (2016) Tweets as impact indicators: examining the implications of automated “bot” accounts on Twitter. J Assoc Inf Sci Technol 67:232–238

    Article  Google Scholar 

  27. Ruano J, Aguilar-Luque M, Gomez-Garcia F et al (2018) The differential impact of scientific quality, bibliometric factors, and social media activity on the influence of systematic reviews and meta-analyses about psoriasis. PLoS ONE 13:e0191124

    Article  Google Scholar 

  28. Bornmann L, Marx W (2016) The journal Impact Factor and alternative metrics: a variety of bibliometric measures has been developed to supplant the impact factor to better assess the impact of individual research papers. EMBO Rep 17:1094–1097

    Article  CAS  Google Scholar 

  29. Bornmann L, Haunschild R (2018) Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000Prime data. PLoS ONE 13:e0197133

    Article  Google Scholar 

  30. Zahedi Z, Haustein S (2018) On the relationships between bibliographic characteristics of scientific documents and citation and Mendeley readership counts: a large-scale analysis of Web of Science publications. J Informet 12:191–202

    Article  Google Scholar 

  31. Didegah F, Bowman TD, Holmberg K (2018) On the differences between citations and altmetrics: an investigation of factors driving altmetrics versus citations for finnish articles. J Am Soc Inf Sci 69:832–843

    Google Scholar 

  32. Holmberg K, Vainio J (2018) Why do some research articles receive more online attention and higher altmetrics? Reasons for online success according to the authors. Scientometrics 116:435–447

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Orthopaedic Surgery, Hospital for Special Surgery, 535 East 70th Street, New York, NY, 10021, USA

    Kyle Nash Kunze MD

  2. Weill Cornell Medicine, New York, NY, USA

    Joseph Emanuele Manzi BS

  3. University of Wisonconsin School of Medicine and Public Health, Madison, WI, USA

    Evan Michael Polce BS & Amar Vadhera BS

  4. Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada

    Mohit Bhandari MD, PhD, FRCSC

  5. Division of Orthopaedic Surgery, Department of Surgery, McMaster University, Hamilton, ON, Canada

    Mohit Bhandari MD, PhD, FRCSC

  6. Department of Orthopaedic Surgery, Cleveland Clinic, Cleveland, OH, USA

    Nicolas Santiago Piuzzi MD

Authors
  1. Kyle Nash Kunze MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joseph Emanuele Manzi BS
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Evan Michael Polce BS
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Amar Vadhera BS
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mohit Bhandari MD, PhD, FRCSC
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Nicolas Santiago Piuzzi MD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kyle Nash Kunze MD.

Ethics declarations

Conflict of interest

The authors report no conflicts of interest related to the current work.

Ethical approval

The current study uses publicaly available data and was exempt from formal ethical approval. Human and animal rights: No human or animal subjects were directly implicated in the current study. Informed consent: Informed consent was not required as this was a review of publicaly available information.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kunze, K.N., Manzi, J.E., Polce, E.M. et al. High social media attention scores are not reflective of study quality: an altmetrics-based content analysis. Intern Emerg Med 17, 1363–1374 (2022). https://doi.org/10.1007/s11739-022-02939-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11739-022-02939-5

Keywords

Search

Navigation