TARU PUBLICATIONS
COLLNET Journal of Scientometrics and Information Management cover
Hybrid ·Peer-reviewed·ISSN (Online): 2168-930X·ISSN (Print): 0973-7766

WoS  JIF 2026 : 0.6 (Q3)

Powered by:Powered by

Half-Yearly Journal: Publishes research and articles on scientometrics and information management, including bibliometric analysis and quality assurance models.

Issues up to 2022 co-published with and available at:Taylor & Francis Online
EDITOR-CJSIM@tarupublications.com
Open Access Research Article

A dozen topics to debate before using bibliometrics to evaluate scholarly research articles : A beginner’s guide

*

* Corresponding author · click or hover a name for details

pp. 171–195Vol. 19Issue 2December 2025DOI: 10.47974/CJSIM-2024-11003XML
Published Online:
17 Dec 2025
Article type:
Research Article
Language:
EN
Article no.:
CJSIM-2024-11003
Pages:
171–195

Abstract

Citation counts, and metrics built thereupon, have transformed how many colleges and universities evaluate the scholarly output of their faculty. Journal impact factors and scholar-level h-indices, among many other metrics, now appear in promotion, renewal, and tenure decisions as well as award and grant deliberations. Correctly using these seemingly objective measures of scholarly performance requires more tact and skill than many administrators and faculty realize. Moreover, the dizzying array of research evaluation metrics can befuddle even the most earnest of research evaluators. This paper concisely reviews twelve key considerations based on abridged theory and frontline bibliometric application. The intended audience is non-expert faculty and administrators that seek basic bibliometric guidance as they move into research evaluation roles. 

Keywords

Subject Classifications

A1M5

References

[1] E. C. McKiernan, L. A. Schimanski, C. Muñoz Nieves, L. Matthias, M. T. Niles, and J. P. Alperin, “Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations,” Elife, vol. 8, e47338 (2019). [Online]. Available: https://doi.org/10.7554/eLife.47338
[2] A. Attema, W. Brouwer, and J. Van Exel, “Your right arm for a publication in AER?” Economic Inquiry, vol. 52, no. 1, pp. 495–502 (2014). [Online]. Available: https://doi.org/10.1111/ecin.12013
[3] M. Haley and M. McGee, “Jointly valuing journal visibility and author citation count: An axiomatic approach,” Journal of Informetrics, vol. 14, no. 1, art. 100988 (2020). [Online]. Available: https://doi.org/10.1016/j.joi.2019.100988
[4] J. Iglesias and C. Pecharromán, “Scaling the h-index for different scientific ISI fields,” Scientometrics, vol. 73, pp. 303–320 (2007). [Online]. Available: https://doi.org/10.1007/s11192-007-1805-x
[5] J. E. Hirsch, “An index to quantify an individual’s scientific research output,” Proceedings of the National Academy of Sciences, vol. 102, no. 46, pp. 16569–16572 (2005). [Online]. Available: https://doi.org/10.1073/pnas.0507655102
[6] R. Ball and D. Tunger, “Science indicators revisited – Science Citation Index versus SCOPUS: A bibliometric comparison of both citation databases,” Information Services & Use, vol. 26, no. 4, pp. 293–301 (2006). [Online]. Available: https://doi.org/10.3233/ISU-2006-26404
[7] D. W. Aksnes, L. Langfeldt, and P. Wouters, “Citations, citation indicators, and research quality: An overview of basic concepts and theories,” Sage Open, vol. 9, no. 1, pp. 1–11 (2019). [Online]. Available: https://doi.org/10.1177/215824401982957
[8] T. J. Phelan, “A compendium of issues for citation analysis,” Scientometrics, vol. 45, no. 1, pp. 117–136 (1999). [Online]. Available: https://doi.org/10.1007/BF02458472
[9] D. Sgroi and A. Oswald, “How should peer-review panels behave?” The Economic Journal, vol. 123, no. 570, pp. F255–F278 (2013). [Online]. Available: https://doi.org/10.1111/ecoj.12070
[10] V. Anauati, S. Galiani, and R. Gálvez, “Quantifying the life cycle of scholarly articles across fields of economic research,” Economic Inquiry, vol. 54, no. 2, pp. 1339–1355 (2016). [Online]. Available: https://doi.org/10.1111/ecin.12292
[11] L. Leydesdorff, “Caveats for the use of citation indicators in research and journal evaluations,” Journal of the American Society for Information Science and Technology, vol. 59, no. 2, pp. 278–287 (2008). [Online]. Available: https://doi.org/10.1002/asi.20743
[12] J. A. Crespo, Y. Li, and J. Ruiz–Castillo, “The measurement of the effect on citation inequality of differences in citation practices across scientific fields,” PLoS One, vol. 8, no. 3, e58727 (2013). [Online]. Available: https://doi.org/10.1371/journal.pone.0058727
[13] W. Marx and L. Bornmann, “On the causes of subject-specific citation rates in Web of Science,” Scientometrics, vol. 102, no. 2, pp. 1823–1827 (2015). [Online]. Available: https://doi.org/10.1007/s11192-014-1499-9
[14] L. Egghe, “Theory and practise of the g-index,” Scientometrics, vol. 69, no. 1, pp. 131–152 (2006). [Online]. Available: https://doi.org/10.1007/s11192-006-0144-7
[15] C. T. Zhang, “The e-index, complementing the h-index for excess citations,” PLoS One, vol. 4, no. 5, p. e5429 (2009). [Online]. Available: https://doi.org/10.1371/journal.pone.0005429
[16] A. Belikov and V. Belikov, “A citation-based, author- and age-normalized, logarithmic index for evaluation of individual researchers independently of publication counts,” F1000Research, vol. 4 (2015). [Online]. Available: https://doi.org/10.12688/f1000research.6033.2
[17] M. Haley, “An EigenFactor-weighted power mean generalization of the Euclidean Index,” PLoS ONE, vol. 14, no. 2, e0212760 (2019a). [Online]. Available: https://doi.org/10.1371/journal.pone.0212760
[18] A. Perianes-Rodriguez, L. Waltman, and N. J. Van Eck, “Constructing bibliometric networks: A comparison between full and fractional counting,” Journal of Informetrics, vol. 10, no. 4, pp. 1178–1195 (2016). [Online]. Available: https://doi.org/10.1016/j.joi.2016.10.006
[19] J. P. A. Ioannidis, K. Boyack, and P. F. Wouters, “Citation metrics: A primer on how (not) to normalize,” PLoS Biology, vol. 14, no. 9, e1002542 (2016). [Online]. Available: https://doi.org/10.1371/journal.pbio.1002542
[20] D. W. Johnston, M. Piatti, and B. Torgler, “Citation success over time: Theory or empirics?” Scientometrics, vol. 95, no. 3, pp. 1023–1029 (2013). [Online]. Available: https://doi.org/10.1007/s11192-012-0910-7
[21] J. Angrist, P. Azoulay, G. Ellison, R. Hill, and S. F. Lu, “Inside job or deep impact? Extramural citations and the influence of economic scholarship,” Journal of Economic Literature, vol. 58, no. 1, pp. 3–52 (2020). [Online]. Available: https://doi.org/10.1257/jel.20181508
[22] R. Miranda and E. Garcia-Carpintero, “Overcitation and overrepresentation of review papers in the most cited papers,” Journal of Informetrics, vol. 12, no. 4, pp. 1015–1030 (2018). [Online]. Available: https://doi.org/10.1016/j.joi.2018.08.006
[23] H. R. Jamali and M. Nikzad, “Article title type and its relation with the number of downloads and citations,” Scientometrics, vol. 88, no. 2, pp. 653–661 (2011). [Online]. Available: https://doi.org/10.1007/s11192-011-0412-z
[24] M. Van Wesel, S. Wyatt, and J. ten Haaf, “What a difference a colon makes: How superficial factors influence subsequent citation,” Scientometrics, vol. 98, no. 3, pp. 1601–1615 (2014). [Online]. Available: https://doi.org/10.1007/s11192-013-1154-x
[25] B. Deng, “Papers with shorter titles get more citations,” Nature, vol. 2, no. 8, article 150266 (2015). [Online]. Available: https://doi.org/10.1038/nature.2015.18246
[26] V. Ramírez-Castañeda, “Disadvantages in preparing and publishing scientific papers caused by the dominance of the English language in science: The case of Colombian researchers in biological sciences,” PLoS ONE, vol. 15, no. 9, e0238372 (2020). [Online]. Available: https://doi.org/10.1371/journal.pone.0238372
[27] M. Perry and P. Reny, “How to count citations if you must,” The American Economic Review, vol. 106, no. 9, pp. 2722–2741 (2016). [Online]. Available: https://doi.org/10.1257/aer.20140850
[28] D. I. Stern and R. S. Tol, “Depth and breadth relevance in citation metrics,” Economic Inquiry, vol. 59, no. 3, pp. 961–977 (2021). [Online]. Available: https://doi.org/10.1111/ecin.12994
[29] M. Errami and H. Garner, “A tale of two citations,” Nature, vol. 451, no. 7177, pp. 397–399 (2008). [Online]. Available: https://doi.org/10.1038/451397a
[30] J. Beel and B. Gipp, “Academic search engine optimization (ASEO): Optimizing scholarly literature for Google Scholar and Co.,” Journal of Scholarly Publishing, vol. 41, no. 2, pp. 176–190 (2010). [Online]. Available: https://doi.org/10.3138/jsp.41.2.176
[31] A. Diem and S. C. Wolter, “The use of bibliometrics to measure research performance in education sciences,” Research in Higher Education, vol. 54, no. 1, pp. 86–114 (2013). [Online]. Available: https://doi.org/10.1007/s11162-012-9264-5
[32] M. Haley, “Ranking top economics and finance journals using Microsoft Academic Search versus Google Scholar: How does the new Publish or Perish option compare?” Journal of the American Society for Information Science and Technology, vol. 65, no. 5, pp. 1079–1084 (2014). [Online]. Available: https://doi.org/10.1002/asi.23080
[33] M. Haley, “A ranking of journals for the aspiring health economist,” Applied Economics, vol. 48, no. 18, pp. 1710–1718 (2016). [Online]. Available: https://doi.org/10.1080/00036846.2015.1105927
[34] A. W. Harzing, “Two new kids on the block: How do Crossref and Dimensions compare with Google Scholar, Microsoft Academic, Scopus and the Web of Science?” Scientometrics, vol. 120, no. 1, pp. 341–349 (2019). [Online]. Available: https://doi.org/10.1007/s11192-019-03114-y
[35] J. Baas, M. Schotten, A. Plume, G. Côté, and R. Karimi, “Scopus as a curated, high-quality bibliometric data source for academic research in quantitative science studies,” Quantitative Science Studies, vol. 1, no. 1, pp. 377–386 (2020). [Online]. Available: https://doi.org/10.1162/qss_a_00019
[36] M. Seeber, M. Cattaneo, M. Meoli, and P. Malighetti, “Self-citation as strategic response to the use of metrics for career decisions,” Research Policy, vol. 48, no. 2, pp. 478–491 (2019). [Online]. Available: https://doi.org/10.1016/j.respol.2017.12.004
[37] A. Wilhite, E. Fong, and S. Wilhite, “The influence of editorial decisions and the academic network on self-citations and journal impact factors,” Research Policy, vol. 48, no. 6, pp. 1513–1522 (2019). [Online]. Available: https://doi.org/10.1016/j.respol.2019.03.003
[38] R. Van Noorden and D. S. Chawla, “Hundreds of extreme self-citing scientists revealed in new database,” Nature, vol. 572, no. 7771, pp. 578–580 (2019). [Online]. Available: https://doi.org/10.1038/d41586-019-02479-7
[39] J. Fowler and D. Aksnes, “Does self-citation pay?” Scientometrics, vol. 72, no. 3, pp. 427–437 (2007). [Online]. Available: https://doi.org/10.1007/s11192-007-1777-2
[40] A. Wilhite and E. Fong, “Coercive citation in academic publishing,” Science, vol. 335, no. 6068, pp. 542–543 (2012). [Online]. Available: https://doi.org/10.1126/science.1212540
[41] B. Martin, “Whither research integrity? Plagiarism, self-plagiarism and coercive citation in an age of research evaluatement,” Research Policy, vol. 42, no. 5, pp. 1005–1014 (2013). [Online]. Available: https://doi.org/10.1016/j.respol.2013.03.011
[42] M. Haley, “On the inauspicious incentives of the scholar-level h-index: An economist’s take on collusive and coercive citation,” Applied Economics Letters, vol. 24, no. 2, pp. 85–89 (2017). [Online]. Available: https://doi.org/10.1080/13504851.2016.1164812
[43] R. Van Noorden, “Brazilian citation scheme outed,” Nature, vol. 500, no. 7464, pp. 510–511 (2013). [Online]. Available: https://doi.org/10.1038/500510a
[44] R. Van Noorden, “Highly cited researcher banned from journal board for citation abuse,” Nature, vol. 578, no. 7794, pp. 200–202 (2020). [Online]. Available: https://doi.org/10.1038/d41586-020-00335-7
[45] A. Grudniewicz, D. Moher, K. D. Cobey, G. L. Bryson, S. Cukier, K. Allen, C. Ardern, P. Balcom, L. Barros, M. Berger, M. Cirocco, M. C. Cobey, M. C. Lalu, L. Laupacis, and J. M. Shamsher, “Predatory journals: no definition, no defence,” Nature, vol. 576, pp. 210–212 (2019).
[46] M. Biagioli, “Watch out for cheats in citation game,” Nature, vol. 535, no. 7611, p. 201 (2016). [Online]. Available: https://doi.org/10.1038/535201a
[47] P. Nattavudh, Y. Riyanto, and J. Knetsch, “Lower-rated publications do lower academics’ judgments of publication lists: Evidence from a survey experiment of economists,” Journal of Economic Psychology, vol. 66, pp. 33–44 (Jun. 2018). [Online]. Available: https://doi.org/10.1016/j.joep.2018.04.003
[48] H. F. Moed, L. Colledge, J. Reedijk, F. Moya-Anegón, V. Guerrero-Bote, A. Plume, and M. Amin, “Citation-based metrics are appropriate tools in journal evaluation provided that they are accurate and used in an informed way,” Scientometrics, vol. 92, no. 2, pp. 367–376 (2012). [Online]. Available: https://doi.org/10.1007/s11192-012-0679-8
[49] I. N. Sengupta, “Bibliometrics, informetrics, scientometrics and librametrics: An overview,” Libri, vol. 42, no. 2, pp. 75–98 (1992). [Online]. Available: https://doi.org/10.1515/libr.1992.42.2.75
[50] L. Bornmann, R. Mutz, S. Hug, and H. D. Daniel, “A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants,” Journal of Informetrics, vol. 5, no. 3, pp. 346–359 (2011). [Online]. Available: https://doi.org/10.1016/j.joi.2011.01.006
[51] L. Wildgaard, J. W. Schneider, and B. Larsen, “A review of the characteristics of 108 author-level bibliometric indicators,” Scientometrics, vol. 101, no. 1, pp. 125–158 (2014). [Online]. Available: https://doi.org/10.1007/s11192-014-1423-3
[52] L. Waltman, “A review of the literature on citation impact indicators,” Journal of Informetrics, vol. 10, no. 2, pp. 365–391 (2016). [Online]. Available: https://doi.org/10.1016/j.joi.2016.02.007
[53] L. Bornmann, R. Mutz, C. Neuhaus, and H. D. Daniel, “Citation counts for research evaluation: Standards of good practice for analyzing bibliometric data and presenting and interpreting results,” Ethics in Science and Environmental Politics, vol. 8, no. 1, pp. 93–102 (2008). [Online]. Available: https://doi.org/10.3354/esep00084
[54] J. Qiu, R. Zhao, S. Yang, and K. Dong, Informetrics: Theory, Methods and Applications. Cham: Springer (2017).
[55] R. Rousseau, L. Egghe, and R. Guns, Becoming Metric-Wise: A Bibliometric Guide for Researchers. Duxford, UK: Chandos Publishing (2018).
[56] E. Roldan-Valadez, S. Y. Salazar-Ruiz, R. Ibarra-Contreras, and C. Rios, “Current concepts on bibliometrics: A brief review about impact factor, Eigenfactor score, CiteScore, SCImago Journal Rank, Source-Normalised Impact per Paper, h-index, and alternative metrics,” Irish Journal of Medical Science (1971-), vol. 188, no. 3, pp. 939–951 (2019). [Online]. Available: https://doi.org/10.1007/s11845-018-1936-5
[57] A. Ahmi, Bibliometric Analysis for Beginners: A starter guide to begin with a bibliometric study using Scopus dataset and tools such as Microsoft Excel, Harzing’s Publish or Perish and VOSviewer software (2021).
[58] N. Donthu, S. Kumar, D. Mukherjee, N. Pandey, and W. M. Lim, “How to conduct a bibliometric analysis: An overview and guidelines,” Journal of Business Research, vol. 133, pp. 285–296 (2021).
[59] N. Bontis and A. Serenko, “A follow-up ranking of academic journals,” Journal of Knowledge Management, vol. 13, no. 1, pp. 16–26 (2009). [Online]. Available: https://doi.org/10.1108/13673270910931134
[60] P. O. Seglen, “Why the impact factor of journals should not be used for evaluating research,” British Medical Journal, vol. 314, pp. 498–502 (1997). [Online]. Available: https://doi.org/10.1136/bmj.314.7079.497
[61] P. O. Seglen, “Citation rates and journal impact factors are not suitable for evaluation of research,” Acta Orthopaedica Scandinavica, vol. 69, no. 3, pp. 224–229 (1998). [Online]. Available: https://doi.org/10.3109/17453679809000920
[62] A. Oswald, “An examination of the reliability of prestigious scholarly journals: Evidence and implications for decision-makers,” Economica, vol. 74, no. 293, pp. 21–31 (2007). [Online]. Available: https://doi.org/10.1111/j.1468-0335.2006.00575.x
[63] J. Bar-Ilan and G. Halevi, “Post retraction citations in context: A case study,” Scientometrics, vol. 113, pp. 547–565 (2017). [Online]. Available: https://doi.org/10.1007/s11192-017-2242-0
[64] V. Larivière and C. R. Sugimoto, “The journal impact factor: A brief history, critique, and discussion of adverse effects,” in Springer Handbook of Science and Technology Indicators, W. Glänzel, H. F. Moed, U. Schmoch, and M. Thelwall, Eds., Cham: Springer, pp. 3–24 (2019). [Online]. Available: https://doi.org/10.1007/978-3-030-02511-3_1
[65] M. Haley and M. McGee, “A parametric ‘parent metric’ approach for comparing maximum-normalized journal ranking metrics,” Journal of the Association for Information Science and Technology, vol. 69, no. 1, pp. 172–176 (2018). [Online]. Available: https://doi.org/10.1002/asi.23908
[66] A. W. Harzing, Publish or Perish (2007). [Online]. Available: https://harzing.com/resources/publish-or-perish
[67] J. J. Heckman and S. Moktan, “Publishing and promotion in economics: The tyranny of the top five,” Journal of Economic Literature, vol. 58, no. 2, pp. 419–470 (2020). [Online]. Available: https://doi.org/10.1257/jel.20191574
[68] A. Pudovkin and E. Garfield, “Rank-normalization impact factor: A way to compare journal performance across subject categories,” in Proc. 67th Annu. Meeting of the American Society for Information Science and Technology, vol. 41, no. 1, pp. 507–515 (2004). [Online]. Available: https://doi.org/10.1002/meet.1450410159
[69] F. Radicchi, S. Fortunato, and C. Castellano, “Universality of citation distributions: Toward an objective measure of scientific impact,” Proceedings of the National Academy of Sciences USA, vol. 105, no. 45, pp. 17268–17272 (2008). [Online]. Available: https://doi.org/10.1073/pnas.0806977105
[70] H. F. Moed, “Measuring contextual citation impact of scientific journals,” Journal of Informetrics, vol. 4, no. 3, pp. 265–277 (2010). [Online]. Available: https://doi.org/10.1016/j.joi.2010.01.002
[71] L. Leydesdorff, P. Zhou, and L. Bornmann, “How can impact factors be normalized across fields of science? An evaluation in terms of percentile ranks and fractional counts,” Journal of the American Society for Information Science and Technology, vol. 64, no. 1, pp. 96–107 (2013). [Online]. Available: https://doi.org/10.1002/asi.22765
[72] L. Waltman and N. J. van Eck, “Source normalized indicators of citation impact: An overview of different approaches and an empirical comparison,” Scientometrics, vol. 96, no. 3, pp. 699–716 (2013). [Online]. Available: https://doi.org/10.1007/s11192-012-0913-4
[73] S. Alonso, F. Cabrerizo, E. Herrera-Viedma, and F. Herrera, “hg-index: A new index to characterize the scientific output of researchers based on the h-and g-indices,” Scientometrics, vol. 82, no. 2, pp. 391–400 (2010). [Online]. Available: https://doi.org/10.1007/s11192-009-0047-5
[74] R. S. Tol, “The Matthew effect defined and tested for the 100 most prolific economists,” Journal of the American Society for Information Science and Technology, vol. 60, no. 2, pp. 420–426 (2009). [Online]. Available: https://doi.org/10.1002/asi.20968
[75] V. Larivière and Y. Gingras, “The impact factor’s Matthew Effect: A natural experiment in bibliometrics,” Journal of the American Society for Information Science and Technology, vol. 61, no. 2, pp. 424–427 (2010). [Online]. Available: https://doi.org/10.1002/asi.21232
[76] K. Drivas and D. Kremmydas, “The Matthew effect of a journal’s ranking,” Research Policy, vol. 49, no. 4, art. 103951 (2020). [Online]. Available: https://doi.org/10.1016/j.respol.2020.103951
[77] R. S. Tol, “The Matthew effect for cohorts of econ

Views: 195Downloads: 106Citations: 0