Citations
Academic Quality Or Qualis Classification? Scientific Production Of Graduate Programs In Collective Health, 2013-2016
DOI:
https://doi.org/10.21713/rbpg.v42.2817Keywords:
scientific production evaluation, quality of articles, Qualis classification, publication profile, citationsAbstract
Objective: to describe the scientific articles produced by professors and students of graduate programs in Public Health, in the period 2013-2016, according to the characteristics of the vehicles and articles, seeking to answer three questions: Do the citations received reflect the quality of the articles?; Are the citations received associated with the Qualis strata?; What is the relationship between quality and Qualis? Method: sample of 1412 articles, from a universe of 16,771, representative of journals according to field of knowledge and Qualis stratum, described according to characteristics of the journals and the articles themselves. Quality was assessed using standardized tools for each study design. The Qualis periodical classification 2017-2020 was used. Citations of articles were obtained from Web of Science, Scopus and Google Scholar, 7 to 10 years after publication. Results: The articles were published in 509 different vehicles, national and foreign, 69% in stratum A of Qualis (47%, A1+A2), 64% in English, 74% empirical research, 75% with 3 to 9 authors, 64% in cooperation between institutions, 70% classified as very good or good from the academic merit point of view. The median number of citations received is proportional to the academic quality of the articles and the Qualis stratum of the journal in which it was published, with each stratum having a gradient according to the quality. Conclusion: stratification by Qualis can discriminate the quality and academic merit of articles, however, the qualitative evaluation of a smaller number of products helps the programs’ evaluation.
References
ABRAMO, Giovanni. The forced battle between peer-review and scientometric research assessment: why the CoARA initiative is unsound. Research Evaluation, v. 33, n. 1, rvae021, 2024. DOI: https://doi.org/10.1093/reseval/rvae021.
ABRAMO, Giovanni; D’ANGELO, Ciriaco Andrea; GRILLI, Leonardo. The role of non-scientific factors vis-à-vis the quality of publications in determining their scholarly impact. Scientometrics, v. 129, n. 7, p. 5003-5019, 2024. DOI: https://doi.org/10.1007/s11192-024-05106-z.
AKSNES, Dag W.; PIRO, Fredrik Niclas; FOSSUM, Lone Wanderås. Citation metrics covary with researchers’ assessments of the quality of their works. Quantitative Science Studies, v. 4, n. 1, p. 105-126, 2023. DOI: https://doi.org/10.1162/qss_a_00241.
ALMEIDA-FILHO, Naomar et al. Research on health inequalities in Latin America and the Caribbean: bibliometric analysis (1971-2000) and descriptive content analysis (1971-1995). American Journal of Public Health, v. 93, n. 12, p. 2037-2043, 2003. DOI: https://doi.org/10.2105/AJPH.93.12.2037.
ARCHAMBAULT, Éric et al. Comparing bibliometric statistics obtained from the Web of Science and Scopus. Journal of the American Society for Information Science and Technology, v. 60, n. 7, p. 1320-1326, 2009. DOI: https://doi.org/10.1002/asi.21062.
BARATA, Rita Barradas. Estratégias de avaliação da produção científica: reflexões sobre suas limitações e complementaridades. Perspectivas em Ciência da Informação, v. 28, e42242, 2023. DOI: https://doi.org/10.1590/1981-5344/42242.
BARATA, Rita de Cássia Barradas. Dez coisas que você deveria saber sobre o Qualis. Revista Brasileira de Pós-Graduação, v. 13, n. 30, p. 13-40, 2016. DOI: https://doi.org/10.21713/2358-2332.2016.v13.947.
BARRETO, Maurício L. Como avaliar as ciências com uma deficiente ciência da avaliação científica? Cadernos de Saúde Pública, v. 29, n. 9, p. 1719-1721, 2013b. DOI: https://doi.org/10.1590/0102-311XCO060913.
BARRETO, Maurício L. Crescimento e tendência da produção científica em epidemiologia no Brasil. Revista de Saúde Pública, v. 40, p. 79-85, 2006. Edição especial. DOI: https://doi.org/10.1590/S0034-89102006000400012.
BARRETO, Maurício L. O desafio de avaliar o impacto das ciências para além da bibliometria. Revista de Saúde Pública, v. 47, n. 4, p. 834-837, 2013a. DOI: https://doi.org/10.1590/S0034-8910.2013047005073.
BORNMANN, Lutz. How to analyze percentile citation impact data meaningfully in bibliometrics: the statistical analysis of distributions, percentile rank classes, and top-cited papers. Journal of the American Society for Information Science and Technology, v. 64, n. 3, p. 587-595, 2013. DOI: https://doi.org/10.1002/asi.22792.
BORNMANN, Lutz; LEYDESDORFF, Loet. The validation of (advanced) bibliometric indicators through peer assessments: a comparative study using data from InCites and F1000. Journal of Informetrics, v. 7, n. 2, p. 286-291, 2013. DOI: https://doi.org/10.1016/j.joi.2012.12.003.
CABEZAS-CLAVIJO, Álvaro; TORRES-SALINAS, Daniel. Bibliometric reports for institutions: best practices in a responsible metrics scenario. Frontiers in Research Metrics and Analytics, v. 6, 696470, 2021. DOI: https://doi.org/10.3389/frma.2021.696470.
CAMARGO JR., Kenneth Rochel de; COELI, Claudia Medina; CAETANO, Rosângela; MAIA, Vanessa Rangel. Produção intelectual em saúde coletiva: epistemologia e evidências de diferentes tradições. Revista de Saúde Pública, v. 44, n. 3, p. 394-398, 2010. DOI: https://doi.org/10.1590/S0034-89102010005000008.
CAPES. Diretoria de Avaliação. Documento orientador de APCN: Área 22: Saúde Coletiva. Brasília, DF: CAPES, 2023. Disponível em: https://www.gov.br/capes/pt-br/acesso-a-informacao/acoes-e-programas/avaliacao/sobre-a-avaliacao/areas-avaliacao/sobre-as-areas-de-avaliacao/colegio-de-ciencias-da-vida/ciencias-da-saude/Saude_Coletiva_Documento_Orientador_APCN_2023.pdf. Acesso em: 17 set. 2026.
CASTILLO, José Antonio; POWELL, Michael A. Análisis de la producción científica del Ecuador e impacto de la colaboración internacional en el periodo 2006-2015. Revista Española de Documentación Científica, v. 42, n. 1, e225, 2019. DOI: https://doi.org/10.3989/redc.2019.1.1567.
CHINCHILLA-RODRÍGUEZ, Zaida; ZACCA-GONZÁLEZ, Grisel; VARGAS-QUESADA, Benjamín; MOYA-ANEGÓN, Félix. Latin American scientific output in Public Health: combined analysis using bibliometric, socioeconomic and health indicators. Scientometrics, v. 102, n. 1, p. 609-628, 2015. DOI: https://doi.org/10.1007/s11192-014-1349-9.
CLARK, Otavio A. C.; CASTRO, Aldemar A. Searching the Literatura Latino Americana e do Caribe em Ciências da Saúde (LILACS) database improves systematic reviews. International Journal of Epidemiology, v. 31, n. 1, p. 112-114, 2002. DOI: https://doi.org/10.1093/ije/31.1.112.
COIMBRA JR., Carlos E. A. Produção científica em saúde pública e as bases bibliográficas internacionais. Cadernos de Saúde Pública, v. 15, n. 4, p. 883-888, 1999. DOI: https://doi.org/10.1590/S0102-311X1999000400022.
D’IPPOLITI, Carlo; GOBBI, Lucio; MONGEAU OSPINA, Christian A.; ZACCHIA, Giulia. Social determinants of citations: an empirical analysis of UK economists. Kyklos, v. 76, n. 4, p. 827-858, 2023. DOI: https://doi.org/10.1111/kykl.12352.
FALAGAS, Matthew E.; PITSOUNI, Eleni I.; MALIETZIS, George A.; PAPPAS, Georgios. Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. The FASEB Journal, v. 22, n. 2, p. 338-342, 2008. DOI: https://doi.org/10.1096/fj.07-9492LSF.
GLÄNZEL, Wolfgang; LETA, Jacqueline; THIJS, Bart. Science in Brazil. Part 1: a macro-level comparative study. Scientometrics, v. 67, n. 1, p. 67-86, 2006. DOI: https://doi.org/10.1007/s11192-006-0055-7.
HARZING, Anne-Wil K.; VAN DER WAL, Ron. Google Scholar as a new source for citation analysis. Ethics in Science and Environmental Politics, v. 8, p. 61-73, 2008. DOI: https://doi.org/10.3354/esep00076.
IRIART, Jorge Alberto Bernstein; TRAD, Leny Alves Bomfim. Perfil da produção bibliográfica em Ciências Sociais e Humanas em Saúde e a percepção de pesquisadores: avanços, limites e desafios. Cadernos de Saúde Pública, v. 36, n. 5, e00085019, 2020. DOI: https://doi.org/10.1590/0102-311X00085019.
JAKAB, Martin; KITTL, Eva; KIESSLICH, Tobias. How many authors are (too) many? A retrospective, descriptive analysis of authorship in biomedical publications. Scientometrics, v. 129, n. 3, p. 1299-1328, 2024. DOI: https://doi.org/10.1007/s11192-024-04928-1.
KIESSLICH, Tobias; BEYREIS, Marlena; ZIMMERMANN, Georg; TRAWEGER, Andreas. Citation inequality and the Journal Impact Factor: median, mean, (does it) matter? Scientometrics, v. 126, n. 2, p. 1249-1269, 2021. DOI: https://doi.org/10.1007/s11192-020-03812-y.
LETA, Jacqueline; GLÄNZEL, Wolfgang; THIJS, Bart. Science in Brazil. Part 2: sectoral and institutional research profiles. Scientometrics, v. 67, n. 1, p. 87-105, 2006. DOI: https://doi.org/10.1007/s11192-006-0051-y.
LOZANO, George A.; LARIVIÈRE, Vincent; GINGRAS, Yves. The weakening relationship between the impact factor and papers’ citations in the digital age. Journal of the American Society for Information Science and Technology, v. 63, n. 11, p. 2140-2145, 2012. DOI: https://doi.org/10.1002/asi.22731.
MARQUES, Fabrício. Código sagrado. Pesquisa FAPESP, n. 181, p. 31-33, mar. 2011. Disponível em: https://revistapesquisa.fapesp.br/codigo-sagrado/. Acesso em: 17 set. 2026.
MENEGHINI, Rogerio; MUGNAINI, Rogerio; PACKER, Abel L. International versus national oriented Brazilian scientific journals: a scientometric analysis based on SciELO and JCR-ISI databases. Scientometrics, v. 69, n. 3, p. 529-538, 2006. DOI: https://doi.org/10.1007/s11192-006-0168-z.
MENEGHINI, Rogerio; PACKER, Abel L. Is there science beyond English? EMBO Reports, v. 8, n. 2, p. 112-116, 2007. DOI: https://doi.org/10.1038/sj.embor.7400906.
MOYA-ANEGÓN, Félix; GUERRERO-BOTE, Vicente P.; BORNMANN, Lutz; MOED, Henk F. The research guarantors of scientific papers and the output counting: a promising new approach. Scientometrics, v. 97, n. 2, p. 421-434, 2013. DOI: https://doi.org/10.1007/s11192-013-1046-0.
NOVAES, Hillegonda Maria Dutilh. Muitas dúvidas, poucas certezas, enquanto isso... Cadernos de Saúde Pública, v. 29, n. 9, p. 1715-1716, 2013. DOI: https://doi.org/10.1590/0102-311XCO030913.
OPPENHEIM, Charles. The correlation between citation counts and the 1992 Research Assessment Exercise ratings for British research in genetics, anatomy and archaeology. Journal of Documentation, v. 53, n. 5, p. 477-487, 1997. DOI: https://doi.org/10.1108/EUM0000000007207.
PACKER, Abel Laerte. Indicadores de centralidade nacional da pesquisa comunicada pelos periódicos de Saúde Coletiva editados no Brasil. Ciência & Saúde Coletiva, v. 20, n. 7, p. 1983-1995, 2015. DOI: https://doi.org/10.1590/1413-81232015207.07122015.
PAIM, Jairnilson Silva; ALMEIDA FILHO, Naomar de. Saúde coletiva: uma “nova saúde pública” ou campo aberto a novos paradigmas? Revista de Saúde Pública, v. 32, n. 4, p. 299-316, 1998. DOI: https://doi.org/10.1590/S0034-89101998000400001.
PAIM, Jairnilson Silva; TEIXEIRA, Carmen Fontes. Política, planejamento e gestão em saúde: balanço do estado da arte. Revista de Saúde Pública, v. 40, p. 73-78, 2006. Edição especial. DOI: https://doi.org/10.1590/S0034-89102006000400011.
PELLEGRINI FILHO, Alberto; GOLDBAUM, Moisés; SILVI, John. Production of scientific articles on health in six Latin American countries, 1973-1992. Revista Panamericana de Salud Pública, v. 2, n. 2, p. 121-132, 1997. DOI: https://doi.org/10.1590/S1020-49891997000800005.
PENDLEBURY, David A. The use and misuse of journal metrics and other citation indicators. Archivum Immunologiae et Therapiae Experimentalis, v. 57, n. 1, p. 1-11, 2009. DOI: https://doi.org/10.1007/s00005-009-0008-y.
PEREIRA, Júlio Cesar Rodrigues. Revista de Saúde Pública: quarenta anos da produção científica no Brasil. Revista de Saúde Pública, v. 40, n. esp., p. 148-159, 2006. DOI: https://doi.org/10.1590/S0034-89102006000400020.
PEREIRA, Júlio Cesar Rodrigues; VASCONCELLOS, Juliana Parreira; FURUSAWA, Lucilla; BARBATI, Augusto de Moura. Who’s who and what’s what in Brazilian Public Health Sciences. Scientometrics, v. 73, p. 37-52, 2007. DOI: https://doi.org/10.1007/s11192-007-1787-8.
REY-ROCHA, Jesús; MARTÍN-SEMPERE, María José; LÓPEZ-VERA, Fernando; MARTÍNEZ-FRÍAS, Jesús. English versus Spanish in science evaluation. Nature, v. 397, n. 6714, p. 14, 1999. DOI: https://doi.org/10.1038/16132.
RUSHFORTH, Alex. Letter: response to Torres-Salinas et al. on “bibliometric denialism”. Scientometrics, v. 128, n. 12, p. 6781-6784, 2023. DOI: https://doi.org/10.1007/s11192-023-04842-y.
SANTOS, Solange Maria dos; FRAUMANN, Grischa; BELLI, Simone; MUGNAINI, Rogerio. The relationship between the language of scientific publication and its impact in the field of public and collective health. Journal of Scientometric Research, v. 10, n. 1s, p. s78-s87, 2021. DOI: https://doi.org/10.5530/jscires.10.1s.24.
SCOTT, Alister. Peer review and the relevance of science. Futures, v. 39, n. 7, p. 827-845, 2007. DOI: https://doi.org/10.1016/j.futures.2006.12.009.
SOBRAL, Natanael Vitor; TRAD, Leny Alves Bomfim; BARROS FILHO, Delma; CRUZ, Tatyane Lúcia. Perfil da produção científica brasileira em Saúde Coletiva: análise preliminar. In: ENCONTRO BRASILEIRO DE BIBLIOMETRIA E CIENTOMETRIA, 6., 2018, Rio de Janeiro. Anais do 6º Encontro Brasileiro de Bibliometria e Cientometria. Rio de Janeiro: UFRJ, 2018. p. 344-352. Disponível em: https://ebbc.inf.br/ojs/index.php/ebbc/article/view/682. Acesso em: 17 set. 2026.
STRUCHINER, Claudio José. Avaliação da qualidade da produção científica e suas consequências imprevistas e indesejadas: um conceito autoevidente? Cadernos de Saúde Pública, v. 29, n. 9, p. 1716-1717, 2013. DOI: https://doi.org/10.1590/0102-311XCO040913.
SZKLO, Moyses. Quality of scientific articles. Revista de Saúde Pública, v. 40, n. esp., p. 30-35, 2006. DOI: https://doi.org/10.1590/S0034-89102006000400005.
TAHAMTAN, Iman; SAFIPOUR AFSHAR, Askar; AHAMDZADEH, Khadijeh. Factors affecting number of citations: a comprehensive review of the literature. Scientometrics, v. 107, n. 3, p. 1195-1225, 2016. DOI: https://doi.org/10.1007/s11192-016-1889-2.
TORRES-SALINAS, Daniel; ARROYO-MACHADO, Wenceslao; ROBINSON-GARCIA, Nicolas. Bibliometric denialism. Scientometrics, v. 128, n. 9, p. 5357-5359, 2023. DOI: https://doi.org/10.1007/s11192-023-04787-2.
TRAAG, Vincent A. Inferring the causal effect of journals on citations. Quantitative Science Studies, v. 2, n. 2, p. 496-504, 2021. DOI: https://doi.org/10.1162/qss_a_00128.
VAN LEEUWEN, Thed N.; MOED, Henk F.; TIJSSEN, Robert J. W.; VISSER, Martijn S.; VAN RAAN, Anthony F. J. Language biases in the coverage of the Science Citation Index and its consequences for international comparisons of national research performance. Scientometrics, v. 51, n. 1, p. 335-346, 2001. DOI: https://doi.org/10.1023/A:1010549719484.
VIACAVA, Francisco. Produção científica dos cursos de pós-graduação em Saúde Coletiva no período 1998-2006. Ciência & Saúde Coletiva, v. 15, n. 4, p. 1977-1988, 2010. DOI: https://doi.org/10.1590/S1413-81232010000400013.
VICTORA, Cesar G.; MOREIRA, Carmen B. Publicações científicas e as relações Norte-Sul: racismo editorial? Revista de Saúde Pública, v. 40, n. esp., p. 36-42, 2006. DOI: https://doi.org/10.1590/S0034-89102006000400006.
VIEIRA, Elizabeth S. The influence of research collaboration on citation impact: the countries in the European Innovation Scoreboard. Scientometrics, v. 128, n. 6, p. 3555-3579, 2023. DOI: https://doi.org/10.1007/s11192-023-04715-4.
WAINER, Jacques; ECKMANN, Michael; ROCHA, Anderson. Peer-selected “best papers”: are they really that “good”? PLOS ONE, v. 10, n. 3, e0118446, 2015. DOI: https://doi.org/10.1371/journal.pone.0118446.
WALTMAN, Ludo; COSTAS, Rodrigo. F1000 recommendations as a potential new data source for research evaluation: a comparison with citations. Journal of the Association for Information Science and Technology, v. 65, n. 3, p. 433-445, 2014. DOI: https://doi.org/10.1002/asi.23040.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Brazilian Journal of Graduate Studies
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors who publish in this magazine agree to the following terms:
- They hold the copyright on the work and grant the magazine the right of first publication, with the work simultaneously licensed under the Creative Commons Attribution License that allows sharing the work with recognition of authorship and initial publication in this magazine.
- They are authorized to assume additional contracts separately, for non-exclusive distribution of the version of the work published in this magazine, with recognition of authorship and initial publication in this magazine.
- They are allowed and are encouraged to publish and distribute their work online (e.g., in institutional repositories or their personal website), provided that the initial publication in this magazine is mentioned, as this can generate changes.
Furthermore, authors represent that:
- All the information contained in the manuscript are facts, true, or based on empirical or scientific research.
- The use of any trademark or copyright in the manuscript has been credited to its owner or permission to use the name was duly granted.
