Purpose: Analyze the diversity of citation distributions to publications in different research topics to investigate the accuracy of size-independent, rank-based indicators. Top percentile-based indicators are the most common indicators of this type, and the evaluations of Japan are the most evident misjudgments. Design/methodology/approach: The distributions of citations to publications from countries and in journals in several research topics were analyzed along with the corresponding global publications using histograms with logarithmic binning, double rank plots, and normal probability plots of log-transformed numbers of citations. Findings: Size-independent, top percentile-based indicators are accurate when the global ranks of local publications fit a power law, but deviations in the least cited papers are frequent in countries and occur in all journals with high impact factors. In these cases, a single indicator is misleading. Comparisons of proportions of uncited papers are the best way to predict these deviations. Research limitations: The study is fundamentally analytical; its results describe mathematical facts that are self-evident. Practical implications: Respectable institutions, such as the OECD, European Commission, US National Science Board, and others, produce research country rankings and individual evaluations using size-independent percentile indicators that are misleading in many countries. These misleading evaluations should be discontinued because they cause confusion among research policymakers and lead to incorrect research policies. Originality/value: Studies linking the lower tail of citation distribution, including uncited papers, to percentile research indicators have not been performed previously. The present results demonstrate that studies of this type are necessary to find reliable procedures for research assessments.
翻译:目的:分析不同研究主题中出版物引用分布的多样性,以探究规模无关、基于排名的指标的准确性。基于顶端百分位数的指标是此类指标中最常见的类型,而对日本的评估则是最明显的误判案例。设计/方法/途径:通过使用对数分箱直方图、双秩图以及对数转换引用次数的正态概率图,分析了多个研究主题中国家和期刊出版物的引用分布及其对应的全球出版物。研究发现:当本地出版物的全球排名符合幂律分布时,规模无关的顶端百分位数指标是准确的,但在引用最少的论文中,国家和所有高影响因子期刊中频繁出现偏差。在这些情况下,单一指标具有误导性。比较未获引用论文的比例是预测这些偏差的最佳方法。研究局限:本研究本质上是分析性的;其结果描述的是不言自明的数学事实。实际意义:诸如经合组织、欧盟委员会、美国国家科学委员会等权威机构,使用在许多国家具有误导性的规模无关百分位数指标来制作研究国家排名和个体评估。这些误导性评估应当停止,因为它们会导致研究政策制定者困惑并引发错误的研究政策。原创性/价值:此前尚未有研究将引用分布的下尾端(包括未获引用论文)与百分位数研究指标联系起来。本研究结果表明,此类研究对于寻找可靠的研究评估程序是必要的。