There is widespread debate on whether to anonymize author identities in peer review. The key argument for anonymization is to mitigate bias, whereas arguments against anonymization posit various uses of author identities in the review process. The Innovations in Theoretical Computer Science (ITCS) 2023 conference adopted a middle ground by initially anonymizing the author identities from reviewers, revealing them after the reviewer had submitted their initial reviews, and allowing the reviewer to change their review subsequently. We present an analysis of the reviews pertaining to the identification and use of author identities. Our key findings are: (I) A majority of reviewers self-report not knowing and being unable to guess the authors' identities for the papers they were reviewing. (II) After the initial submission of reviews, 7.1% of reviews changed their overall merit score and 3.8% changed their self-reported reviewer expertise. (III) There is a very weak and statistically insignificant correlation of the rank of authors' affiliations with the change in overall merit; there is a weak but statistically significant correlation with respect to change in reviewer expertise. We also conducted an anonymous survey to obtain opinions from reviewers and authors. The main findings from the 200 survey responses are: (i) A vast majority of participants favor anonymizing author identities in some form. (ii) The "middle-ground" initiative of ITCS 2023 was appreciated. (iii) Detecting conflicts of interest is a challenge that needs to be addressed if author identities are anonymized. Overall, these findings support anonymization of author identities in some form (e.g., as was done in ITCS 2023), as long as there is a robust and efficient way to check conflicts of interest.
翻译:关于同行评审中是否对作者身份进行匿名化存在广泛争议。匿名化的核心论点是减少偏见,而反对匿名化的论点则主张在评审过程中利用作者身份的各种作用。2023年理论计算机科学创新会议(ITCS 2023)采取了一种折中方案:最初对审稿人匿名作者身份,在审稿人提交初步评审后揭晓作者身份,并允许审稿人随后修改其评审。我们分析了涉及作者身份识别和使用的评审数据。主要发现如下:(I) 大多数审稿人自述不了解且无法猜测所审论文的作者身份。(II) 在初步评审提交后,7.1%的评审改变了整体评分,3.8%的评审改变了自述审稿专业水平。(III) 作者所属机构的排名与整体评分变化之间存在非常微弱且统计不显著的相关性;但审稿专业水平变化与机构排名之间存在微弱但统计显著的相关性。我们还进行了一项匿名调查,以收集审稿人和作者的意见。基于200份调查回复的主要发现包括:(i) 绝大多数参与者支持以某种形式对作者身份进行匿名化。(ii) ITCS 2023的"折中方案"受到赞赏。(iii) 若对作者身份进行匿名化,检测利益冲突是一项需要解决的挑战。总体而言,这些发现支持以某种形式(如ITCS 2023的做法)对作者身份进行匿名化,前提是存在一种稳健且高效的方式来检查利益冲突。