Peer review often involves reviewers submitting their independent reviews, followed by a discussion among reviewers of each paper. A question among policymakers is whether the reviewers of a paper should be anonymous to each other during the discussion. We shed light on this by conducting a randomized controlled trial at the UAI 2022 conference. We randomly split the reviewers and papers into two conditions--one with anonymous discussions and the other with non-anonymous discussions, and conduct an anonymous survey of all reviewers, to address the following questions: 1. Do reviewers discuss more in one of the conditions? Marginally more in anonymous (n = 2281, p = 0.051). 2. Does seniority have more influence on final decisions when non-anonymous? Yes, the decisions are closer to senior reviewers' scores in the non-anonymous condition than in anonymous (n = 484, p = 0.04). 3. Are reviewers more polite in one of the conditions? No significant difference in politeness of reviewers' text-based responses (n = 1125, p = 0.72). 4. Do reviewers' self-reported experiences differ across the two conditions? No significant difference for each of the five questions asked (n = 132 and p > 0.3). 5. Do reviewers prefer one condition over the other? Yes, there is a weak preference for anonymous discussions (n = 159 and Cohen's d= 0.25). 6. What do reviewers consider important to make policy on anonymity among reviewers? Reviewers' feeling of safety in expressing their opinions was rated most important, while polite communication among reviewers was rated least important (n = 159). 7. Have reviewers experienced dishonest behavior due to non-anonymity in discussions? Yes, roughly 7% of respondents answered affirmatively (n = 167). Overall, this experiment reveals evidence supporting an anonymous discussion setup in the peer-review process, in terms of the evaluation criteria considered.

翻译：同行评审通常包括审稿人提交独立评审意见，随后就每篇论文展开审稿人之间的讨论。政策制定者面临的一个问题是：审稿人在讨论过程中是否应相互匿名。我们通过在UAI 2022会议上进行随机对照试验来阐明这一问题。我们将审稿人和论文随机分为两组——一组采用匿名讨论，另一组采用非匿名讨论，并对所有审稿人进行匿名问卷调查，以回答以下问题：1. 审稿人在哪种条件下讨论更积极？匿名条件下略多（n=2281, p=0.051）。2. 非匿名条件下资历是否对最终决策影响更大？是的，非匿名条件下最终决策更接近资深审稿人的评分（n=484, p=0.04）。3. 审稿人在哪种条件下更有礼貌？审稿人文本回复的礼貌性无显著差异（n=1125, p=0.72）。4. 审稿人的自我报告体验在两种条件下是否存在差异？五个问题均无显著差异（n=132, p>0.3）。5. 审稿人是否偏好某种条件？是的，存在对匿名讨论的较弱偏好（n=159, Cohen's d=0.25）。6. 审稿人认为制定审稿人匿名政策需考虑哪些重要因素？审稿人表达观点的安全感被评为最重要，而审稿人之间的礼貌沟通被评为最不重要（n=159）。7. 审稿人是否因讨论非匿名性而经历过不诚实行为？是的，约7%的受访者给出肯定回答（n=167）。整体而言，本实验发现证据支持在同行评审过程中采用匿名讨论机制，基于所考虑的评估标准。