We present the NeurIPS 2021 consistency experiment, a larger-scale variant of the 2014 NeurIPS experiment in which 10% of conference submissions were reviewed by two independent committees to quantify the randomness in the review process. We observe that the two committees disagree on their accept/reject recommendations for 23% of the papers and that, consistent with the results from 2014, approximately half of the list of accepted papers would change if the review process were randomly rerun. Our analysis suggests that making the conference more selective would increase the arbitrariness of the process. Taken together with previous research, our results highlight the inherent difficulty of objectively measuring the quality of research, and suggest that authors should not be excessively discouraged by rejected work.

翻译：我们开展了 NeurIPS 2021 一致性实验，这是 2014 年 NeurIPS 实验的大规模版本。实验中，10% 的会议投稿由两个独立的评审委员会进行评审，以量化审稿过程中的随机性。我们观察到，两个委员会对 23% 的论文在接受/拒绝建议上存在分歧，并且与 2014 年的结果一致，如果重新随机进行审稿过程，已接受的论文列表中大约有一半会发生变化。我们的分析表明，提高会议的选择性会增加过程的随意性。结合先前的研究，我们的结果突显了客观衡量研究质量的固有困难，并建议作者不应因被拒稿而过度气馁。