Peer selection, the evaluation and selection of agents by their peers, is an important problem in the field of computational social choice; with applications to grading in massively online courses (MOOCs) and academic peer review. Current existing algorithmic and empirical work focuses on developing and analyzing novel \emph{strategyproof} mechanisms, wherein no agent has an incentive to misreport their evaluations. However, the majority of published mechanisms share a flaw: they do not \emph{reward} agents for any effort expended during the evaluation process. In cases where high quality evaluations are costly to produce this missing incentive fails to align agents with an overall goal of accurate selection. To address this gap we first prove theoretically that incentivizing effort in peer selection requires information beyond a single evaluation. We then propose \textsc{PeerBTS}, a mechanism that combines a peer-prediction lottery, leveraging work on the Robust Bayesian Truth Serum, with any existing peer-selection mechanism to incentivize effort while remaining Bayes-Nash incentive compatible. We find that while an incentive-compatible peer-selection mechanism using agent predictions to incentivize effort is possible it requires adjustments to the assumed problem context and limits other mechanistics properties. We additionally present a series of non-strategic simulations to validate incentives and evaluate the performance of PeerBTS relative to existing strategyproof peer selection mechanisms. Finally, we discuss the results of an initial study on the validity of peer-prediction from a small academic workshop.

翻译：同伴选择，即由同行进行评估和选择，是计算社会选择领域中的一个重要问题；其应用包括大规模在线课程（MOOC）中的评分和学术同行评审。当前已有的算法和实证研究侧重于开发和设计新型防策略机制，在这种机制下，没有任何代理有动机虚报其评估结果。然而，大多数已发表的机制存在一个缺陷：它们不会因代理在评估过程中付出的任何努力而奖励代理。在高质量评估成本高昂的情况下，这种缺失的激励无法使代理的行为与准确选择这一总体目标保持一致。为了解决这一差距，我们首先从理论上证明，在同伴选择中激励努力需要超出单一评估的信息。然后，我们提出PeerBTS，这是一种结合了同伴预测彩票（借鉴了Robust Bayesian Truth Serum的研究成果）与任何现有同伴选择机制的机制，旨在激励努力，同时保持贝叶斯-纳什激励相容性。我们发现，虽然使用代理预测来激励努力的激励相容的同伴选择机制是可能的，但它需要对假定的问题背景进行调整，并会限制其他机制特性。此外，我们进行了一系列非策略模拟，以验证激励效果并评估PeerBTS相对于现有防策略同伴选择机制的性能。最后，我们讨论了一项关于从一个小型学术研讨会验证同伴预测有效性的初步研究结果。