In 2023, the International Conference on Machine Learning (ICML) required authors with multiple submissions to rank their submissions based on perceived quality. In this paper, we aim to employ these author-specified rankings to enhance peer review in machine learning and artificial intelligence conferences by extending the Isotonic Mechanism to exponential family distributions. This mechanism generates adjusted scores that closely align with the original scores while adhering to author-specified rankings. Despite its applicability to a broad spectrum of exponential family distributions, implementing this mechanism does not require knowledge of the specific distribution form. We demonstrate that an author is incentivized to provide accurate rankings when her utility takes the form of a convex additive function of the adjusted review scores. For a certain subclass of exponential family distributions, we prove that the author reports truthfully only if the question involves only pairwise comparisons between her submissions, thus indicating the optimality of ranking in truthful information elicitation. Moreover, we show that the adjusted scores improve dramatically the estimation accuracy compared to the original scores and achieve nearly minimax optimality when the ground-truth scores have bounded total variation. We conclude with a numerical analysis of the ICML 2023 ranking data, showing substantial estimation gains in approximating a proxy ground-truth quality of the papers using the Isotonic Mechanism.

翻译：2023年，国际机器学习大会（ICML）要求有多篇投稿的作者根据其感知的质量对投稿进行排序。本文旨在通过将保序机制推广至指数族分布，利用这些作者指定的排序来改进机器学习与人工智能会议的同行评审。该机制生成与原始分数高度一致且遵循作者指定排序的调整分数。尽管该机制适用于广泛的指数族分布，其实施并不需要了解具体的分布形式。我们证明，当作者的效用函数为调整后评审分数的凸可加函数时，作者有动机提供准确的排序。对于某一特定子类的指数族分布，我们证明作者仅当问题仅涉及其投稿间的两两比较时才会如实报告，从而表明排序在真实信息获取中的最优性。此外，我们证明与原始分数相比，调整分数显著提高了估计精度，并且当真实分数具有有界全变差时，调整分数几乎达到极小极大最优性。最后，我们对ICML 2023排序数据进行了数值分析，结果表明使用保序机制在近似论文代理真实质量方面获得了显著的估计效益。