Solver competitions play a prominent role in assessing and advancing the state of the art for solving many problems in AI and beyond. Notably, in many areas of AI, competitions have had substantial impact in guiding research and applications for many years, and for a solver to be ranked highly in a competition carries considerable weight. But to which extent can we expect competition results to generalise to sets of problem instances different from those used in a particular competition? This is the question we investigate here, using statistical resampling techniques. We show that the rankings resulting from the standard interpretation of competition results can be very sensitive to even minor changes in the benchmark instance set used as the basis for assessment and can therefore not be expected to carry over to other samples from the same underlying instance distribution. To address this problem, we introduce a novel approach to statistically meaningful analysis of competition results based on resampling performance data. Our approach produces confidence intervals of competition scores as well as statistically robust solver rankings with bounded error. Applied to recent SAT, AI planning and computer vision competitions, our analysis reveals frequent statistical ties in solver performance as well as some inversions of ranks compared to the official results based on simple scoring.

翻译：求解器竞赛在评估和推动人工智能及更广泛领域中许多问题的解决水平方面发挥着重要作用。值得注意的是，在人工智能的多个领域，竞赛多年来一直对指导研究和应用产生重大影响，而求解器在竞赛中获得高排名具有相当大的分量。但是，竞赛结果能在多大程度上推广到与特定竞赛中使用的不同的问题实例集？这是本文通过统计重采样技术研究的问题。我们表明，基于竞赛结果标准解释得出的排名，即使基准实例集发生微小变化也可能非常敏感，因此不能预期这些排名会延续到来自同一底层实例分布的其他样本。为了解决这个问题，我们引入了一种基于重采样性能数据对竞赛结果进行统计意义分析的新方法。我们的方法为竞赛得分提供置信区间，并生成具有有界误差的统计稳健的求解器排名。应用于最近的SAT、AI规划和计算机视觉竞赛，我们的分析揭示了求解器性能中频繁出现的统计平局，以及与基于简单评分的官方结果相比的一些排名倒置。