If machine learning models were to achieve superhuman abilities at various reasoning or decision-making tasks, how would we go about evaluating such models, given that humans would necessarily be poor proxies for ground truth? In this paper, we propose a framework for evaluating superhuman models via consistency checks. Our premise is that while the correctness of superhuman decisions may be impossible to evaluate, we can still surface mistakes if the model's decisions fail to satisfy certain logical, human-interpretable rules. We instantiate our framework on three tasks where correctness of decisions is hard to evaluate due to either superhuman model abilities, or to otherwise missing ground truth: evaluating chess positions, forecasting future events, and making legal judgments. We show that regardless of a model's (possibly superhuman) performance on these tasks, we can discover logical inconsistencies in decision making. For example: a chess engine assigning opposing valuations to semantically identical boards; GPT-4 forecasting that sports records will evolve non-monotonically over time; or an AI judge assigning bail to a defendant only after we add a felony to their criminal record.

翻译：如果机器学习模型在各种推理或决策任务中达到超人类能力，鉴于人类必然难以作为真实的基准，我们该如何评估这类模型？本文提出了一种通过一致性检验来评估超人类模型的框架。其核心前提是：尽管超人类决策的正确性可能无法直接评估，但当模型的决策未能满足某些符合逻辑、人类可解释的规则时，我们仍能发现其错误。我们将该框架应用于三个任务中，在这些任务中，由于模型能力超人类或缺乏真实基准，决策的正确性难以评估：评估棋局位置、预测未来事件以及做出法律判决。我们证明，无论模型在这些任务上表现如何（可能超人类），都能发现其决策中的逻辑不一致性。例如：国际象棋引擎对语义相同的棋局赋予相反估值；GPT-4预测体育纪录将随时间非单调演进；或AI法官仅在被告犯罪记录中添加重罪后才准予保释。