Multiple-choice (MCQA) benchmarks are the standard for evaluating pretrained large language models, but their reliance on log-likelihood scoring makes them unreliable. Specifically, standard scores are highly sensitive to the exact phrasing (surface form) of the answers, conflating a model's familiarity with a specific phrase with its actual capability. We demonstrate this flaw using a controlled testbed of 1B-8B models trained on the same knowledge. Despite having identical knowledge, standard metrics falsely report a performance gap of over 2 points. To solve this, we propose ParaEval, an evaluation framework that queries models using multiple paraphrases per answer option. By scoring each model based on its most favorable phrasing, ParaEval successfully reduces the false performance gap to below 1 point. We confirm that these evaluation artifacts, and the improvements from ParaEval, persist in frontier 70B and 120B open-source models. Ultimately, ParaEval provides a robust and efficient way to evaluate true underlying capability rather than surface-form familiarity.

翻译：多项选择问答（MCQA）基准是评估预训练大语言模型的标准方法，但此类基准对对数似然评分的依赖使其结果不可靠。具体而言，标准评分对答案的具体措辞（表层形式）高度敏感，混淆了模型对特定短语的熟悉程度与其实际能力。我们通过在相同知识上训练的1B-8B模型组成的受控测试集上，证实了这一缺陷——尽管模型拥有等同的知识，但标准指标错误地报告了超过2个百分点的性能差距。为解决该问题，我们提出ParaEval评估框架：对每个答案选项使用多个释义变体查询模型。通过基于模型最有利的措辞进行评分，ParaEval成功将虚假性能差距缩小至1个百分点以下。我们证实这类评估伪影以及ParaEval的改进效果，在前沿的70B和120B开源模型中依然存在。最终，ParaEval提供了一种稳健高效的方法来评估模型的真实底层能力，而非其对表层形式的熟悉度。