The rapid proliferation of benchmarks for evaluating large language models (LLMs) has created an urgent need for systematic methods to assess benchmark quality itself. We propose Benchmark^2, a comprehensive framework comprising three complementary metrics: (1) Cross-Benchmark Ranking Consistency, measuring whether a benchmark produces model rankings aligned with peer benchmarks; (2) Discriminability Score, quantifying a benchmark's ability to differentiate between models; and (3) Capability Alignment Deviation, identifying problematic instances where stronger models fail but weaker models succeed within the same model family. We conduct extensive experiments across 15 benchmarks spanning mathematics, reasoning, and knowledge domains, evaluating 11 LLMs across four model families. Our analysis reveals significant quality variations among existing benchmarks and demonstrates that selective benchmark construction based on our metrics can achieve comparable evaluation performance with substantially reduced test sets.

翻译：大型语言模型（LLM）评估基准的快速激增，催生了对系统性评估基准本身质量方法的迫切需求。我们提出了Benchmark^2，一个包含三个互补指标的综合性框架：（1）跨基准排名一致性，用于衡量一个基准产生的模型排名是否与同类基准一致；（2）区分度分数，用于量化一个基准区分不同模型的能力；（3）能力对齐偏差，用于识别同一模型家族内更强模型失败而更弱模型成功的异常实例。我们在涵盖数学、推理和知识领域的15个基准上进行了广泛的实验，评估了来自四个模型家族的11个LLM。我们的分析揭示了现有基准之间存在显著的质量差异，并证明了基于我们的指标进行选择性基准构建，能够以大幅缩减的测试集实现可比的评估性能。