Large language model (LLM) benchmarks inform LLM use decisions (e.g., "is this LLM safe to deploy for my use case and context?"). However, benchmarks may be rendered unreliable by various failure modes that impact benchmark bias, variance, coverage, or people's capacity to understand benchmark evidence. Using the National Institute of Standards and Technology's risk management process as a foundation, this research iteratively analyzed 26 popular benchmarks, identifying 57 potential failure modes and 196 corresponding mitigation strategies. The mitigations reduce failure likelihood and/or severity, providing a frame for evaluating "benchmark risk," which is scored to provide a metaevaluation benchmark: BenchRisk. Higher scores indicate that benchmark users are less likely to reach an incorrect or unsupported conclusion about an LLM. All 26 scored benchmarks present significant risk within one or more of the five scored dimensions (comprehensiveness, intelligibility, consistency, correctness, and longevity), which points to important open research directions for the field of LLM benchmarking. The BenchRisk workflow allows for comparison between benchmarks; as an open-source tool, it also facilitates the identification and sharing of risks and their mitigations.

翻译：大型语言模型（LLM）基准测试为LLM使用决策提供依据（例如“该LLM是否可安全部署于我的用例和场景中？”）。然而，基准测试可能因多种失效模式而变得不可靠，这些失效模式会影响基准测试的偏差、方差、覆盖范围或人们理解基准测试证据的能力。本研究以美国国家标准与技术研究院的风险管理流程为基础，通过迭代分析26个主流基准测试，识别出57种潜在失效模式和196项对应缓解策略。这些缓解措施通过降低失效可能性及/或严重程度，构建了评估“基准测试风险”的框架，该风险经量化评分后形成元评估基准：BenchRisk。更高的评分表明基准测试使用者对LLM得出错误或缺乏依据结论的可能性更低。所有26个已评分基准测试均在五个评估维度（全面性、可理解性、一致性、正确性、持久性）中的一个或多个维度存在显著风险，这为LLM基准测试领域指出了重要的开放研究方向。BenchRisk工作流程支持不同基准测试间的比较分析；作为开源工具，它还有助于识别和共享风险及其缓解措施。