Benchmarking is the de-facto standard for evaluating LLMs, due to its speed, replicability and low cost. However, recent work has pointed out that the majority of the open source benchmarks available today have been contaminated or leaked into LLMs, meaning that LLMs have access to test data during pretraining and/or fine-tuning. This raises serious concerns about the validity of benchmarking studies conducted so far and the future of evaluation using benchmarks. To solve this problem, we propose Private Benchmarking, a solution where test datasets are kept private and models are evaluated without revealing the test data to the model. We describe various scenarios (depending on the trust placed on model owners or dataset owners), and present solutions to avoid data contamination using private benchmarking. For scenarios where the model weights need to be kept private, we describe solutions from confidential computing and cryptography that can aid in private benchmarking. Finally, we present solutions the problem of benchmark dataset auditing, to ensure that private benchmarks are of sufficiently high quality.

翻译：基准测试因其快速性、可重复性和低成本，已成为评估大语言模型的通用标准。然而近期研究表明，当前可用的开源基准测试数据集大多已受到污染或泄露，导致模型在预训练和/或微调阶段即可接触测试数据。这严重动摇了现有基准测试研究的有效性，并对基于基准测试的未来评估体系构成根本性挑战。为解决该问题，我们提出私有基准测试方案——通过将测试数据集保持私有状态，在模型无法获取测试数据的前提下完成评估。根据对模型所有者与数据集所有者的信任程度差异，我们描述了多种应用场景，并提出利用私有基准测试避免数据污染的解决方案。针对需要保护模型权重的场景，我们介绍了可信计算和密码学领域中适用于私有基准测试的技术方案。最后，我们提出基准测试数据集审计方案，以确保私有基准测试具备足够高的质量水准。