Minimizing the inference cost and latency of foundation models has become a crucial area of research. Optimization approaches include theoretically lossless methods and others without accuracy guarantees like quantization. In all of these cases it is crucial to ensure that the model quality has not degraded. However, even at temperature zero, model generations are not necessarily robust even to theoretically lossless model optimizations due to numerical errors. We thus require statistical tools to decide whether a finite-sample accuracy deviation is an evidence of a model's degradation or whether it can be attributed to (harmless) noise in the evaluation. We propose a statistically sound hypothesis testing framework based on McNemar's test allowing to efficiently detect model degradations, while guaranteeing a controlled rate of false positives. The crucial insight is that we have to confront the model scores on each sample, rather than aggregated on the task level. Furthermore, we propose three approaches to aggregate accuracy estimates across multiple benchmarks into a single decision. We provide an implementation on top of the largely adopted open source LM Evaluation Harness and provide a case study illustrating that the method correctly flags degraded models, while not flagging model optimizations that are provably lossless. We find that with our tests even empirical accuracy degradations of 0.3% can be confidently attributed to actual degradations rather than noise.

翻译：降低基础模型的推理成本与延迟已成为关键研究领域。优化方法包括理论无损方法以及量化等无精度保证技术。在所有情形下，确保模型质量未退化至关重要。然而，即使温度参数为零，由于数值误差，模型生成结果对理论无损的模型优化也未必稳健。因此，我们需要统计工具来判断有限样本下的精度偏差是模型退化的证据，还是可归因于评估中的（无害）噪声。我们提出了一种基于McNemar检验的统计严谨假设检验框架，该框架能在保证可控假阳性率的同时高效检测模型退化。关键洞见在于必须针对每个样本比照模型得分，而非在任务层面聚合。此外，我们提出了三种跨多个基准聚合精度估计以形成单一决策的方法。我们在广泛采用的开源LM Evaluation Harness基础上实现了该框架，并通过案例研究表明该方法能正确标记退化模型，同时不标记可证明无损的模型优化。实验发现，采用我们的检验方法，即使0.3%的经验精度退化也能被可靠地归因于实际退化而非噪声。