When Large Language Models (LLMs) are compressed using techniques such as quantization, the predominant way to demonstrate the validity of such techniques is by measuring the model's accuracy on various benchmarks.If the accuracies of the baseline model and the compressed model are close, it is assumed that there was negligible degradation in quality.However, even when the accuracy of baseline and compressed model are similar, we observe the phenomenon of flips, wherein answers change from correct to incorrect and vice versa in proportion.We conduct a detailed study of metrics across multiple compression techniques, models and datasets, demonstrating that the behavior of compressed models as visible to end-users is often significantly different from the baseline model, even when accuracy is similar.We further evaluate compressed models qualitatively and quantitatively using MT-Bench and show that compressed models are significantly worse than baseline models in this free-form generative task.Thus, we argue that compression techniques should also be evaluated using distance metrics.We propose two such metrics, KL-Divergence and flips, and show that they are well correlated.

翻译：当大型语言模型（LLMs）通过量化等技术进行压缩时，验证此类技术有效性的主流方法是测量模型在各种基准测试中的准确率。若基线模型与压缩模型的准确率相近，通常认为其质量下降可忽略不计。然而，即使基线模型与压缩模型的准确率相似，我们仍观察到答案翻转现象——即正确答案转为错误答案与错误答案转为正确答案的比例变化。我们通过对多种压缩技术、模型及数据集的指标进行详细研究，证明即使准确率相近，压缩模型在终端用户可见的行为表现往往与基线模型存在显著差异。我们进一步使用MT-Bench对压缩模型进行定性与定量评估，结果表明在自由生成任务中，压缩模型的性能明显劣于基线模型。因此，我们认为压缩技术的评估还应引入距离度量指标。我们提出两种此类指标——KL散度与翻转率，并证明二者具有良好相关性。