Post-Training Quantization (PTQ) enhances the efficiency of Large Language Models (LLMs) by enabling faster operation and compatibility with more accessible hardware through reduced memory usage, at the cost of small performance drops. We explore the role of calibration sets in PTQ, specifically their effect on hidden activations in various notable open-source LLMs. Calibration sets are crucial for evaluating activation magnitudes and identifying outliers, which can distort the quantization range and negatively impact performance. Our analysis reveals a marked contrast in quantization effectiveness across models. The older OPT model, which much of the quantization literature is based on, shows significant performance deterioration and high susceptibility to outliers with varying calibration sets. In contrast, newer models like Llama-2 7B, Llama-3 8B, Command-R 35B, and Mistral 7B demonstrate strong robustness, with Mistral 7B showing near-immunity to outliers and stable activations. These findings suggest a shift in PTQ strategies might be needed. As advancements in pre-training methods reduce the relevance of outliers, there is an emerging need to reassess the fundamentals of current quantization literature. The emphasis should pivot towards optimizing inference speed, rather than primarily focusing on outlier preservation, to align with the evolving characteristics of state-of-the-art LLMs.

翻译：训练后量化通过降低内存占用，使大语言模型能够以更快的速度运行并兼容更易获取的硬件，从而提升其效率，但会带来轻微的性能损失。本文探讨了校准集在训练后量化中的作用，特别是其对各种知名开源大语言模型中隐藏激活的影响。校准集对于评估激活幅度和识别离群值至关重要，这些离群值可能扭曲量化范围并对性能产生负面影响。我们的分析揭示了不同模型间量化效果的显著差异。较旧的OPT模型（当前大量量化文献的基础）表现出明显的性能下降，并且对校准集的变化及离群值高度敏感。相比之下，Llama-2 7B、Llama-3 8B、Command-R 35B和Mistral 7B等新模型展现出强大的鲁棒性，其中Mistral 7B几乎不受离群值影响，且激活稳定。这些发现表明，可能需要调整训练后量化策略。随着预训练方法的进步降低了离群值的相关性，重新评估当前量化文献的基础原理的需求日益凸显。重点应转向优化推理速度，而非主要关注离群值保留，以适应当前最先进大语言模型不断演变的特性。