Post-training quantization (\ptq) had been recently shown as a compromising method to reduce the memory consumption and/or compute cost for large language models. However, a comprehensive study about the effect of different quantization schemes, different model families, different \ptq methods, different quantization bit precision, etc, is still missing. In this work, we provide an extensive study on those components over tens of thousands of zero-shot experiments. Our results show that (1) Fine-grained quantization and \ptq methods (instead of naive round-to-nearest quantization) are necessary to achieve good accuracy and (2) Higher bits (e.g., 5 bits) with coarse-grained quantization is more powerful than lower bits (e.g., 4 bits) with very fine-grained quantization (whose effective bits is similar to 5-bits). We also present recommendations about how to utilize quantization for \llms with different sizes, and leave suggestions of future opportunities and system work that are not resolved in this work.

翻译：后训练量化（\ptq）最近被证明是一种降低大型语言模型内存消耗和/或计算成本的有效折衷方法。然而，关于不同量化方案、不同模型家族、不同\ptq方法、不同量化比特精度等因素影响的综合性研究仍然缺失。在本工作中，我们针对这些组件在数万次零样本实验中进行了广泛研究。结果表明：（1）细粒度量化和\ptq方法（而非简单的最近邻取整量化）对于实现良好精度是必要的；（2）采用粗粒度量化的高位宽（如5比特）比采用极细粒度量化的低位宽（如4比特，其有效比特与5比特相似）更具优势。我们还针对不同规模的大型语言模型（\llms）如何利用量化提出了建议，并指出了本工作尚未解决但值得未来机遇与系统研究的方向。