Low-precision fine-tuning of language models has gained prominence as a cost-effective and energy-efficient approach to deploying large-scale models in various applications. However, this approach is susceptible to the existence of outlier values in activation. The outlier values in the activation can negatively affect the performance of fine-tuning language models in the low-precision regime since they affect the scaling factor and thus make representing smaller values harder. This paper investigates techniques for mitigating outlier activation in low-precision integer fine-tuning of the language models. Our proposed novel approach enables us to represent the outlier activation values in 8-bit integers instead of floating-point (FP16) values. The benefit of using integers for outlier values is that it enables us to use operator tiling to avoid performing 16-bit integer matrix multiplication to address this problem effectively. We provide theoretical analysis and supporting experiments to demonstrate the effectiveness of our approach in improving the robustness and performance of low-precision fine-tuned language models.

翻译：语言模型的低精度微调作为一种在各类应用中部署大规模模型的成本效益高且能效优异的方案，已日益受到关注。然而，该方法容易受到激活值中异常值存在的影响。激活值中的异常值可能对低精度条件下语言模型微调的性能产生负面影响，因为这些异常值会改变缩放因子，从而使得较小值的表示更加困难。本文研究了在语言模型低精度整数微调中缓解异常激活的技术。我们提出的新型方法能够将异常激活值表示为8位整数而非浮点（FP16）数值。采用整数表示异常值的优势在于，它使我们能够利用算子分块技术来避免执行16位整数矩阵乘法，从而有效解决这一问题。我们提供了理论分析与支持性实验，证明了该方法在提升低精度微调语言模型鲁棒性与性能方面的有效性。