Despite the remarkable strides of Large Language Models (LLMs) in various fields, the wide applications of LLMs on edge devices are limited due to their massive parameters and computations. To address this, quantization is commonly adopted to generate lightweight LLMs with efficient computations and fast inference. However, Post-Training Quantization (PTQ) methods dramatically degrade in quality when quantizing weights, activations, and KV cache together to below 8 bits. Besides, many Quantization-Aware Training (QAT) works quantize model weights, leaving the activations untouched, which do not fully exploit the potential of quantization for inference acceleration on the edge. In this paper, we propose EdgeQAT, the Entropy and Distribution Guided QAT for the optimization of lightweight LLMs to achieve inference acceleration on Edge devices. We first identify that the performance drop of quantization primarily stems from the information distortion in quantized attention maps, demonstrated by the different distributions in quantized query and key of the self-attention mechanism. Then, the entropy and distribution guided QAT is proposed to mitigate the information distortion. Moreover, we design a token importance-aware adaptive method to dynamically quantize the tokens with different bit widths for further optimization and acceleration. Our extensive experiments verify the substantial improvements with our framework across various datasets. Furthermore, we achieve an on-device speedup of up to 2.37x compared with its FP16 counterparts across multiple edge devices, signaling a groundbreaking advancement.

翻译：尽管大语言模型（LLMs）在各领域取得了显著进展，但由于其庞大的参数量和计算量，其在边缘设备上的广泛应用仍受到限制。为解决这一问题，通常采用量化方法生成具有高效计算和快速推理能力的轻量级LLMs。然而，当权重、激活值和KV缓存同时量化至低于8比特时，训练后量化（PTQ）方法的质量会大幅下降。此外，许多量化感知训练（QAT）工作仅量化模型权重而保留激活值不变，未能充分挖掘量化在边缘设备推理加速中的潜力。本文提出EdgeQAT——一种面向边缘设备轻量级LLM推理加速优化的熵与分布引导QAT方法。我们首先发现量化性能下降主要源于量化注意力图中的信息失真，这通过自注意力机制中量化查询和键的不同分布得到验证。随后提出熵与分布引导的QAT来减轻这种信息失真。此外，我们设计了一种令牌重要性感知自适应方法，通过动态量化不同位宽的令牌实现进一步优化和加速。大量实验验证了本文框架在各数据集上的显著提升效果。最终，与FP16基线相比，我们在多个边缘设备上实现了高达2.37倍的设备端加速，标志着突破性进展。