Key-value (KV) caching has become the de-facto to accelerate generation speed for large language models (LLMs) inference. However, the growing cache demand with increasing sequence length has transformed LLM inference to be a memory bound problem, significantly constraining the system throughput. Existing methods rely on dropping unimportant tokens or quantizing all entries uniformly. Such methods, however, often incur high approximation errors to represent the compressed matrices. The autoregressive decoding process further compounds the error of each step, resulting in critical deviation in model generation and deterioration of performance. To tackle this challenge, we propose GEAR, an efficient KV cache compression framework that achieves near-lossless high-ratio compression. GEAR first applies quantization to majority of entries of similar magnitudes to ultra-low precision. It then employs a low rank matrix to approximate the quantization error, and a sparse matrix to remedy individual errors from outlier entries. By adeptly integrating three techniques, GEAR is able to fully exploit their synergistic potentials. Our experiments demonstrate that compared to alternatives, GEAR achieves near-lossless 4-bit KV cache compression with up to 2.38x throughput improvement, while reducing peak-memory size up to 2.29x. Our code is publicly available at https://github.com/HaoKang-Timmy/GEAR.

翻译：键值（KV）缓存已成为加速大语言模型（LLM）推理生成速度的事实标准。然而，随着序列长度增加带来的缓存需求增长，将LLM推理转变为内存受限问题，严重制约了系统吞吐量。现有方法依赖丢弃不重要词元或对所有条目进行统一量化。然而这类方法在表示压缩矩阵时往往引入较高近似误差。自回归解码过程进一步加剧了每一步的误差累积，导致模型生成出现关键偏差并致使性能退化。为应对这一挑战，我们提出GEAR——一种实现近无损高比率压缩的高效KV缓存压缩框架。GEAR首先将大部分幅度相似的条目量化至极低精度，随后采用低秩矩阵近似量化误差，并利用稀疏矩阵修正异常条目带来的个体误差。通过巧妙融合三种技术，GEAR能够充分释放其协同潜力。实验表明，相比现有方案，GEAR在实现近无损4比特KV缓存压缩的同时，吞吐量提升达2.38倍，峰值内存占用降低达2.29倍。我们的代码已开源至https://github.com/HaoKang-Timmy/GEAR。