Generative large language models (LLMs) have revolutionized multiple domains. Modern LLMs predominantly rely on an autoregressive decoding strategy, which generates output tokens sequentially and employs a key-value cache (KV cache) to avoid redundant computation. However, the widespread deployment of LLMs has raised serious privacy concerns, as users are feeding all types of data into the model, motivating the development of secure inference frameworks based on fully homomorphic encryption (FHE). A major limitation of existing FHE-based frameworks is their inability to effectively integrate the KV cache, resulting in prohibitively high latency for autoregressive decoding. In this paper, we propose Cachemir, a KV Cache Accelerated Homomorphic Encrypted LLM Inference Regime to overcome this limitation. Cachemir comprises three key technical contributions: 1) a set of novel HE packing algorithms specifically designed to leverage the computational advantages of the KV cache; 2) an interleaved replicated packing algorithm to efficiently compute the vector-matrix multiplications that result from using the KV cache in Transformer linear layers; and 3) an augmented bootstrapping placement strategy that accounts for the KV cache to minimize bootstrapping cost. We demonstrate that Cachemir achieves $48.83\times$ and $67.16\times$ speedup over MOAI (ICML'25) and THOR (CCS'25) respectively on CPU and consumes less than 100 seconds on GPU to generate an output token for Llama-3-8B.

翻译：生成式大语言模型（LLM）已在多个领域引发革命性变革。现代LLM主要依赖自回归解码策略，该策略顺序生成输出令牌，并采用键值缓存（KV cache）以避免冗余计算。然而，LLM的广泛部署引发了严重的隐私担忧，因为用户将各类数据输入模型，这推动了基于全同态加密（FHE）的安全推理框架的发展。现有基于FHE的框架主要局限在于无法有效集成KV缓存，导致自回归解码延迟极高。本文提出Cachemir，一种KV缓存加速的同态加密LLM推理机制，以克服此限制。Cachemir包含三项关键技术贡献：1）一套专为利用KV缓存计算优势而设计的新型HE打包算法；2）一种交错复制打包算法，用于高效计算Transformer线性层中因使用KV缓存而产生的向量-矩阵乘法；3）一种考虑KV缓存的增强型自举放置策略，以最小化自举开销。实验表明，Cachemir在CPU上分别比MOAI（ICML'25）和THOR（CCS'25）实现了48.83倍和67.16倍的加速，在GPU上为Llama-3-8B生成单个输出令牌的耗时低于100秒。