To reduce LLM costs and latency, semantic caching systems must accurately identify when a new prompt matches a cached one. Current methods often rely on simplistic similarity measures, which limit their effectiveness. We introduce MVR-cache, a novel semantic caching approach that significantly improves retrieval accuracy by integrating Multi-Vector Retrieval (MVR). MVR-cache is built upon a learnable segmentation model that intelligently splits prompts, enabling fine-grained similarity comparisons via MaxSim. We derive the model's training objective from a rigorous theoretical analysis. This can ensure that optimizing this objective directly maximizes cache hits under strict correctness constraints. To solve the resulting non-differentiable combinatorial optimization problem, we leverage a reinforcement learning-based training strategy with the theoretically grounded objectives as the reward. Experimental results on established benchmarks across diverse tasks confirm that in comparison to the state-of-the-art, MVR-cache consistently increases the cache hit rates by up to 37% while maintaining the same correctness guarantees. MVR-cache is available at https://github.com/PKU-SDS-lab/MVR-Cache

翻译：为降低大语言模型的使用成本与延迟，语义缓存系统需准确识别新提示是否与缓存中的提示匹配。当前方法通常依赖简单的相似度度量，限制了其有效性。我们提出MVR-cache，一种通过集成多向量检索（MVR）显著提升检索准确率的新型语义缓存方法。MVR-cache基于可学习的分割模型构建，该模型能智能地划分提示，并通过MaxSim实现细粒度相似度比较。我们从严谨的理论分析出发推导模型训练目标，确保优化该目标可直接在严格正确性约束下最大化缓存命中率。为求解由此产生的不可微组合优化问题，我们采用基于强化学习的训练策略，将理论推导的目标函数作为奖励函数。在跨多种任务的权威基准测试中，实验结果表明：与现有最优方法相比，MVR-cache在保持相同正确性保证的前提下，缓存命中率最高可提升37%。MVR-cache开源代码见 https://github.com/PKU-SDS-lab/MVR-Cache