LLM-based agent applications have shown increasingly remarkable capabilities in complex workflows but incur substantial costs and latency due to extensive planning and reasoning requirements. Existing LLM caching techniques (like context caching and semantic caching), primarily designed for serving chatbots, are insufficient for agent applications where outputs depend on external data and environmental contexts. We propose Agentic Plan Caching (APC), a novel test-time memory that extracts, stores, adapts, and reuses structured plan templates from planning stages of agent applications across semantically similar tasks to reduce the cost and latency of serving. Unlike traditional semantic caching, our system extracts plan templates from completed agent executions at test-time, employs keyword extraction to match new requests against cached plans, and utilizes lightweight models to adapt these templates to task-specific plans with contexts. Evaluation across multiple real-world agent applications shows that our system can reduce costs by 50.31% and latency by 27.28% on average while maintaining performance, offering a more efficient solution for serving LLM-based agents that complements existing LLM serving infrastructures.

翻译：基于大语言模型（LLM）的智能体应用在复杂工作流中展现出日益显著的能力，但由于其广泛的规划与推理需求，往往伴随着高昂的成本与延迟。现有的LLM缓存技术（如上下文缓存与语义缓存）主要面向聊天机器人服务设计，对于输出依赖外部数据与环境上下文的智能体应用而言存在不足。本文提出智能体规划缓存（Agentic Plan Caching, APC），一种新颖的测试时记忆机制，通过从智能体应用的规划阶段提取、存储、适配并复用结构化规划模板，跨语义相似任务实现服务成本与延迟的降低。与传统语义缓存不同，本系统在测试时从已完成的智能体执行过程中提取规划模板，采用关键词提取技术将新请求与缓存规划进行匹配，并利用轻量级模型结合具体上下文将这些模板适配为任务专属规划。在多个现实场景智能体应用上的评估表明，本系统在保持性能的同时，平均可降低50.31%的成本与27.28%的延迟，为基于LLM的智能体服务提供了更高效的解决方案，并与现有LLM服务基础设施形成互补。