Retrieval-augmented generation (RAG) has become a de facto paradigm in various large language model (LLM)-driven applications on mobile devices, such as mobile assistants leveraging personal emails or meeting records. However, due to lengthy prompts and resource constraints, mobile RAG systems exhibit significantly high response latency. To address this issue, one promising approach is to reuse intermediate computational results across different queries to eliminate redundant computation. But most existing approaches, such as key-value (KV) cache reuse and semantic cache reuse, are designed for cloud settings and perform poorly on mobile devices as they overlook the distinctive characteristics of mobile RAG. We propose PerCache, a novel hierarchical cache framework designed for reducing end-to-end latency of personalized RAG applications on mobile platforms. PerCache adopts a hierarchical architecture that progressively matches similar queries and query-key-value (QKV) cache to maximize the reuse of intermediate results at different computing stages. To improve cache hit rate, PerCache applies a predictive method to populate the cache with queries that are likely to be raised in the future. In addition, PerCache can adapt its configurations to dynamic system loads, aiming to maximize caching utility with constrained resource consumption. We implement PerCache on top of an existing mobile LLM inference engine and deploy it on commodity mobile phones. Extensive evaluations show that PerCache achieves the lowest average end-to-end latency across five datasets and two LLM architectures, reducing latency by up to 63.5% over the strongest baseline while maintaining comparable answer quality.

翻译：暂无翻译