Since local LLM inference on resource-constrained edge devices imposes a severe performance bottleneck, this paper proposes distributed prompt caching to enhance inference performance by cooperatively sharing intermediate processing states across multiple low-end edge devices. To fully utilize prompt similarity, our distributed caching mechanism also supports partial matching. As this approach introduces communication overhead associated with state sharing over a wireless network, we introduce a Bloom-filter-based data structure, referred to as a catalog, to determine whether a remote server possesses the desired internal states, thereby suppressing unnecessary communication. Experiments using the Gemma-3 270M model and the MMLU dataset on the Raspberry Pi Zero 2W platform demonstrate that the proposed approach reduces TTFT (Time to First Token) and TTLT (Time to Last Token) by 93.12% and 50.07% on average, respectively.

翻译：由于在资源受限的边缘设备上运行本地大语言模型推理会带来严重的性能瓶颈，本文提出分布式提示缓存技术，通过让多个低端边缘设备协同共享中间处理状态来提升推理性能。为充分利用提示相似性，我们的分布式缓存机制还支持部分匹配。由于该方法引入了通过无线网络进行状态共享所带来的通信开销，我们提出一种基于布隆过滤器的数据结构（称为目录），用于判断远程服务器是否拥有所需的内部状态，从而抑制不必要的通信。在Raspberry Pi Zero 2W平台上使用Gemma-3 270M模型和MMLU数据集进行的实验表明，所提方法平均将首令牌生成时间（TTFT）和尾令牌生成时间（TTLT）分别降低了93.12%和50.07%。