Running Large Language Models (LLMs) on edge devices is crucial for reducing latency, improving real-time processing, and enhancing privacy. By performing inference directly on the device, data does not need to be sent to the cloud, ensuring faster responses and reducing reliance on network connectivity. However, implementing LLMs on edge devices presents challenges, particularly with managing key-value (KV) caches, which plays a pivotal role in LLM serving. As the input text lengthens, the size of the KV cache increases linearly with the sequence length, leading to a significant memory footprint and data access costs. On the other hand, edge devices have limited memory and computational power, making it hard to store and efficiently access the large caches needed for LLM inference. To mitigate the substantial overhead caused by KV cache, we propose using embedded DRAM (eDRAM) as the primary storage for LLM serving in edge device, which offers higher storage density compared to SRAM. However, to ensure data integrity, eDRAM needs periodic refresh operations, which are power-intensive. To reduce eDRAM costs and improve overall system performance, we propose~\textit{Kelle}, a software-hardware co-design solution optimized for deploying LLMs on eDRAM-based edge systems. Combined with our fine-grained memory eviction, recomputation, and refresh control algorithms, the \textit{Kelle} accelerator delivers a $3.9\times$ speedup and $4.5\times$ energy savings compared to existing baseline solutions.

翻译：在边缘设备上运行大语言模型对于降低延迟、提升实时处理能力及增强隐私保护至关重要。通过在设备端直接执行推理，数据无需传输至云端，从而确保更快的响应速度并减少对网络连接的依赖。然而，在边缘设备上部署大语言模型面临诸多挑战，其中管理键值缓存尤为关键，该组件在大语言模型服务中扮演着核心角色。随着输入文本长度的增加，KV缓存的大小随序列长度线性增长，导致显著的内存占用和数据访问开销。另一方面，边缘设备的内存和计算能力有限，难以存储并高效访问大语言模型推理所需的大规模缓存。为缓解KV缓存带来的巨大开销，我们提出采用嵌入式动态随机存取存储器作为边缘设备中大语言模型服务的主要存储介质，其相较于静态随机存取存储器具有更高的存储密度。然而，为确保数据完整性，eDRAM需要执行周期性的刷新操作，这一过程能耗较高。为降低eDRAM成本并提升整体系统性能，我们提出~\textit{Kelle}——一种软硬件协同设计解决方案，专为在基于eDRAM的边缘系统中部署大语言模型而优化。结合我们提出的细粒度内存置换、重计算及刷新控制算法，\textit{Kelle}加速器相较于现有基线解决方案实现了$3.9\times$的加速比和$4.5\times$的能耗节省。