With the increasing computational capability of mobile devices, deploying agentic retrieval-augmented generation (RAG) locally on heterogeneous System-on-Chips (SoCs) has become a promising way to enhance LLM-based applications. However, agentic RAG induces multi-stage workflows with heterogeneous models and dynamic execution flow, while mobile SoCs exhibit strong accelerator affinity, shape sensitivity, and shared-memory bandwidth contention, making naive scheduling ineffective. We present HeRo, a heterogeneous-aware framework for low-latency agentic RAG on mobile SoCs. HeRo builds profiling-based performance models for each sub-stage and model-PU configuration, capturing latency, workload shape, and contention-induced slowdown, and leverages them in a lightweight online scheduler that combines shape-aware sub-stage partitioning, criticality-based accelerator mapping, and bandwidth-aware concurrency control. Experiments on commercial mobile devices show that HeRo reduces end-to-end latency by up to $10.94\times$ over existing deployment strategies, enabling practical on-device agentic RAG.

翻译：随着移动设备计算能力的不断提升，在异构系统级芯片（SoC）上本地部署基于智能体检索增强生成（Agentic RAG）已成为增强大语言模型应用的一种有前景的方式。然而，Agentic RAG引入了包含异构模型和动态执行流的多阶段工作流，而移动SoC表现出强加速器亲和性、形状敏感性和共享内存带宽争用，使得简单的调度策略效率低下。我们提出HeRo，一种面向移动SoC上低延迟Agentic RAG的异构感知框架。HeRo为每个子阶段和模型-PU配置构建基于性能分析的性能模型，捕获延迟、工作负载形状和争用导致的减速，并在轻量级在线调度器中利用这些信息，该调度器结合了形状感知的子阶段划分、基于关键性的加速器映射和带宽感知的并发控制。在商用移动设备上的实验表明，与现有部署策略相比，HeRo将端到端延迟降低了最多$10.94\times$，实现了实用的设备端Agentic RAG。