Device-side Large Language Models (LLMs) have witnessed explosive growth, offering higher privacy and availability compared to cloud-side LLMs. During LLM inference, both model weights and user data are valuable, and attackers may even compromise the OS kernel to steal them. ARM TrustZone is the de facto hardware-based isolation technology on mobile devices, used to protect sensitive applications from a compromised OS. However, protecting LLM inference with TrustZone incurs significant overhead due to its inflexible isolation of memory and the NPU. To address these challenges, this paper introduces FlexServe, a fast and secure LLM serving system for mobile devices. It first introduces a Flexible Resource Isolation mechanism to construct Flexible Secure Memory (Flex-Mem) and Flexible Secure NPU (Flex-NPU). Both memory pages and the NPU can be efficiently switched between unprotected and protected modes. Based on these mechanisms, FlexServe designs a fast and secure LLM inference framework within TrustZone's secure world. The LLM-Aware Memory Management and Secure Inference Pipeline are introduced to accelerate inference. A Multi-Model Scheduler is proposed to optimize multi-model workflows. We implement a prototype of FlexServe and compare it with two TrustZone-based strawman designs. The results show that FlexServe achieves an average $10.05\times$ speedup in Time to First Token (TTFT) compared to the strawman, and an average $2.44\times$ TTFT speedup compared to an optimized strawman with pipeline and secure NPU enabled. For multi-model agent workflows, the end-to-end speedup is up to $24.30\times$ and $4.05\times$ compared to the strawman and optimized strawman, respectively.

翻译：设备端大语言模型（LLM）近年迎来爆发式增长，相较于云端LLM具有更高的隐私性与可用性。在LLM推理过程中，模型权重与用户数据均具有重要价值，攻击者甚至可能通过攻陷操作系统内核窃取这些数据。ARM TrustZone作为移动设备上事实标准的硬件隔离技术，被用于在受损操作系统环境下保护敏感应用。然而，基于TrustZone保护LLM推理会带来显著开销，原因在于其对内存与NPU的隔离机制缺乏灵活性。针对上述挑战，本文提出FlexServe——面向移动设备的快速安全大模型服务系统。该系统首先引入灵活资源隔离机制，构建灵活安全内存（Flex-Mem）与灵活安全NPU（Flex-NPU），使内存页与NPU能够在非保护模式与保护模式间高效切换。基于这些机制，FlexServe在TrustZone安全世界中设计了快速安全的LLM推理框架，通过引入LLM感知内存管理与安全推理流水线加速推理过程，并提出了多模型调度器以优化多模型工作流。我们实现了FlexServe原型系统，并与两种基于TrustZone的简易设计方案进行对比。实验结果表明：相比简易方案，FlexServe在首令牌延迟（TTFT）上平均加速10.05倍；相比启用流水线与安全NPU的优化版简易方案，TTFT平均加速2.44倍。对于多模型代理工作流，端到端加速比分别达到24.30倍（对比简易方案）与4.05倍（对比优化版简易方案）。