Modern CPUs increasingly integrate matrix extensions, such as Arm Scalable Matrix Extension (SME), that provide high-throughput matrix execution within the CPU. For LLM inference, however, these units are not a universal replacement for conventional CPU cores: prefill, decode, attention, and KV-cache operations expose different arithmetic intensities, vector behavior, and layout requirements, while SME units and CPU cores still compete for shared memory bandwidth. This paper studies this mismatch through a roofline-based characterization of SME-enabled CPUs and uses the resulting model to guide operator-level execution choices. We present SMEPilot, an LLM inference engine that selects CPU-only, SME-only, or cooperative SME+CPU execution for each operator shape. SMEPilot partitions matrix work across SME and CPU cores at tile granularity, overlaps SME-suitable matrix stages with CPU-suitable vector stages in attention, and maintains layout state so packed tensor representations are reused rather than repeatedly rebuilt on critical paths. Across Llama-3.2-3B, Qwen3-4B, and Qwen3-30BA3B on phone, PC, and server platforms, SMEPilot improves end-to-end inference performance by up to 3.94$\times$.

翻译：现代CPU越来越多地集成矩阵扩展单元（如Arm可扩展矩阵扩展SME），使其能在CPU内部实现高吞吐量的矩阵执行。然而，在LLM推理场景中，这些单元并不能完全替代传统CPU核心：预填充、解码、注意力及KV缓存操作呈现出不同的计算密度、向量行为与布局要求，同时SME单元与CPU核心仍需竞争共享内存带宽。本文通过基于屋顶线模型的特征分析方法，研究了SME型CPU中的这种不匹配现象，并利用所得模型指导算子级别的执行策略选择。我们提出SMEPilot——一个LLM推理引擎，能够针对每个算子的不同形态，分别选择纯CPU执行、纯SME执行或SME+CPU协同执行方案。SMEPilot在磁贴粒度上将矩阵计算任务划分至SME与CPU核心，在注意力计算中重叠SME适配的矩阵阶段与CPU适配的向量阶段，并维持布局状态，使得压缩张量表示得以复用，而非在关键路径上反复重建。在手机、PC及服务器平台上，针对Llama-3.2-3B、Qwen3-4B和Qwen3-30BA3B模型，SMEPilot实现了最高3.94倍的端到端推理性能提升。