We present the first end-to-end deployment of the Gemma3 family of large language and vision models on a tiled edge dataflow architecture (AMD Ryzen AI NPU). Our work introduces a set of hardware-aware techniques. For prefill, we introduce an efficient dequantization engine, optimize tiled matrix multiplication kernels, and propose FlowQKV, a chunked, pipelined attention mechanism. For decoding, we introduce FusedDQP, which fuses dequantization and projection into a single kernel, and FlowKV, which re-structures attention to sustain high memory bandwidth utilization. Together with a compact Q4NX 4-bit quantization format, these methods yield up to $5.2\times$ faster prefill and $4.8\times$ faster decoding versus the iGPU, and $33.5\times$ and $2.2\times$ over the CPU, respectively. Power efficiency improves by as much as $67.2\times$ and $222.9\times$ compared to the iGPU and CPU. The proposed approach demonstrates that modern NPUs can deliver practical, low-power LLM and VLM inference at the edge, and provides a generalizable blueprint for mapping transformer-based models onto tiled dataflow accelerators.

翻译：本文首次实现了Gemma3系列大型语言与视觉模型在分片边缘数据流架构（AMD Ryzen AI NPU）上的端到端部署。我们提出了一系列硬件感知技术：在预填充阶段，设计了高效反量化引擎，优化了分片矩阵乘法内核，并提出了FlowQKV——一种分块流水线注意力机制；在解码阶段，提出了将反量化与投影融合为单一内核的FusedDQP技术，以及通过重构注意力机制实现高内存带宽持续利用的FlowKV方法。结合紧凑的Q4NX 4位量化格式，这些技术使预填充速度较iGPU提升最高达$5.2\times$，解码速度提升$4.8\times$；相较于CPU分别提升$33.5\times$和$2.2\times$。能效比较iGPU和CPU最高提升$67.2\times$和$222.9\times$。该方案证明现代NPU能够在边缘端实现实用化、低功耗的LLM与VLM推理，并为基于Transformer的模型映射至分片数据流加速器提供了可推广的蓝图。