With the widespread adoption of Large Language Models (LLMs), the demand for high-performance LLM inference services continues to grow. To meet this demand, a growing number of AI accelerators have been proposed, such as Google TPU, Huawei NPU, Graphcore IPU, and Cerebras WSE, etc. Most of these accelerators adopt multi-core architectures to achieve enhanced scalability, but lack the flexibility of SIMT architectures. Therefore, without careful configuration of the hardware architecture, as well as deliberate design of tensor parallelism and core placement strategies, computational resources may be underutilized, resulting in suboptimal inference performance. To address these challenges, we first present a multi-level simulation framework with both transaction-level and performance-model-based simulation for multi-core NPUs. Using this simulator, we conduct a systematic analysis and further propose the optimal solutions for tensor parallelism strategies, core placement policies, memory management methods, as well as the selection between PD-disaggregation and PD-fusion on multi-core NPUs. We conduct comprehensive experiments on representative LLMs and various NPU configurations. The evaluation results demonstrate that, our solution can achieve 1.32x-6.03x speedup compared to SOTA designs for multi-core NPUs across different hardware configurations. As for LLM serving, our work offers guidance on designing optimal hardware architectures and serving strategies for multi-core NPUs across various LLM workloads.

翻译：随着大语言模型（LLM）的广泛应用，对高性能LLM推理服务的需求持续增长。为满足这一需求，越来越多的AI加速器被提出，例如Google TPU、华为NPU、Graphcore IPU和Cerebras WSE等。这些加速器大多采用多核架构以实现增强的可扩展性，但缺乏SIMT架构的灵活性。因此，若未对硬件架构进行精心配置，也未对张量并行策略与核心布局策略进行审慎设计，计算资源可能无法得到充分利用，导致推理性能欠佳。为应对这些挑战，我们首先提出了一种面向多核NPU的多层次仿真框架，该框架同时包含事务级仿真和基于性能模型的仿真。利用该仿真器，我们进行了系统性分析，并进一步提出了多核NPU上张量并行策略、核心布局策略、内存管理方法以及PD解聚与PD融合之间选择的最优解决方案。我们在代表性LLM及多种NPU配置上进行了全面实验。评估结果表明，在不同硬件配置下，我们的方案相比多核NPU的SOTA设计可实现1.32倍至6.03倍的加速。对于LLM服务而言，我们的工作为针对不同LLM负载设计多核NPU的最优硬件架构与服务策略提供了指导。