Large language models (LLMs) bring huge computational demands, which makes multi-chiplet accelerators that can integrate large-scale computing resources a powerful solution. However, existing design space exploration (DSE) efforts for such accelerators primarily focus on traditional CNN/Transformer workloads and fall short in supporting the highly dynamic behavior of real-world LLM inference services. This dynamic nature manifests in two key aspects: 1) Mixed request types: the prefill and decode phases exhibit significantly different computational patterns and are frequently interleaved by modern system-level service schedulers; 2) Variable sequence lengths: the sequence length differences across requests can span several orders of magnitude, rendering padding-based assumptions inefficient. Moreover, many prior works assume homogeneous chiplets and overlook the potential beneficial interaction between LLM dynamics and heterogeneous chiplet architectures. To bridge this gap, we introduce Compass, a co-exploration framework designed to optimize mapping strategies and hardware design for multi-chiplet accelerators, specifically tailored for dynamic LLM workloads. First, we propose a computation execution graph-based mapping encoding scheme that decouples micro-batch and layer dimensions, enabling fine-grained execution control on heterogeneous chiplets and flexibly representing various parallelism strategies. Second, based on this scheme, we develop the Compass framework itself, which integrates an evaluation engine, a mapping generation engine based on genetic algorithm, and a hardware sampling engine based on Bayesian optimization, enabling fast and flexible cross-level co-design. Compared with the SOTA DSE works Gemini and MOHaM, Compass reduces latency by 63.92\% and energy by 40.32\% on average in various scenarios, with only a 3.11\% increase in monetary cost.

翻译：大语言模型（LLM）带来了巨大的计算需求，使得能够集成大规模计算资源的多芯粒加速器成为一种强大的解决方案。然而，当前针对此类加速器的设计空间探索（DSE）工作主要聚焦于传统的CNN/Transformer工作负载，难以支持真实LLM推理服务的高度动态行为。这种动态性体现在两个关键方面：1）混合请求类型：预填充阶段和解码阶段展现出显著不同的计算模式，且常被现代系统级服务调度器频繁交错执行；2）可变序列长度：不同请求的序列长度差异可达数个数量级，使得基于填充的假设效率低下。此外，许多先前工作假设采用同质芯粒，忽视了LLM动态性与异构芯粒架构之间潜在的有利交互。为填补这一空白，我们引入指南针（Compass），一个专为动态LLM工作负载定制的协同探索框架，用于优化多芯粒加速器的映射策略与硬件设计。首先，我们提出一种基于计算执行图的映射编码方案，该方案解耦了微批次和层维度，从而实现对异构芯粒的细粒度执行控制，并灵活表达多种并行策略。其次，基于该方案，我们开发了Compass框架本身，它集成了评估引擎、基于遗传算法的映射生成引擎和基于贝叶斯优化的硬件采样引擎，实现了快速且灵活的跨层级协同设计。与当前最先进的DSE工作Gemini和MOHaM相比，Compass在各种场景下平均降低了63.92%的延迟和40.32%的能耗，仅增加3.11%的货币成本。