As large language models (LLMs) are increasingly deployed in latency- and cost-sensitive settings, inference efficiency has become a central systems challenge. While GPUs dominate current deployments, a growing number of AI accelerators claim advantages for LLM inference, yet it remains unclear under which conditions such accelerators outperform GPUs in practice. Recent inference systems decompose execution into Prefill and Decode phases, which exhibit distinct computational characteristics and latency metrics, commonly captured by time to first token (TTFT) and time per output token (TPOT). This paper presents a phase-aware evaluation of LLM inference performance across GPUs and emerging AI accelerators using a common model, Llama2-7B. By separately measuring Prefill and Decode performance, we reveal that accelerator advantages differ by phase and metric. Our results show that GPUs consistently excel in the compute-intensive Prefill phase, while GroqRack achieves significantly lower TPOT during Decode (batching not currently supported). However, GPUs regain an advantage in Decode throughput as batch size increases. These findings demonstrate that each platform exhibits distinct phase-dependent strengths. We further analyze heterogeneous Prefill/Decode disaggregation across different accelerator platforms, identifying performance gains and the workload and network conditions under which such gains are realized.

翻译：随着大语言模型（LLM）越来越多地部署在时延敏感和成本敏感场景中，推理效率已成为核心系统挑战。尽管GPU主导当前部署，但日益增多的AI加速器宣称在LLM推理方面具有优势，然而目前尚不清楚这些加速器在何种条件下能实际超越GPU的性能。近期推理系统将执行过程分解为Prefill和Decode阶段，这两个阶段表现出截然不同的计算特征和时延指标，通常通过首个令牌时延（TTFT）和每个输出令牌时延（TPOT）来衡量。本文基于通用模型Llama2-7B，针对GPU和新兴AI加速器提出了一种分阶段感知的LLM推理性能评估方法。通过分别测量Prefill和Decode阶段性能，我们发现加速器优势因阶段和指标而异。实验结果表明：GPU在计算密集型Prefill阶段始终表现优异，而GroqRack在Decode阶段实现了显著更低的TPOT（当前不支持批处理）。但随着批处理规模增大，GPU在Decode吞吐量方面重新获得优势。这些发现表明每个平台都具有差异化的阶段依赖性优势。我们进一步研究了不同加速器平台间的异构Prefill/Decode解耦，识别出其性能增益以及实现这些增益所需的工作负载与网络条件。