Transformer-based large language models (LLMs) exhibit impressive performance in generative tasks but introduce significant challenges in real-world serving due to inefficient use of the expensive, computation-optimized accelerators. This mismatch arises from the autoregressive nature of LLMs, where the generation phase comprises operators with varying resource demands. Specifically, the attention operator is memory-intensive, exhibiting a memory access pattern that clashes with the strengths of modern accelerators, especially as context length increases. To enhance the efficiency and cost-effectiveness of LLM serving, we introduce the concept of attention offloading. This approach leverages a collection of cheap, memory-optimized devices for the attention operator while still utilizing high-end accelerators for other parts of the model. This heterogeneous setup ensures that each component is tailored to its specific workload, maximizing overall performance and cost efficiency. Our comprehensive analysis and experiments confirm the viability of splitting the attention computation over multiple devices. Also, the communication bandwidth required between heterogeneous devices proves to be manageable with prevalent networking technologies. To further validate our theory, we develop Lamina, an LLM inference system that incorporates attention offloading. Experimental results indicate that Lamina can provide 1.48x-12.1x higher estimated throughput per dollar than homogeneous solutions.

翻译：基于Transformer的大语言模型（LLMs）在生成任务中展现了卓越性能，但实际部署时因未能高效利用昂贵且计算优化的加速器而面临重大挑战。这种不匹配源于LLMs的自回归特性——生成阶段包含资源需求各异的算子：其中注意力算子具有高内存密集型特征，其内存访问模式与现代加速器的优势相冲突，尤其在上下文长度增加时更为显著。为提升LLM服务的效率与成本效益，我们提出注意力卸载的概念。该方法利用一组廉价、内存优化的设备处理注意力算子，而模型其余部分仍由高端加速器执行。这种异构配置确保每个组件适配其特定工作负载，最大化整体性能与成本效率。综合分析与实验验证了跨设备拆分注意力计算的可行性，同时异构设备间的通信带宽需求可通过现有网络技术实现。为验证理论，我们开发了集成注意力卸载的LLM推理系统Lamina。实验结果表明，与同构方案相比，Lamina每美元成本可带来的预估吞吐量提升1.48倍至12.1倍。