Transformer based Large Language Models (LLMs) have been widely used in many fields, and the efficiency of LLM inference becomes hot topic in real applications. However, LLMs are usually complicatedly designed in model structure with massive operations and perform inference in the auto-regressive mode, making it a challenging task to design a system with high efficiency. In this paper, we propose an efficient LLM inference solution with low latency and high throughput. Firstly, we simplify the LLM decoder layer by fusing data movement and element-wise operations to reduce the memory access frequency and lower system latency. We also propose a segment KV cache policy to keep key/value of the request and response tokens in separate physical memory for effective device memory management, helping enlarge the runtime batch size and improve system throughput. A customized Scaled-Dot-Product-Attention kernel is designed to match our fusion policy based on the segment KV cache solution. We implement our LLM inference solution on Intel GPU and publish it publicly. Compared with the standard HuggingFace implementation, the proposed solution achieves up to 7x lower token latency and 27x higher throughput for some popular LLMs on Intel GPU.

翻译：基于Transformer的大语言模型已在众多领域广泛应用，其推理效率已成为实际应用中的热点议题。然而，大语言模型通常采用复杂的模型结构设计，包含海量运算操作，并以自回归模式执行推理，这使得设计高效系统成为一项艰巨挑战。本文提出一种兼具低延迟与高吞吐特性的高效大语言模型推理解决方案。首先，我们通过融合数据搬运与逐元素运算来简化大语言模型解码层，从而降低内存访问频率与系统延迟。同时引入分段KV缓存策略，将请求与响应令牌的键值对存储于独立物理内存中，实现设备内存的高效管理，有助于扩大运行时批处理规模并提升系统吞吐量。基于分段KV缓存方案，我们设计了定制化缩放点积注意力核以匹配融合策略。该推理方案已在英特尔GPU上完成实现并公开发布。与标准HuggingFace实现相比，本方案在英特尔GPU上针对主流大语言模型实现了最高7倍的令牌延迟降低和27倍的吞吐量提升。