In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost $O(1)$ inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation facilitates efficient long-sequence modeling with linear complexity, where each chunk is encoded parallelly while recurrently summarizing the chunks. Experimental results on language modeling show that RetNet achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference. The intriguing properties make RetNet a strong successor to Transformer for large language models. Code will be available at https://aka.ms/retnet.

翻译：本文提出记忆网络（RetNet）作为大型语言模型的基础架构，同时实现了训练并行性、低成本推理和良好性能。我们从理论上推导了循环机制与注意力机制之间的联系，进而提出用于序列建模的保留机制，该机制支持三种计算范式：并行、循环和分块循环。具体而言，并行表示实现了训练并行性；循环表示支持低成本的O(1)复杂度推理，可在不牺牲性能的前提下提升解码吞吐量、降低延迟和GPU内存占用；分块循环表示则通过线性复杂度实现高效的长序列建模，其中每个分块被并行编码，同时以循环方式汇总各分块信息。语言建模实验表明，RetNet在扩展性能、并行训练、低成本部署和高效推理方面均表现出色。这些特性使RetNet成为Transformer在大型语言模型领域的有力后继者。代码将发布于https://aka.ms/retnet。