In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost $O(1)$ inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation facilitates efficient long-sequence modeling with linear complexity, where each chunk is encoded parallelly while recurrently summarizing the chunks. Experimental results on language modeling show that RetNet achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference. The intriguing properties make RetNet a strong successor to Transformer for large language models. Code will be available at https://aka.ms/retnet.

翻译：本文提出Retentive Network（RetNet）作为大型语言模型的基础架构，同步实现了训练并行性、低成本推理与优异性能。我们从理论上推导了循环机制与注意力机制之间的关联，进而提出用于序列建模的保留机制，该机制支持三种计算范式：并行、循环以及分块循环。具体而言，并行表征支持训练并行化；循环表征实现了低成本O(1)推理，在不牺牲性能的前提下提升了解码吞吐量、延迟和GPU内存利用率；分块循环表征则通过线性复杂度实现高效的长序列建模——每个分块并行编码的同时以循环方式汇总各分块。语言建模实验表明，RetNet在缩放结果、并行训练、低成本部署与高效推理方面均展现出显著优势。这些引人注目的特性使RetNet成为大型语言模型中Transformer的有力后继者。代码将发布于https://aka.ms/retnet。