Transformers are the current architecture of choice for NLP, but their attention layers do not scale well to long contexts. Recent works propose to replace attention with linear recurrent layers -- this is the case for state space models, which enjoy efficient training and inference. However, it remains unclear whether these models are competitive with transformers in machine translation (MT). In this paper, we provide a rigorous and comprehensive experimental comparison between transformers and linear recurrent models for MT. Concretely, we experiment with RetNet, Mamba, and hybrid versions of Mamba which incorporate attention mechanisms. Our findings demonstrate that Mamba is highly competitive with transformers on sentence and paragraph-level datasets, where in the latter both models benefit from shifting the training distribution towards longer sequences. Further analysis show that integrating attention into Mamba improves translation quality, robustness to sequence length extrapolation, and the ability to recall named entities.

翻译：Transformer 是目前自然语言处理领域的首选架构，但其注意力层难以有效扩展至长上下文场景。近期研究提出用线性循环层替代注意力机制——状态空间模型正是此类方法，其具备高效的训练与推理能力。然而，这些模型在机器翻译任务中是否具备与 Transformer 相竞争的实力仍不明确。本文通过严谨而全面的实验，系统比较了 Transformer 与线性循环模型在机器翻译中的性能。具体而言，我们测试了 RetNet、Mamba 以及融合注意力机制的 Mamba 混合架构。实验结果表明：在句子级与段落级数据集上，Mamba 与 Transformer 表现出相当的竞争力；在段落级任务中，两种模型均能通过将训练数据分布向长序列偏移而获得性能提升。进一步分析表明，在 Mamba 中引入注意力机制可提升翻译质量、增强序列长度外推的鲁棒性，并改善命名实体召回能力。