Transformers have emerged as the cornerstone of state-of-the-art natural language processing models, showcasing exceptional performance across a wide range of AI applications. However, the memory demands posed by the self-attention mechanism and the large feedforward network in Transformers limit their ability to handle long sequences, thereby creating challenges for tasks involving multiple long sequences or long-term dependencies. We present a distinct approach, Blockwise Parallel Transformer (BPT), that leverages blockwise computation of self-attention and feedforward network fusion to minimize memory costs. By processing longer input sequences while maintaining memory efficiency, BPT enables training sequences up to 32 times longer than vanilla Transformers and 2 to 4 times longer than previous memory-efficient methods. Extensive experiments on language modeling and reinforcement learning tasks demonstrate the effectiveness of BPT in reducing memory requirements and improving performance.

翻译：Transformer已成为最先进自然语言处理模型的基石，在各类人工智能应用中展现出卓越性能。然而，自注意力机制和大规模前馈网络带来的内存需求限制了其处理长序列的能力，从而为涉及多个长序列或长期依赖关系的任务带来了挑战。我们提出了一种独特的方法——分块并行Transformer（BPT），该方法利用自注意力的分块计算与前馈网络融合来最小化内存成本。通过在处理更长输入序列时保持内存效率，BPT支持训练的序列长度相比原始Transformer提升32倍，相比此前内存高效方法提升2至4倍。在语言建模和强化学习任务上的大量实验表明，BPT在降低内存需求与提升性能方面具有显著有效性。