Scaling large models with long sequences across applications like language generation, video generation and multimodal tasks requires efficient sequence parallelism. However, existing sequence parallelism methods all assume a single sequence dimension and fail to adapt to multi-dimensional transformer architectures that perform attention calculations across different dimensions. This paper introduces Dynamic Sequence Parallelism (DSP), a novel approach to enable efficient sequence parallelism for multi-dimensional transformer models. The key idea is to dynamically switch the parallelism dimension according to the current computation stage, leveraging the potential characteristics of multi-dimensional attention. This dynamic dimension switching allows sequence parallelism with minimal communication overhead compared to applying traditional single-dimension parallelism to multi-dimensional models. Experiments show DSP improves end-to-end throughput by 42.0% to 216.8% over prior sequence parallelism methods.

翻译：在语言生成、视频生成和多模态任务等需要长序列训练的大规模模型扩展中，高效的序列并行至关重要。然而，现有序列并行方法均假设单一序列维度，无法适应在不同维度执行注意力计算的多维变换器架构。本文提出动态序列并行（DSP），一种面向多维变换器模型的新型高效序列并行方法。其核心思想是根据当前计算阶段动态切换并行维度，充分利用多维注意力的潜在特性。与传统单维度并行方法应用于多维模型相比，这种动态维度切换能以最小通信开销实现序列并行。实验表明，与现有序列并行方法相比，DSP将端到端吞吐量提升42.0%至216.8%。