Context parallelism has emerged as a key technique to support long-context training, a growing trend in generative AI for modern large models. However, existing context parallel methods rely on static parallelization configurations that overlook the dynamic nature of training data, specifically, the variability in sequence lengths and token relationships (i.e., attention patterns) across samples. As a result, these methods often suffer from unnecessary communication overhead and imbalanced computation. In this paper, we present DCP, a dynamic context parallel training framework that introduces fine-grained blockwise partitioning of both data and computation. By enabling flexible mapping of data and computation blocks to devices, DCP can adapt to varying sequence characteristics, effectively reducing communication and improving memory and computation balance. Micro-benchmarks demonstrate that DCP accelerates attention by 1.19x~2.45x under causal masks and 2.15x~3.77x under sparse attention patterns. Additionally, we observe up to 0.94x~1.16x end-to-end training speed-up for causal masks, and 1.00x~1.46x for sparse masks.

翻译：上下文并行性已成为支持长上下文训练的关键技术，这是现代大型生成式人工智能模型日益增长的趋势。然而，现有的上下文并行方法依赖于静态的并行化配置，忽略了训练数据的动态特性，具体而言，即不同样本间序列长度和标记关系（即注意力模式）的变异性。因此，这些方法常常遭受不必要的通信开销和计算不均衡的问题。本文提出了DCP，一个动态上下文并行训练框架，它引入了数据和计算的细粒度分块划分。通过实现数据和计算块到设备的灵活映射，DCP能够适应变化的序列特征，有效减少通信并改善内存和计算均衡性。微基准测试表明，在因果掩码下，DCP将注意力计算加速了1.19倍至2.45倍；在稀疏注意力模式下，加速了2.15倍至3.77倍。此外，我们观察到，对于因果掩码，端到端训练速度最高提升了0.94倍至1.16倍；对于稀疏掩码，提升了1.00倍至1.46倍。