Diffusion Transformers (DiTs) are increasingly adopted in scientific computing, yet growing model sizes and resolutions make distributed multi-GPU inference essential. Ulysses sequence parallelism scales DiT inference but introduces frequent all-to-all collectives that dominate latency. Overlapping these with computation is difficult due to tight data dependencies, large message volumes, and asymmetric interconnect bandwidths. We introduce CoCoDiff, a distributed DiT inference engine exploiting two observations: (1) V requires only linear projection while Q/K need additional normalization and RoPE, creating opportunities to overlap V's communication with Q/K computation; (2) adjacent denoising steps produce similar tensors, yielding temporal redundancy. CoCoDiff introduces three mechanisms: Tile-Aware Parallel All-to-all (TAPA) decomposes collectives into topology-aligned phases; V-First scheduling hides V's communication behind Q/K computation; and V-Major selective communication transmits only active projections on slow interconnects. On the Aurora supercomputer with four DiT models across 1-8 nodes (up to 96 Intel GPU tiles), CoCoDiff achieves an average speedup of 3.6x, peaking at 8.4x.

翻译：扩散Transformer（DiTs）正越来越多地被应用于科学计算领域，然而不断增长的模型尺寸与分辨率使得分布式多GPU推理成为必需。Ulysses序列并行虽能扩展DiT推理，但会引入频繁的全到全集体通信操作，成为延迟的主要来源。由于严格的数据依赖、大量消息体量以及非对称互连带宽，将这些通信操作与计算进行重叠十分困难。我们提出CoCoDiff——一种分布式DiT推理引擎，基于以下两个观察： (1) V仅需线性投影，而Q/K需要额外的归一化与RoPE，这为将V的通信与Q/K的计算重叠创造了机会； (2) 相邻去噪步骤生成的张量相似，由此产生时间冗余。CoCoDiff引入了三种机制：瓦片感知并行全到全（TAPA）将集体通信分解为拓扑对齐的阶段；V优先调度将V的通信隐藏在Q/K计算之后；以及V主导选择性通信仅在慢速互连上传输活跃投影。在配备四个DiT模型、覆盖1至8个节点（最多96个Intel GPU瓦片）的Aurora超级计算机上，CoCoDiff实现了平均3.6倍、峰值8.4倍的加速比。