Diffusion models have exhibited exciting capabilities in generating images and are also very promising for video creation. However, the inference speed of diffusion models is limited by the slow sampling process, restricting its use cases. The sequential denoising steps required for generating a single sample could take tens or hundreds of iterations and thus have become a significant bottleneck. This limitation is more salient for applications that are interactive in nature or require small latency. To address this challenge, we propose Partially Conditioned Patch Parallelism (PCPP) to accelerate the inference of high-resolution diffusion models. Using the fact that the difference between the images in adjacent diffusion steps is nearly zero, Patch Parallelism (PP) leverages multiple GPUs communicating asynchronously to compute patches of an image in multiple computing devices based on the entire image (all patches) in the previous diffusion step. PCPP develops PP to reduce computation in inference by conditioning only on parts of the neighboring patches in each diffusion step, which also decreases communication among computing devices. As a result, PCPP decreases the communication cost by around $70\%$ compared to DistriFusion (the state of the art implementation of PP) and achieves $2.36\sim 8.02\times$ inference speed-up using $4\sim 8$ GPUs compared to $2.32\sim 6.71\times$ achieved by DistriFusion depending on the computing device configuration and resolution of generation at the cost of a possible decrease in image quality. PCPP demonstrates the potential to strike a favorable trade-off, enabling high-quality image generation with substantially reduced latency.

翻译：扩散模型在图像生成方面展现出令人兴奋的能力，在视频创作领域也极具前景。然而，扩散模型的推理速度受限于缓慢的采样过程，限制了其应用场景。生成单个样本所需的顺序去噪步骤可能需要数十甚至数百次迭代，已成为一个显著的瓶颈。对于本质上具有交互性或要求低延迟的应用，这一限制更为突出。为应对此挑战，我们提出部分条件化块并行化方法，以加速高分辨率扩散模型的推理。利用相邻扩散步骤间图像差异近乎为零这一事实，块并行化方法通过多个GPU进行异步通信，基于前一个扩散步骤的完整图像（所有图像块）在多个计算设备上并行计算图像块。PCPP进一步发展了PP，通过在每个扩散步骤中仅以相邻图像块的部分区域为条件来减少推理计算量，同时降低了计算设备间的通信开销。因此，与DistriFusion（当前PP的最先进实现）相比，PCPP将通信成本降低了约$70\%$，并在使用$4\sim 8$个GPU时实现了$2.36\sim 8.02\times$的推理加速，而DistriFusion在相同计算设备配置和生成分辨率下的加速比为$2.32\sim 6.71\times$，代价是可能出现的图像质量下降。PCPP展示了在显著降低延迟的同时实现高质量图像生成的有利权衡潜力。