Diffusion-based generative models have achieved remarkable success in various domains. It trains a shared model on denoising tasks that encompass different noise levels simultaneously, representing a form of multi-task learning (MTL). However, analyzing and improving diffusion models from an MTL perspective remains under-explored. In particular, MTL can sometimes lead to the well-known phenomenon of negative transfer, which results in the performance degradation of certain tasks due to conflicts between tasks. In this paper, we first aim to analyze diffusion training from an MTL standpoint, presenting two key observations: (O1) the task affinity between denoising tasks diminishes as the gap between noise levels widens, and (O2) negative transfer can arise even in diffusion training. Building upon these observations, we aim to enhance diffusion training by mitigating negative transfer. To achieve this, we propose leveraging existing MTL methods, but the presence of a huge number of denoising tasks makes this computationally expensive to calculate the necessary per-task loss or gradient. To address this challenge, we propose clustering the denoising tasks into small task clusters and applying MTL methods to them. Specifically, based on (O2), we employ interval clustering to enforce temporal proximity among denoising tasks within clusters. We show that interval clustering can be solved using dynamic programming, utilizing signal-to-noise ratio, timestep, and task affinity for clustering objectives. Through this, our approach addresses the issue of negative transfer in diffusion models by allowing for efficient computation of MTL methods. We validate the efficacy of proposed clustering and its integration with MTL methods through various experiments, demonstrating 1) improved generation quality and 2) faster training convergence of diffusion models.

翻译：基于扩散的生成模型已在多个领域取得显著成功。这类模型通过训练一个共享模型，同时处理包含不同噪声水平的去噪任务，构成了一种多任务学习形式。然而，从多任务学习角度分析和改进扩散模型的研究仍相对不足。特别地，多任务学习有时会导致著名的负迁移现象，即任务间的冲突导致某些任务的性能下降。本文首先从多任务学习视角分析扩散训练过程，提出两个关键发现：（O1）随着噪声水平差距的增大，去噪任务间的任务亲和度逐渐降低；（O2）扩散训练中甚至可能出现负迁移。基于这些发现，我们致力于通过缓解负迁移来增强扩散训练效果。为此，我们尝试利用现有的多任务学习方法，但海量去噪任务的存在使得计算每个任务所需的损失或梯度在计算上代价高昂。为解决这一挑战，我们提出将去噪任务聚类为小型任务簇，并在各簇内应用多任务学习方法。具体而言，基于（O2）发现，我们采用区间聚类强制使同一簇内的去噪任务在时间上邻近。研究表明，区间聚类可通过动态规划求解，并利用信噪比、时间步长和任务亲和度作为聚类目标。通过这种方法，我们的方案能够高效计算多任务学习方法，从而解决扩散模型中的负迁移问题。我们通过一系列实验验证了所提聚类方法及其与多任务学习方案结合的有效性，实验结果表明：1）生成质量提升，2）扩散模型训练收敛速度加快。