Diffusion models have achieved impressive results in generating diverse and realistic data by employing multi-step denoising processes. However, the need for accommodating significant variations in input noise at each time-step has led to diffusion models requiring a large number of parameters for their denoisers. We have observed that diffusion models effectively act as filters for different frequency ranges at each time-step noise. While some previous works have introduced multi-expert strategies, assigning denoisers to different noise intervals, they overlook the importance of specialized operations for high and low frequencies. For instance, self-attention operations are effective at handling low-frequency components (low-pass filters), while convolutions excel at capturing high-frequency features (high-pass filters). In other words, existing diffusion models employ denoisers with the same architecture, without considering the optimal operations for each time-step noise. To address this limitation, we propose a novel approach called Multi-architecturE Multi-Expert (MEME), which consists of multiple experts with specialized architectures tailored to the operations required at each time-step interval. Through extensive experiments, we demonstrate that MEME outperforms large competitors in terms of both generation performance and computational efficiency.

翻译：扩散模型通过采用多步去噪过程，在生成多样且逼真的数据方面取得了显著成果。然而，由于需要适应每个时间步输入噪声的显著变化，扩散模型的去噪器需要大量参数。我们观察到，扩散模型在每个时间步噪声中有效地充当不同频率范围的滤波器。尽管一些先前的工作引入了多专家策略，将去噪器分配给不同的噪声区间，但它们忽略了针对高频和低频进行专门操作的重要性。例如，自注意力操作能有效处理低频分量（低通滤波器），而卷积擅长捕捉高频特征（高通滤波器）。换句话说，现有的扩散模型采用了相同架构的去噪器，没有考虑每个时间步噪声的最优操作。为了解决这一局限性，我们提出了一种新颖的方法，称为多架构多专家（MEME），它由多个专家组成，每个专家都拥有针对每个时间步区间所需操作定制的专门架构。通过大量实验，我们证明了MEME在生成性能和计算效率方面均优于大型竞争对手。