It is currently difficult to distill discrete diffusion models. In contrast, continuous diffusion literature has many distillation approaches methods that can reduce sampling steps to a handful. Our method, Discrete Moment Matching Distillation (D-MMD), leverages ideas that have been highly successful in the continuous domain. Whereas previous discrete distillation methods collapse, D-MMD maintains high quality and diversity (given sufficient sampling steps). This is demonstrated on both text and image datasets. Moreover, the newly distilled generators can outperform their teachers.

翻译：当前离散扩散模型的蒸馏仍具挑战性。相比之下，连续扩散领域已涌现出多种可将采样步数缩减至个位数的蒸馏方法。本方法——离散矩匹配蒸馏（D-MMD），借鉴了连续域中已获显著成效的核心思想。不同于以往离散蒸馏方法易出现的性能崩塌问题，D-MMD在充足采样步数下能保持生成质量与多样性。该方法在文本与图像数据集上均得到验证，且新蒸馏的生成器性能可超越其教师模型。