Diffusion Models (DMs), also referred to as score-based diffusion models, utilize neural networks to specify score functions. Unlike most other probabilistic models, DMs directly model the score functions, which makes them more flexible to parametrize and potentially highly expressive for probabilistic modeling. DMs can learn fine-grained knowledge, i.e., marginal score functions, of the underlying distribution. Therefore, a crucial research direction is to explore how to distill the knowledge of DMs and fully utilize their potential. Our objective is to provide a comprehensible overview of the modern approaches for distilling DMs, starting with an introduction to DMs and a discussion of the challenges involved in distilling them into neural vector fields. We also provide an overview of the existing works on distilling DMs into both stochastic and deterministic implicit generators. Finally, we review the accelerated diffusion sampling algorithms as a training-free method for distillation. Our tutorial is intended for individuals with a basic understanding of generative models who wish to apply DM's distillation or embark on a research project in this field.

翻译：扩散模型（DMs），亦称基于得分的扩散模型，利用神经网络定义得分函数。与大多数其他概率模型不同，DMs直接对得分函数进行建模，这使得它们参数化更灵活，并可能对概率建模具有高度表现力。DMs能够学习潜在分布的细粒度知识（即边缘得分函数）。因此，如何蒸馏DMs的知识并充分发挥其潜力已成为关键研究方向。本文旨在提供现代DMs蒸馏方法的全面概述，首先介绍DMs并讨论将其蒸馏为神经向量场时面临的挑战，随后综述将DMs蒸馏为随机与确定性隐式生成器的现有工作，最后梳理作为无训练蒸馏方法的加速扩散采样算法。本教程面向具备生成模型基础认知、希望应用DMs蒸馏或在该领域开展研究项目的读者。