Diffusion models deliver high quality in image synthesis but remain expensive during training and inference. Recent works have leveraged the inherent redundancy in visual content to make training more affordable by training only on a subset of visual information. While these methods were successful in providing cheaper and more effective training, sparsely trained diffusion models struggle in inference. This is due to their lacking response to Classifier-free Guidance (CFG) leading to underwhelming performance during inference. To overcome this, we propose Sparse Guidance (SG). Instead of using conditional dropout as a signal to guide diffusion models, SG uses token-level sparsity. As a result, SG preserves the high-variance of the conditional prediction better, achieving good quality and high variance outputs. Leveraging token-level sparsity at inference, SG improves fidelity at lower compute, achieving 1.58 FID on the commonly used ImageNet-256 benchmark with 25% fewer FLOPs, and yields up to 58% FLOP savings at matched baseline quality. To demonstrate the effectiveness of Sparse Guidance, we train a 2.5B text-to-image diffusion model using training time sparsity and leverage SG during inference. SG achieves improvements in composition and human preference score while increasing throughput at the same time.

翻译：扩散模型在图像合成方面展现出卓越质量，但其训练与推理过程仍存在高昂计算成本。近期研究利用视觉内容固有的冗余性，通过仅训练部分视觉信息子集来降低训练开销。尽管这些方法成功实现了更经济高效的训练，但稀疏训练的扩散模型在推理阶段面临挑战。这主要源于其对无分类器引导（CFG）的响应不足，导致推理性能未达预期。为解决此问题，我们提出稀疏引导（SG）方法。SG不再使用条件丢弃作为引导扩散模型的信号，而是采用令牌级稀疏性。因此，SG能更好地保持条件预测的高方差特性，实现优质且高方差的输出。通过在推理阶段利用令牌级稀疏性，SG以更低计算成本提升生成保真度：在广泛使用的ImageNet-256基准测试中以减少25%浮点运算量达到1.58 FID分数，并在保持基线质量的前提下实现最高58%的浮点运算节省。为验证稀疏引导的有效性，我们使用训练时稀疏性训练了25亿参数的文生图扩散模型，并在推理阶段应用SG方法。SG在提升构图质量和人类偏好评分的同时，显著提高了生成吞吐量。