Typical diffusion models are trained to accept a particular form of conditioning, most commonly text, and cannot be conditioned on other modalities without retraining. In this work, we propose a universal guidance algorithm that enables diffusion models to be controlled by arbitrary guidance modalities without the need to retrain any use-specific components. We show that our algorithm successfully generates quality images with guidance functions including segmentation, face recognition, object detection, and classifier signals. Code is available at https://github.com/arpitbansal297/Universal-Guided-Diffusion.

翻译：典型的扩散模型经过训练可接受特定形式的条件输入（最常见的是文本），且无法在不重新训练的情况下接受其他模态的条件控制。本文提出一种通用引导算法，使扩散模型无需重新训练任何特定任务组件即可被任意引导模态控制。我们证明，该算法能够通过包括分割、人脸识别、目标检测及分类器信号在内的引导函数成功生成高质量图像。代码见 https://github.com/arpitbansal297/Universal-Guided-Diffusion。