Large text-to-image diffusion models have impressive capabilities in generating photorealistic images from text prompts. How to effectively guide or control these powerful models to perform different downstream tasks becomes an important open problem. To tackle this challenge, we introduce a principled finetuning method -- Orthogonal Finetuning (OFT), for adapting text-to-image diffusion models to downstream tasks. Unlike existing methods, OFT can provably preserve hyperspherical energy which characterizes the pairwise neuron relationship on the unit hypersphere. We find that this property is crucial for preserving the semantic generation ability of text-to-image diffusion models. To improve finetuning stability, we further propose Constrained Orthogonal Finetuning (COFT) which imposes an additional radius constraint to the hypersphere. Specifically, we consider two important finetuning text-to-image tasks: subject-driven generation where the goal is to generate subject-specific images given a few images of a subject and a text prompt, and controllable generation where the goal is to enable the model to take in additional control signals. We empirically show that our OFT framework outperforms existing methods in generation quality and convergence speed.

翻译：大规模文本到图像扩散模型在根据文本提示生成逼真图像方面展现出惊人能力。如何有效引导或控制这些强大模型以执行不同下游任务，已成为一个重要的开放性问题。针对这一挑战，我们提出一种基于原理的微调方法——正交微调（OFT），用于将文本到图像扩散模型适配至下游任务。与现有方法不同，OFT可证明地保持超球面能量，该能量表征单位超球面上神经元对的相互关系。我们发现这一性质对于保持文本到图像扩散模型的语义生成能力至关重要。为提升微调稳定性，我们进一步提出约束正交微调（COFT），对超球面施加额外的半径约束。具体而言，我们考虑两类重要的文本到图像微调任务：主体驱动生成（目标是根据少量主体图像和文本提示生成主体专属图像）与可控生成（目标是使模型能够接收额外控制信号）。实验表明，我们的OFT框架在生成质量与收敛速度上均优于现有方法。