With large-scale text-to-image (T2I) diffusion models achieving significant advancements in open-domain image creation, increasing attention has been focused on their natural extension to the realm of text-driven image-to-image (I2I) translation, where a source image acts as visual guidance to the generated image in addition to the textual guidance provided by the text prompt. We propose FBSDiff, a novel framework adapting off-the-shelf T2I diffusion model into the I2I paradigm from a fresh frequency-domain perspective. Through dynamic frequency band substitution of diffusion features, FBSDiff realizes versatile and highly controllable text-driven I2I in a plug-and-play manner (without need for model training, fine-tuning, or online optimization), allowing appearance-guided, layout-guided, and contour-guided I2I translation by progressively substituting low-frequency band, mid-frequency band, and high-frequency band of latent diffusion features, respectively. In addition, FBSDiff flexibly enables continuous control over I2I correlation intensity simply by tuning the bandwidth of the substituted frequency band. To further promote image translation efficiency, flexibility, and functionality, we propose FBSDiff++ which improves upon FBSDiff mainly in three aspects: (1) accelerate inference speed by a large margin (8.9$\times$ speedup in inference) with refined model architecture; (2) improve the Frequency Band Substitution module to allow for input source images of arbitrary resolution and aspect ratio; (3) extend model functionality to enable localized image manipulation and style-specific content creation with only subtle adjustments to the core method. Extensive qualitative and quantitative experiments verify superiority of FBSDiff++ in I2I translation visual quality, efficiency, versatility, and controllability compared to related advanced approaches.

翻译：随着大规模文本到图像扩散模型在开放域图像生成领域取得显著进展，研究者日益关注其在文本驱动图像到图像翻译任务中的自然拓展——该任务中，源图像与文本提示共同为生成图像提供视觉与语义引导。本文提出FBSDiff，一种从频域视角切入的新型框架，能够将现成的文本到图像扩散模型适配至文本驱动图像到图像范式。通过对扩散特征进行动态频带替换，FBSDiff以即插即用方式（无需模型训练、微调或在线优化）实现多样化且高度可控的文本驱动图像到图像翻译：通过逐次替换潜扩散特征的低频、中频和高频频带，分别实现外观引导、布局引导和轮廓引导的图像到图像翻译。此外，FBSDiff仅通过调节替换频带的带宽即可灵活控制图像到图像的相关性强度。为进一步提升图像翻译效率、灵活性与功能性，我们提出FBSDiff++，其相较于FBSDiff主要在三个方面进行改进：(1) 通过优化模型架构大幅提升推理速度（推理加速达8.9倍）；(2) 改进频带替换模块以支持任意分辨率和长宽比的输入源图像；(3) 仅通过对核心方法进行细微调整即可扩展模型功能，支持局部图像编辑和风格化内容生成。大量定性与定量实验证明，FBSDiff++在图像到图像翻译的视觉质量、效率、多样性和可控性方面均优于相关先进方法。