Text-to-image diffusion models (SD) exhibit significant advancements while requiring extensive computational resources. Existing acceleration methods usually require extensive training and are not universally applicable. LCM-LoRA, trainable once for diverse models, offers universality but rarely considers ensuring the consistency of generated content before and after acceleration. This paper proposes SpeedUpNet (SUN), an innovative acceleration module, to address the challenges of universality and consistency. Exploiting the role of cross-attention layers in U-Net for SD models, we introduce an adapter specifically designed for these layers, quantifying the offset in image generation caused by negative prompts relative to positive prompts. This learned offset demonstrates stability across a range of models, enhancing SUN's universality. To improve output consistency, we propose a Multi-Step Consistency (MSC) loss, which stabilizes the offset and ensures fidelity in accelerated content. Experiments on SD v1.5 show that SUN leads to an overall speedup of more than 10 times compared to the baseline 25-step DPM-solver++, and offers two extra advantages: (1) training-free integration into various fine-tuned Stable-Diffusion models and (2) state-of-the-art FIDs of the generated data set before and after acceleration guided by random combinations of positive and negative prompts. Code is available: https://williechai.github.io/speedup-plugin-for-stable-diffusions.github.io.

翻译：文本到图像扩散模型（SD）在取得显著进展的同时，需要大量的计算资源。现有的加速方法通常需要大量训练，且不具备普适性。LCM-LoRA 虽可一次训练适用于多种模型，提供了通用性，但很少考虑确保加速前后生成内容的一致性。本文提出 SpeedUpNet（SUN），一种创新的加速模块，以应对通用性和一致性方面的挑战。通过利用 U-Net 中交叉注意力层在 SD 模型中的作用，我们引入了一个专门为此类层设计的适配器，用于量化负向提示词相对于正向提示词在图像生成中引起的偏移。这种学习到的偏移在一系列模型中表现出稳定性，从而增强了 SUN 的通用性。为了提高输出一致性，我们提出了一种多步一致性（MSC）损失，该损失能稳定偏移并确保加速后内容的保真度。在 SD v1.5 上的实验表明，与基线 25 步 DPM-solver++ 相比，SUN 实现了超过 10 倍的整体加速，并提供了两个额外优势：（1）无需训练即可集成到各种微调的 Stable-Diffusion 模型中；（2）在正向和负向提示词随机组合引导下，加速前后生成的数据集均达到最先进的 FID 分数。代码已公开：https://williechai.github.io/speedup-plugin-for-stable-diffusions.github.io。