The landscape of image generation has been forever changed by open vocabulary diffusion models. However, at their core these models use transformers, which makes generation slow. Better implementations to increase the throughput of these transformers have emerged, but they still evaluate the entire model. In this paper, we instead speed up diffusion models by exploiting natural redundancy in generated images by merging redundant tokens. After making some diffusion-specific improvements to Token Merging (ToMe), our ToMe for Stable Diffusion can reduce the number of tokens in an existing Stable Diffusion model by up to 60% while still producing high quality images without any extra training. In the process, we speed up image generation by up to 2x and reduce memory consumption by up to 5.6x. Furthermore, this speed-up stacks with efficient implementations such as xFormers, minimally impacting quality while being up to 5.4x faster for large images. Code is available at https://github.com/dbolya/tomesd.

翻译：图像生成领域因开放词汇扩散模型而发生了根本性变革。然而，这些模型的核心采用Transformer架构，导致生成速度缓慢。尽管已有提升Transformer吞吐量的优化实现，但这些方法仍需评估整个模型。本文通过利用生成图像中的自然冗余性，合并冗余Token来加速扩散模型。在对Token合并方法（ToMe）进行扩散特化改进后，我们的稳定扩散ToMe方法能在无需额外训练的情况下，将现有稳定扩散模型的Token数量减少高达60%，同时保持高质量图像生成。在此过程中，我们实现了最高2倍的图像生成加速和最高5.6倍的内存消耗降低。此外，该加速效果可与xFormers等高效实现叠加，在图像尺寸较大时，质量影响极小，速度提升可达5.4倍。代码已开源至https://github.com/dbolya/tomesd。