Recent advancements in diffusion models and large language models (LLMs) have significantly propelled the field of AIGC. Text-to-Audio (TTA), a burgeoning AIGC application designed to generate audio from natural language prompts, is attracting increasing attention. However, existing TTA studies often struggle with generation quality and text-audio alignment, especially for complex textual inputs. Drawing inspiration from state-of-the-art Text-to-Image (T2I) diffusion models, we introduce Auffusion, a TTA system adapting T2I model frameworks to TTA task, by effectively leveraging their inherent generative strengths and precise cross-modal alignment. Our objective and subjective evaluations demonstrate that Auffusion surpasses previous TTA approaches using limited data and computational resource. Furthermore, previous studies in T2I recognizes the significant impact of encoder choice on cross-modal alignment, like fine-grained details and object bindings, while similar evaluation is lacking in prior TTA works. Through comprehensive ablation studies and innovative cross-attention map visualizations, we provide insightful assessments of text-audio alignment in TTA. Our findings reveal Auffusion's superior capability in generating audios that accurately match textual descriptions, which further demonstrated in several related tasks, such as audio style transfer, inpainting and other manipulations. Our implementation and demos are available at https://auffusion.github.io.

翻译：近年来，扩散模型与大型语言模型（LLM）的进展显著推动了AIGC领域的发展。文本到音频（TTA）作为一种新兴的AIGC应用，旨在从自然语言提示生成音频，正受到越来越多的关注。然而，现有的TTA研究在生成质量与文本-音频对齐方面仍面临挑战，尤其对于复杂文本输入。受先进文本到图像（T2I）扩散模型的启发，我们提出了Auffusion——一种将T2I模型框架适配至TTA任务的系统，通过有效利用其固有的生成优势与精准的跨模态对齐能力。客观与主观评估结果表明，Auffusion在有限数据和计算资源下超越了先前的TTA方法。此外，T2I领域的先前研究已认识到编码器选择对跨模态对齐（如细粒度细节与对象绑定）的显著影响，而TTA领域尚缺乏类似评估。通过全面的消融研究与创新的交叉注意力图可视化，我们对TTA中的文本-音频对齐进行了深入评估。研究结果揭示了Auffusion在生成与文本描述精确匹配的音频方面的卓越能力，这进一步在音频风格迁移、修复及其他操控等多项相关任务中得到验证。我们的实现与演示可在https://auffusion.github.io获取。