Generative models have shown significant achievements in audio generation tasks. However, existing models struggle with complex and detailed prompts, leading to potential performance degradation. We hypothesize that this problem stems from the simplicity and scarcity of the training data. This work aims to create a large-scale audio dataset with rich captions for improving audio generation models. We first develop an automated pipeline to generate detailed captions by transforming predicted visual captions, audio captions, and tagging labels into comprehensive descriptions using a Large Language Model (LLM). The resulting dataset, Sound-VECaps, comprises 1.66M high-quality audio-caption pairs with enriched details including audio event orders, occurred places and environment information. We then demonstrate that training the text-to-audio generation models with Sound-VECaps significantly improves the performance on complex prompts. Furthermore, we conduct ablation studies of the models on several downstream audio-language tasks, showing the potential of Sound-VECaps in advancing audio-text representation learning. Our dataset and models are available online.

翻译：生成模型在音频生成任务中已展现出显著成就。然而，现有模型在处理复杂且详细的提示时存在困难，可能导致性能下降。我们假设该问题源于训练数据的简单性和稀缺性。本研究旨在创建一个具有丰富描述的大规模音频数据集，以改进音频生成模型。我们首先开发了一个自动化流程，通过使用大型语言模型（LLL）将预测的视觉描述、音频描述和标签转化为综合性叙述，从而生成详细描述。所构建的数据集Sound-VECaps包含166万对高质量音频-描述对，其描述细节涵盖音频事件顺序、发生场景及环境信息。实验表明，使用Sound-VECaps训练文本到音频生成模型能显著提升模型在复杂提示下的性能。此外，我们在多个下游音频-语言任务上对模型进行了消融研究，结果揭示了Sound-VECaps在推进音频-文本表征学习方面的潜力。我们的数据集与模型已开源发布。