Text-to-audio (TTA) system has recently gained attention for its ability to synthesize general audio based on text descriptions. However, previous studies in TTA have limited generation quality with high computational costs. In this study, we propose AudioLDM, a TTA system that is built on a latent space to learn the continuous audio representations from contrastive language-audio pretraining (CLAP) latents. The pretrained CLAP models enable us to train LDMs with audio embedding while providing text embedding as a condition during sampling. By learning the latent representations of audio signals and their compositions without modeling the cross-modal relationship, AudioLDM is advantageous in both generation quality and computational efficiency. Trained on AudioCaps with a single GPU, AudioLDM achieves state-of-the-art TTA performance measured by both objective and subjective metrics (e.g., frechet distance). Moreover, AudioLDM is the first TTA system that enables various text-guided audio manipulations (e.g., style transfer) in a zero-shot fashion. Our implementation and demos are available at https://audioldm.github.io.

翻译：文本到音频（TTA）系统因其基于文本描述合成通用音频的能力而近期备受关注。然而，以往TTA研究存在生成质量有限且计算成本高昂的问题。本研究提出AudioLDM，一种基于潜在空间构建的TTA系统，通过学习对比语言-音频预训练（CLAP）潜在表征来获取连续音频表示。预训练的CLAP模型使我们能够在训练潜在扩散模型（LDMs）时使用音频嵌入，同时在采样过程中提供文本嵌入作为条件。通过直接学习音频信号及其构成的潜在表征而无需建模跨模态关系，AudioLDM在生成质量和计算效率方面均具有优势。在单个GPU上基于AudioCaps数据集训练的AudioLDM，在客观与主观指标（如弗雷歇距离）上均实现了最先进的TTA性能。此外，AudioLDM是首个支持零样本方式下多种文本引导音频操作（如风格迁移）的TTA系统。我们的实现代码与演示见https://audioldm.github.io。