This paper presents VoiceLDM, a model designed to produce audio that accurately follows two distinct natural language text prompts: the description prompt and the content prompt. The former provides information about the overall environmental context of the audio, while the latter conveys the linguistic content. To achieve this, we adopt a text-to-audio (TTA) model based on latent diffusion models and extend its functionality to incorporate an additional content prompt as a conditional input. By utilizing pretrained contrastive language-audio pretraining (CLAP) and Whisper, VoiceLDM is trained on large amounts of real-world audio without manual annotations or transcriptions. Additionally, we employ dual classifier-free guidance to further enhance the controllability of VoiceLDM. Experimental results demonstrate that VoiceLDM is capable of generating plausible audio that aligns well with both input conditions, even surpassing the speech intelligibility of the ground truth audio on the AudioCaps test set. Furthermore, we explore the text-to-speech (TTS) and zero-shot text-to-audio capabilities of VoiceLDM and show that it achieves competitive results. Demos and code are available at https://voiceldm.github.io.

翻译：本文提出了VoiceLDM，一种能够根据两个不同的自然语言文本提示精准生成音频的模型：描述提示与内容提示。前者提供音频的整体环境上下文信息，后者传达语言内容。为实现这一目标，我们采用基于潜在扩散模型的文本到音频（TTA）模型，并将其功能扩展为将额外内容提示作为条件输入。通过利用预训练的对比语言-音频预训练（CLAP）和Whisper模型，VoiceLDM可在无需人工标注或转录的情况下，基于大量真实世界音频进行训练。此外，我们采用双分类器无引导方法，以进一步增强VoiceLDM的可控性。实验结果表明，VoiceLDM能够生成与两种输入条件高度一致的合理音频，甚至在AudioCaps测试集上，其语音清晰度超越了真实音频。同时，我们探索了VoiceLDM的文本到语音（TTS）与零样本文本到音频能力，并验证其达到了具有竞争力的效果。演示和代码可在https://voiceldm.github.io获取。