Recent work has studied text-to-audio synthesis using large amounts of paired text-audio data. However, audio recordings with high-quality text annotations can be difficult to acquire. In this work, we approach text-to-audio synthesis using unlabeled videos and pretrained language-vision models. We propose to learn the desired text-audio correspondence by leveraging the visual modality as a bridge. We train a conditional diffusion model to generate the audio track of a video, given a video frame encoded by a pretrained contrastive language-image pretraining (CLIP) model. At test time, we first explore performing a zero-shot modality transfer and condition the diffusion model with a CLIP-encoded text query. However, we observe a noticeable performance drop with respect to image queries. To close this gap, we further adopt a pretrained diffusion prior model to generate a CLIP image embedding given a CLIP text embedding. Our results show the effectiveness of the proposed method, and that the pretrained diffusion prior can reduce the modality transfer gap. While we focus on text-to-audio synthesis, the proposed model can also generate audio from image queries, and it shows competitive performance against a state-of-the-art image-to-audio synthesis model in a subjective listening test. This study offers a new direction of approaching text-to-audio synthesis that leverages the naturally-occurring audio-visual correspondence in videos and the power of pretrained language-vision models.

翻译：近期研究利用大量文本-音频配对数据实现了文本到音频合成。然而，带有高质量文本标注的音频记录往往难以获取。本文提出一种利用无标注视频与预训练语言-视觉模型实现文本到音频合成的方法，通过将视觉模态作为桥梁来学习所需的文本-音频对应关系。我们训练条件扩散模型，在给定经预训练对比语言-图像预训练（CLIP）模型编码的视频帧条件下，生成视频的音频轨道。在测试阶段，我们首先尝试执行零样本模态迁移，利用CLIP编码的文本查询对扩散模型进行条件约束。然而，我们发现其相较于图像查询存在显著的性能下降。为缩小这一差距，我们进一步采用预训练扩散先验模型，根据CLIP文本嵌入生成CLIP图像嵌入。实验结果表明了所提方法的有效性，且预训练扩散先验能有效减少模态迁移差距。尽管本研究聚焦于文本到音频合成，但所提模型也可通过图像查询生成音频，并在主观听感测试中展现出与最先进的图像到音频合成模型相竞争的性能。本研究为利用视频中自然存在的音频-视觉对应关系以及预训练语言-视觉模型的力量实现文本到音频合成开辟了新方向。