Recent work has studied text-to-audio synthesis using large amounts of paired text-audio data. However, audio recordings with high-quality text annotations can be difficult to acquire. In this work, we approach text-to-audio synthesis using unlabeled videos and pretrained language-vision models. We propose to learn the desired text-audio correspondence by leveraging the visual modality as a bridge. We train a conditional diffusion model to generate the audio track of a video, given a video frame encoded by a pretrained contrastive language-image pretraining (CLIP) model. At test time, we first explore performing a zero-shot modality transfer and condition the diffusion model with a CLIP-encoded text query. However, we observe a noticeable performance drop with respect to image queries. To close this gap, we further adopt a pretrained diffusion prior model to generate a CLIP image embedding given a CLIP text embedding. Our results show the effectiveness of the proposed method, and that the pretrained diffusion prior can reduce the modality transfer gap. While we focus on text-to-audio synthesis, the proposed model can also generate audio from image queries, and it shows competitive performance against a state-of-the-art image-to-audio synthesis model in a subjective listening test. This study offers a new direction of approaching text-to-audio synthesis that leverages the naturally-occurring audio-visual correspondence in videos and the power of pretrained language-vision models.

翻译：近期研究利用大量配对的文本-音频数据实现了文本到音频合成，然而获取带有高质量文本标注的音频记录十分困难。本研究提出利用无标注视频与预训练语言-视觉模型实现文本到音频合成，通过将视觉模态作为桥梁来学习所需的文本-音频对应关系。我们训练了一个条件扩散模型，在给定由预训练对比语言-图像预训练（CLIP）模型编码的视频帧的情况下，生成该视频的音频轨道。在测试阶段，我们首先探索零样本模态迁移，用CLIP编码的文本查询对扩散模型进行条件化，但发现其相对于图像查询存在明显的性能下降。为弥合这一差距，我们进一步采用预训练扩散先验模型，根据CLIP文本嵌入生成CLIP图像嵌入。结果表明了所提方法的有效性，且预训练扩散先验能缩小模态迁移差距。虽然本研究聚焦于文本到音频合成，但所提模型也能从图像查询生成音频，并在主观听觉测试中展现出与最先进图像到音频合成模型相竞争的性能。该研究为利用视频中天然的视听对应关系及预训练语言-视觉模型能力实现文本到音频合成提供了新方向。