Building artificial intelligence (AI) systems on top of a set of foundation models (FMs) is becoming a new paradigm in AI research. Their representative and generative abilities learnt from vast amounts of data can be easily adapted and transferred to a wide range of downstream tasks without extra training from scratch. However, leveraging FMs in cross-modal generation remains under-researched when audio modality is involved. On the other hand, automatically generating semantically-relevant sound from visual input is an important problem in cross-modal generation studies. To solve this vision-to-audio (V2A) generation problem, existing methods tend to design and build complex systems from scratch using modestly sized datasets. In this paper, we propose a lightweight solution to this problem by leveraging foundation models, specifically CLIP, CLAP, and AudioLDM. We first investigate the domain gap between the latent space of the visual CLIP and the auditory CLAP models. Then we propose a simple yet effective mapper mechanism (V2A-Mapper) to bridge the domain gap by translating the visual input between CLIP and CLAP spaces. Conditioned on the translated CLAP embedding, pretrained audio generative FM AudioLDM is adopted to produce high-fidelity and visually-aligned sound. Compared to previous approaches, our method only requires a quick training of the V2A-Mapper. We further analyze and conduct extensive experiments on the choice of the V2A-Mapper and show that a generative mapper is better at fidelity and variability (FD) while a regression mapper is slightly better at relevance (CS). Both objective and subjective evaluation on two V2A datasets demonstrate the superiority of our proposed method compared to current state-of-the-art approaches - trained with 86% fewer parameters but achieving 53% and 19% improvement in FD and CS, respectively.

翻译：构建基于一组基础模型（FMs）的人工智能（AI）系统正成为AI研究的新范式。这些模型从海量数据中习得的表征与生成能力，可轻松适配并迁移至广泛的 downstream 任务，无需从头训练。然而，当涉及音频模态时，利用FMs进行跨模态生成的研究仍显不足。另一方面，从视觉输入自动生成语义相关声音是跨模态生成研究中的重要问题。为解决这类视觉到音频（V2A）生成问题，现有方法倾向于使用中等规模数据集从头设计并构建复杂系统。本文提出一种轻量级解决方案，通过利用CLIP、CLAP和AudioLDM等基础模型实现该目标。我们首先探究了视觉CLIP与听觉CLAP模型潜空间之间的域差距，继而提出一种简洁高效的映射机制（V2A-Mapper），通过将视觉输入在CLIP与CLAP空间间转换来弥合域差距。基于转换后的CLAP嵌入，采用预训练音频生成基础模型AudioLDM生成高保真且视觉对齐的声音。与先前方法相比，本方案仅需快速训练V2A-Mapper。进一步，我们对V2A-Mapper的选择进行了分析并开展大量实验，表明生成式映射器在保真度与多样性（FD）方面更优，而回归式映射器在相关性（CS）上略胜一筹。在两个V2A数据集上的客观与主观评估均显示，所提方法相比当前最先进方案具有优越性——参数减少86%，而在FD和CS指标上分别提升53%和19%。