Building artificial intelligence (AI) systems on top of a set of foundation models (FMs) is becoming a new paradigm in AI research. Their representative and generative abilities learnt from vast amounts of data can be easily adapted and transferred to a wide range of downstream tasks without extra training from scratch. However, leveraging FMs in cross-modal generation remains under-researched when audio modality is involved. On the other hand, automatically generating semantically-relevant sound from visual input is an important problem in cross-modal generation studies. To solve this vision-to-audio (V2A) generation problem, existing methods tend to design and build complex systems from scratch using modestly sized datasets. In this paper, we propose a lightweight solution to this problem by leveraging foundation models, specifically CLIP, CLAP, and AudioLDM. We first investigate the domain gap between the latent space of the visual CLIP and the auditory CLAP models. Then we propose a simple yet effective mapper mechanism (V2A-Mapper) to bridge the domain gap by translating the visual input between CLIP and CLAP spaces. Conditioned on the translated CLAP embedding, pretrained audio generative FM AudioLDM is adopted to produce high-fidelity and visually-aligned sound. Compared to previous approaches, our method only requires a quick training of the V2A-Mapper. We further analyze and conduct extensive experiments on the choice of the V2A-Mapper and show that a generative mapper is better at fidelity and variability (FD) while a regression mapper is slightly better at relevance (CS). Both objective and subjective evaluation on two V2A datasets demonstrate the superiority of our proposed method compared to current state-of-the-art approaches - trained with 86% fewer parameters but achieving 53% and 19% improvement in FD and CS, respectively.

翻译：将人工智能系统构建在多个基础模型（FMs）之上正成为AI研究的新范式。这些模型从海量数据中学习到的表征与生成能力，可轻松适配并迁移至各类下游任务，无需从头开始额外训练。然而，当涉及音频模态时，如何在跨模态生成中利用基础模型仍鲜有研究。另一方面，从视觉输入自动生成语义相关的声音是跨模态生成领域的重要课题。为解决这一视觉到音频（V2A）生成问题，现有方法倾向于使用中等规模数据集从头设计并构建复杂系统。本文提出利用CLIP、CLAP和AudioLDM等基础模型的轻量级解决方案：首先研究视觉CLIP与听觉CLAP模型潜在空间之间的域差距，继而提出简单高效的映射机制（V2A-Mapper），通过将视觉输入在CLIP和CLAP空间之间进行转换来弥合域差距。基于转换后的CLAP嵌入，采用预训练音频生成基础模型AudioLDM生成高保真且视觉对齐的声音。与现有方法相比，本方法仅需快速训练V2A-Mapper。我们进一步分析并开展针对V2A-Mapper选择策略的广泛实验，结果表明生成式映射器在保真度和多样性（FD）指标上表现更优，而回归式映射器在相关性（CS）指标上略胜一筹。在两个V2A数据集上的客观与主观评估均显示，本方法优于当前最先进方法——参数减少86%的同时，FD与CS指标分别提升53%和19%。