Audio-Language models jointly learn multimodal text and audio representations that enable Zero-Shot inference. Models rely on the encoders to create powerful representations of the input and generalize to multiple tasks ranging from sounds, music, and speech. Although models have achieved remarkable performance, there is still a performance gap with task-specific models. In this paper, we propose a Contrastive Language-Audio Pretraining model that is pretrained with a diverse collection of 4.6M audio-text pairs employing two innovative encoders for Zero-Shot inference. To learn audio representations, we trained an audio encoder on 22 audio tasks, instead of the standard training of sound event classification. To learn language representations, we trained an autoregressive decoder-only model instead of the standard encoder-only models. Then, the audio and language representations are brought into a joint multimodal space using Contrastive Learning. We used our encoders to improve the downstream performance by a margin. We extensively evaluated the generalization of our representations on 26 downstream tasks, the largest in the literature. Our model achieves state of the art results in several tasks leading the way towards general-purpose audio representations.

翻译：音频-语言模型联合学习多模态文本与音频表示，从而实现零样本推理。模型依赖编码器为输入生成强表征，并泛化至声音、音乐及语音等多类任务。尽管现有模型已取得显著性能，但与任务专用模型仍存在性能差距。本文提出一种对比语言-音频预训练模型，该模型基于包含460万对音频-文本的多样化数据集进行预训练，采用两种创新编码器实现零样本推理。为学习音频表征，我们训练音频编码器完成22项音频任务（而非标准的声音事件分类训练）；为学习语言表征，我们采用自回归解码器专用架构（而非标准编码器专用架构）。随后通过对比学习将音频与语言表征映射至联合多模态空间。实验表明，所提编码器能显著提升下游任务性能。我们在文献规模最大的26项下游任务上全面评估表征泛化能力，模型在多项任务中达到最优水平，为通用音频表示研究开辟了新路径。