Audio-Language models jointly learn multimodal text and audio representations that enable Zero-Shot inference. Models rely on the encoders to create powerful representations of the input and generalize to multiple tasks ranging from sounds, music, and speech. Although models have achieved remarkable performance, there is still a performance gap with task-specific models. In this paper, we propose a Contrastive Language-Audio Pretraining model that is pretrained with a diverse collection of 4.6M audio-text pairs employing two innovative encoders for Zero-Shot inference. To learn audio representations, we trained an audio encoder on 22 audio tasks, instead of the standard training of sound event classification. To learn language representations, we trained an autoregressive decoder-only model instead of the standard encoder-only models. Then, the audio and language representations are brought into a joint multimodal space using Contrastive Learning. We used our encoders to improve the downstream performance by a margin. We extensively evaluated the generalization of our representations on 26 downstream tasks, the largest in the literature. Our model achieves state of the art results in several tasks leading the way towards general-purpose audio representations.

翻译：音频-语言模型通过联合学习多模态文本与音频表示，实现了零样本推理能力。这类模型依赖编码器构建强大的输入表示，能够泛化到声音、音乐、语音等多项任务。尽管现有模型取得了显著性能，但与任务专用模型相比仍存在差距。本文提出了一种基于对比学习的语言-音频预训练模型，该模型采用两个创新编码器在包含460万对音频-文本的多样化数据集上预训练，专为零样本推理设计。为学习音频表示，我们训练了一个覆盖22项音频任务的编码器，而非标准的声音事件分类训练范式；为学习语言表示，我们采用自回归解码器-仅模型替代常规编码器-仅模型。随后通过对比学习将音频与语言表示映射至联合多模态空间。实验表明，所提编码器可显著提升下游任务性能。我们在文献最大规模的26项下游任务上全面评估了表示泛化能力，模型在多项任务中达到最优水平，为通用音频表示研究奠定了方向。