Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zero-shot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-630K and the proposed model are both available to the public.

翻译：对比学习在多模态表示学习领域取得了显著成功。本文提出了一种对比语言-音频预训练管线，通过将音频数据与自然语言描述相结合来开发音频表示。为实现这一目标，我们首先发布了LAION-Audio-630K数据集——一个包含来自不同数据源的633,526个音频-文本对的大规模集合。其次，我们通过考虑不同的音频编码器和文本编码器构建了对比语言-音频预训练模型，并在模型设计中融合了特征融合机制与关键词到标题增强方法，以增强模型处理可变长度音频输入的能力并提升性能。第三，我们开展了全面的实验，在三个任务上评估模型：文本到音频检索、零样本音频分类和监督式音频分类。结果表明，我们的模型在文本到音频检索任务中取得了优越性能。在音频分类任务中，模型在零样本设定下达到了最先进水平，并能获得与非零样本设定下模型结果相当的性能。LAION-Audio-630K数据集及所提出的模型均已向公众开放。