Despite recent advancements, audio-text models still lag behind their image-text counterparts in scale and performance. In this paper, we propose to improve both the data scale and the training procedure of audio-text contrastive models. Specifically, we craft a large-scale audio-text dataset containing 13,000 hours of text-labeled audio, using pretrained language models to process noisy text descriptions and automatic captioning to obtain text descriptions for unlabeled audio samples. We first train on audio-only data with a masked autoencoder (MAE) objective, which allows us to benefit from the scalability of unlabeled audio datasets. We then train a contrastive model with an auxiliary captioning objective with the audio encoder initialized from the MAE model. Our final model, which we name Cacophony, achieves state-of-the-art performance on audio-text retrieval tasks, and exhibits competitive results on the HEAR benchmark and other downstream tasks such as zero-shot classification.

翻译：尽管近期取得了进展，音频-文本模型在规模和性能上仍落后于图像-文本模型。本文提出从数据规模和训练流程两方面改进音频-文本对比模型。具体而言，我们构建了一个包含13,000小时文本标注音频的大规模数据集，采用预训练语言模型处理噪声文本描述，并利用自动标注技术为未标注音频样本生成文本描述。我们首先通过掩码自编码器（MAE）目标在纯音频数据上进行预训练，从而充分利用未标注音频数据的可扩展性。随后以MAE模型初始化的音频编码器为基础，结合辅助标注目标训练对比模型。最终模型命名为Cacophony，在音频-文本检索任务中达到最先进性能，并在HEAR基准测试及零样本分类等下游任务中展现出具有竞争力的结果。