Automated audio captioning (AAC) aims to generate informative descriptions for various sounds from nature and/or human activities. In recent years, AAC has quickly attracted research interest, with state-of-the-art systems now relying on a sequence-to-sequence (seq2seq) backbone powered by strong models such as Transformers. Following the macro-trend of applied machine learning research, in this work, we strive to improve the performance of seq2seq AAC models by extensively leveraging pretrained models and large language models (LLMs). Specifically, we utilize BEATs to extract fine-grained audio features. Then, we employ Instructor LLM to fetch text embeddings of captions, and infuse their language-modality knowledge into BEATs audio features via an auxiliary InfoNCE loss function. Moreover, we propose a novel data augmentation method that uses ChatGPT to produce caption mix-ups (i.e., grammatical and compact combinations of two captions) which, together with the corresponding audio mixtures, increase not only the amount but also the complexity and diversity of training data. During inference, we propose to employ nucleus sampling and a hybrid reranking algorithm, which has not been explored in AAC research. Combining our efforts, our model achieves a new state-of-the-art 32.6 SPIDEr-FL score on the Clotho evaluation split, and wins the 2023 DCASE AAC challenge.

翻译：自动音频描述（AAC）旨在为自然界和/或人类活动中的各种声音生成信息丰富的文字描述。近年来，AAC迅速引起研究兴趣，当前最先进的系统依赖于由Transformer等强大模型驱动的序列到序列（seq2seq）主干架构。遵循应用机器学习研究的宏观趋势，本研究致力于通过广泛利用预训练模型和大语言模型（LLM）来提升seq2seq AAC模型的性能。具体而言，我们采用BEATs提取细粒度音频特征，随后使用Instructor大语言模型获取描述的文本嵌入，并通过辅助的InfoNCE损失函数将语言模态知识融入BEATs音频特征。此外，我们提出一种新颖的数据增强方法，利用ChatGPT生成描述混合样本（即两段描述符合语法规则的紧凑组合），这些混合样本与对应音频混合数据共同作用，不仅增加了训练数据量，还提升了数据的复杂性和多样性。在推理阶段，我们提出采用核采样与混合重排序算法——该技术此前在AAC研究中尚未被探索。综合上述改进，我们的模型在Clotho评估分割集上以32.6 SPIDEr-FL分数达到新最优水平，并荣获2023年DCASE AAC挑战赛冠军。