State-of-the-art audio captioning methods typically use the encoder-decoder structure with pretrained audio neural networks (PANNs) as encoders for feature extraction. However, the convolution operation used in PANNs is limited in capturing the long-time dependencies within an audio signal, thereby leading to potential performance degradation in audio captioning. This letter presents a novel method using graph attention (GraphAC) for encoder-decoder based audio captioning. In the encoder, a graph attention module is introduced after the PANNs to learn contextual association (i.e. the dependency among the audio features over different time frames) through an adjacency graph, and a top-k mask is used to mitigate the interference from noisy nodes. The learnt contextual association leads to a more effective feature representation with feature node aggregation. As a result, the decoder can predict important semantic information about the acoustic scene and events based on the contextual associations learned from the audio signal. Experimental results show that GraphAC outperforms the state-of-the-art methods with PANNs as the encoders, thanks to the incorporation of the graph attention module into the encoder for capturing the long-time dependencies within the audio signal. The source code is available at https://github.com/LittleFlyingSheep/GraphAC.

翻译：最先进的音频字幕生成方法通常采用编码器-解码器结构，并使用预训练音频神经网络（PANNs）作为编码器进行特征提取。然而，PANNs中使用的卷积运算在捕捉音频信号内的长期依赖关系方面存在局限，从而导致音频字幕生成性能潜在下降。本文提出了一种新颖方法，利用图注意力（GraphAC）实现基于编码器-解码器的音频字幕生成。在编码器中，引入一个图注意力模块置于PANNs之后，通过邻接图学习上下文关联（即不同时间帧上音频特征之间的依赖关系），并采用top-k掩码来减轻噪声节点的干扰。学习到的上下文关联通过特征节点聚合，形成更有效的特征表示。因此，解码器能够基于从音频信号中学习到的上下文关联，预测关于声学场景和事件的重要语义信息。实验结果表明，由于将图注意力模块融入编码器以捕捉音频信号内的长期依赖关系，GraphAC在性能上优于使用PANNs作为编码器的最先进方法。源代码可在https://github.com/LittleFlyingSheep/GraphAC获取。