In text-audio retrieval (TAR) tasks, due to the heterogeneity of contents between text and audio, the semantic information contained in the text is only similar to certain frames within the audio. Yet, existing works aggregate the entire audio without considering the text, such as mean-pooling over the frames, which is likely to encode misleading audio information not described in the given text. In this paper, we present a text-aware attention pooling (TAP) module for TAR, which is essentially a scaled dot product attention for a text to attend to its most semantically similar frames. Furthermore, previous methods only conduct the softmax for every single-side retrieval, ignoring the potential cross-retrieval information. By exploring the intrinsic prior of each text-audio pair, we introduce a prior matrix revised (PMR) loss to filter the hard case with high (or low) text-to-audio but low (or high) audio-to-text similarity scores, thus achieving the dual optimal match. Experiments show that our TAP significantly outperforms various text-agnostic pooling functions. Moreover, our PMR loss also shows stable performance gains on multiple datasets.

翻译：在文本-音频检索（TAR）任务中，由于文本与音频内容存在异质性，文本所蕴含的语义信息仅与音频中的特定帧片段相似。然而，现有方法在聚合整个音频信号时未考虑文本特征，例如采用帧级平均池化操作，这可能导致编码与给定文本描述无关的误导性音频信息。本文提出了一种面向TAR的文本感知注意力池化（TAP）模块，其本质是通过缩放点积注意力机制使文本聚焦于语义最相似的音频帧。此外，现有方法仅对单侧检索进行softmax归一化，忽略了跨检索方向的潜在关联信息。通过挖掘每个文本-音频对的内在先验，我们引入先验矩阵修正（PMR）损失函数，以过滤具有高（低）文本-音频相似度但低（高）音频-文本相似度得分的困难样本，从而实现双向最优匹配。实验表明，我们的TAP模块显著优于各类文本无关池化函数。同时，PMR损失在多个数据集上均展现出稳定性能提升。