In text-audio retrieval (TAR) tasks, due to the heterogeneity of contents between text and audio, the semantic information contained in the text is only similar to certain frames within the audio. Yet, existing works aggregate the entire audio without considering the text, such as mean-pooling over the frames, which is likely to encode misleading audio information not described in the given text. In this paper, we present a text-aware attention pooling (TAP) module for TAR, which is essentially a scaled dot product attention for a text to attend to its most semantically similar frames. Furthermore, previous methods only conduct the softmax for every single-side retrieval, ignoring the potential cross-retrieval information. By exploring the intrinsic prior of each text-audio pair, we introduce a prior matrix revised (PMR) loss to filter the hard case with high (or low) text-to-audio but low (or high) audio-to-text similarity scores, thus achieving the dual optimal match. Experiments show that our TAP significantly outperforms various text-agnostic pooling functions. Moreover, our PMR loss also shows stable performance gains on multiple datasets.

翻译：在文本-音频检索（TAR）任务中，由于文本与音频内容之间存在异质性，文本蕴含的语义信息仅与音频中特定帧相似。然而现有方法未考虑文本特征而直接聚合整个音频（如对全帧进行均值池化），这可能导致编码出文本未描述的误导性音频信息。本文提出一种面向TAR的文本感知注意力池化（TAP）模块，其本质是一种缩放点积注意力机制，使文本能关注与之语义最相似的音频帧。此外，现有方法仅对单向检索进行softmax归一化，忽略了潜在的交叉检索信息。通过探索每个文本-音频对的内在先验，我们引入先验矩阵修正（PMR）损失，对高文本-音频相似度但低音频-文本相似度（或反之）的难例进行过滤，从而实现双向最优匹配。实验表明，TAP模块显著优于各类文本无关池化函数，同时PMR损失在多个数据集上展现出稳定的性能增益。