In text-audio retrieval (TAR) tasks, due to the heterogeneity of contents between text and audio, the semantic information contained in the text is only similar to certain frames within the audio. Yet, existing works aggregate the entire audio without considering the text, such as mean-pooling over the frames, which is likely to encode misleading audio information not described in the given text. In this paper, we present a text-aware attention pooling (TAP) module for TAR, which is essentially a scaled dot product attention for a text to attend to its most semantically similar frames. Furthermore, previous methods only conduct the softmax for every single-side retrieval, ignoring the potential cross-retrieval information. By exploring the intrinsic prior of each text-audio pair, we introduce a prior matrix revised (PMR) loss to filter the hard case with high (or low) text-to-audio but low (or high) audio-to-text similarity scores, thus achieving the dual optimal match. Experiments show that our TAP significantly outperforms various text-agnostic pooling functions. Moreover, our PMR loss also shows stable performance gains on multiple datasets.

翻译：在文本音频检索（TAR）任务中，由于文本和音频之间的内容异质性，文本中包含的语义信息仅与音频中的某些帧类似。然而，现有作品聚合整个音频时未考虑文本，例如平均池化所有帧，可能会编码与给定文本描述不符的误导性音频信息。在本文中，我们提出了一种文本感知注意池（TAP）模块用于TAR，本质上是文本对其最具语义相似性的帧的缩放点乘注意力。此外，以前的方法仅对每个单侧检索进行Softmax，忽略了潜在的跨检索信息。通过探索每个文本-音频对的固有先验，我们引入了先验矩阵修正（PMR）损失，以过滤具有高（或低）文本-音频但低（或高）音频-文本相似得分的困难案例，从而实现双重最优匹配。实验表明，我们的TAP明显优于各种不考虑文本的汇集函数。此外，我们的PMR损失也在多个数据集上表现出稳定的性能增益。