The aim of this research is to refine knowledge transfer on audio-image temporal agreement for audio-text cross retrieval. To address the limited availability of paired non-speech audio-text data, learning methods for transferring the knowledge acquired from a large amount of paired audio-image data to shared audio-text representation have been investigated, suggesting the importance of how audio-image co-occurrence is learned. Conventional approaches in audio-image learning assign a single image randomly selected from the corresponding video stream to the entire audio clip, assuming their co-occurrence. However, this method may not accurately capture the temporal agreement between the target audio and image because a single image can only represent a snapshot of a scene, though the target audio changes from moment to moment. To address this problem, we propose two methods for audio and image matching that effectively capture the temporal information: (i) Nearest Match wherein an image is selected from multiple time frames based on similarity with audio, and (ii) Multiframe Match wherein audio and image pairs of multiple time frames are used. Experimental results show that method (i) improves the audio-text retrieval performance by selecting the nearest image that aligns with the audio information and transferring the learned knowledge. Conversely, method (ii) improves the performance of audio-image retrieval while not showing significant improvements in audio-text retrieval performance. These results indicate that refining audio-image temporal agreement may contribute to better knowledge transfer to audio-text retrieval.

翻译：本研究旨在细化音频-图像时序一致性知识迁移，以服务于音频-文本跨模态检索任务。针对非语音音频-文本配对数据稀缺的问题，学术界探索了将大量音频-图像配对数据习得的知识迁移至共享音频-文本表示的学习方法，这凸显了音频-图像共现关系学习模式的重要性。传统音频-图像学习方法会从对应视频流中随机选取单帧图像与整段音频片段配对，以此假设二者的共现关系。然而，由于单帧图像仅能表征场景的瞬时快照，而目标音频随时间动态变化，该方法难以准确捕捉目标音频与图像间的时序一致性。为解决该问题，我们提出两种能有效捕获时序信息的音频-图像匹配方法：(i) 最近邻匹配——基于音频与多时间帧图像的相似度选取最佳匹配图像；(ii) 多帧匹配——采用多时间帧的音频-图像对进行学习。实验结果表明，方法(i)通过选取与音频信息最匹配的最近邻图像并迁移习得知识，有效提升了音频-文本检索性能。相比之下，方法(ii)虽能改善音频-图像检索性能，但对音频-文本检索性能的提升不显著。上述结果证明，细化音频-图像时序一致性可能有助于更有效地将知识迁移至音频-文本检索任务。