The burgeoning short video industry has accelerated the advancement of video-music retrieval technology, assisting content creators in selecting appropriate music for their videos. In self-supervised training for video-to-music retrieval, the video and music samples in the dataset are separated from the same video work, so they are all one-to-one matches. This does not match the real situation. In reality, a video can use different music as background music, and a music can be used as background music for different videos. Many videos and music that are not in a pair may be compatible, leading to false negative noise in the dataset. A novel inter-intra modal (II) loss is proposed as a solution. By reducing the variation of feature distribution within the two modalities before and after the encoder, II loss can reduce the model's overfitting to such noise without removing it in a costly and laborious way. The video-music retrieval framework, II-CLVM (Contrastive Learning for Video-Music Retrieval), incorporating the II Loss, achieves state-of-the-art performance on the YouTube8M dataset. The framework II-CLVTM shows better performance when retrieving music using multi-modal video information (such as text in videos). Experiments are designed to show that II loss can effectively alleviate the problem of false negative noise in retrieval tasks. Experiments also show that II loss improves various self-supervised and supervised uni-modal and cross-modal retrieval tasks, and can obtain good retrieval models with a small amount of training samples.

翻译：短视频行业的蓬勃发展推动了视频-音乐检索技术的进步，帮助内容创作者为其视频选择合适的背景音乐。在视频到音乐检索的自监督训练中，数据集中的视频和音乐样本均来自同一视频作品，因此它们均为一对一匹配关系。这与实际情况并不相符。现实中，一个视频可以采用不同的音乐作为背景音乐，而同一段音乐也可作为不同视频的背景音乐。许多未成对的视频与音乐可能具有兼容性，这导致数据集中存在假阴性噪声。本文提出一种新颖的模态间-模态内损失作为解决方案。通过减小两个模态在编码器前后特征分布的变异度，II损失能够在不以高成本、费力的方式消除噪声的情况下，降低模型对此类噪声的过拟合。融入II损失的视频-音乐检索框架II-CLVM（面向视频-音乐检索的对比学习框架）在YouTube8M数据集上取得了最先进的性能。当使用多模态视频信息（如视频中的文本）进行音乐检索时，II-CLVTM框架展现出更优的性能。实验设计表明，II损失能有效缓解检索任务中的假阴性噪声问题。实验还证明，II损失可提升各类自监督与监督式单模态及跨模态检索任务的性能，并能在少量训练样本下获得良好的检索模型。