In recent years, the explosion of web videos makes text-video retrieval increasingly essential and popular for video filtering, recommendation, and search. Text-video retrieval aims to rank relevant text/video higher than irrelevant ones. The core of this task is to precisely measure the cross-modal similarity between texts and videos. Recently, contrastive learning methods have shown promising results for text-video retrieval, most of which focus on the construction of positive and negative pairs to learn text and video representations. Nevertheless, they do not pay enough attention to hard negative pairs and lack the ability to model different levels of semantic similarity. To address these two issues, this paper improves contrastive learning using two novel techniques. First, to exploit hard examples for robust discriminative power, we propose a novel Dual-Modal Attention-Enhanced Module (DMAE) to mine hard negative pairs from textual and visual clues. By further introducing a Negative-aware InfoNCE (NegNCE) loss, we are able to adaptively identify all these hard negatives and explicitly highlight their impacts in the training loss. Second, our work argues that triplet samples can better model fine-grained semantic similarity compared to pairwise samples. We thereby present a new Triplet Partial Margin Contrastive Learning (TPM-CL) module to construct partial order triplet samples by automatically generating fine-grained hard negatives for matched text-video pairs. The proposed TPM-CL designs an adaptive token masking strategy with cross-modal interaction to model subtle semantic differences. Extensive experiments demonstrate that the proposed approach outperforms existing methods on four widely-used text-video retrieval datasets, including MSR-VTT, MSVD, DiDeMo and ActivityNet.

翻译：近年来，网络视频的爆炸式增长使文本-视频检索在视频过滤、推荐和搜索中愈发重要且普及。文本-视频检索旨在将相关文本/视频排序至不相关项之前，其核心在于精确度量文本与视频之间的跨模态相似度。近期，对比学习方法在文本-视频检索中展现出优异效果，该类方法多聚焦于构建正负样本对以学习文本和视频表征。然而，现有方法对困难负样本关注不足，且缺乏建模不同层次语义相似度的能力。针对这两个问题，本文通过两项创新技术改进对比学习。首先，为利用困难样本增强判别鲁棒性，我们提出新型双模态注意力增强模块（DMAE），从文本和视觉线索中挖掘困难负样本。通过进一步引入负感知InfoNCE（NegNCE）损失，能够自适应识别所有困难负样本并显式突显其在训练损失中的影响。其次，本研究论证了相较于成对样本，三元组样本能更好地建模细粒度语义相似度。据此提出三元组局部边界对比学习（TPM-CL）模块，通过自动生成匹配文本-视频对的细粒度困难负样本，构建偏序三元组样本。所提出的TPM-CL设计了一种带跨模态交互的自适应标记掩码策略，以建模细微语义差异。大量实验表明，本方法在MSR-VTT、MSVD、DiDeMo和ActivityNet这四个广泛使用的文本-视频检索数据集上均优于现有方法。