Recently, a series of Image-Text Matching (ITM) methods achieve impressive performance. However, we observe that most existing ITM models suffer from gradients vanishing at the beginning of training, which makes these models prone to falling into local minima. Most ITM models adopt triplet loss with Hard Negative mining (HN) as the optimization objective. We find that optimizing an ITM model using only the hard negative samples can easily lead to gradient vanishing. In this paper, we derive the condition under which the gradient vanishes during training. When the difference between the positive pair similarity and the negative pair similarity is close to 0, the gradients on both the image and text encoders will approach 0. To alleviate the gradient vanishing problem, we propose a Selectively Hard Negative Mining (SelHN) strategy, which chooses whether to mine hard negative samples according to the gradient vanishing condition. SelHN can be plug-and-play applied to existing ITM models to give them better training behavior. To further ensure the back-propagation of gradients, we construct a Residual Visual Semantic Embedding model with SelHN, denoted as RVSE++. Extensive experiments on two ITM benchmarks demonstrate the strength of RVSE++, achieving state-of-the-art performance.

翻译：近期，一系列图文匹配方法取得了显著性能。然而，我们观察到现有的大多数图文匹配模型在训练初期存在梯度消失现象，导致这些模型容易陷入局部最优解。多数图文匹配模型采用基于困难负样本挖掘的三元组损失作为优化目标。我们发现，仅使用困难负样本优化图文匹配模型容易引发梯度消失。本文推导了训练过程中梯度消失的条件：当正样本对相似度与负样本对相似度之差接近0时，图像编码器和文本编码器的梯度均趋近于0。为缓解梯度消失问题，我们提出选择性困难负样本挖掘策略，该策略根据梯度消失条件决定是否挖掘困难负样本。该策略可即插即用于现有图文匹配模型，改善其训练行为。为进一步确保梯度反向传播，我们构建了基于选择性困难负样本挖掘的残差视觉语义嵌入模型（记为RVSE++）。在两个图文匹配基准上的大量实验证明了RVSE++的优越性，其达到了最先进的性能水平。