The key challenge in image-text retrieval is effectively leveraging semantic information to measure the similarity between vision and language data. However, using instance-level binary labels, where each image is paired with a single text, fails to capture multiple correspondences between different semantic units, leading to uncertainty in multi-modal semantic understanding. Although recent research has captured fine-grained information through more complex model structures or pre-training techniques, few studies have directly modeled uncertainty of correspondence to fully exploit binary labels. To address this issue, we propose an Uncertainty-Aware Multi-View Visual Semantic Embedding (UAMVSE)} framework that decomposes the overall image-text matching into multiple view-text matchings. Our framework introduce an uncertainty-aware loss function (UALoss) to compute the weighting of each view-text loss by adaptively modeling the uncertainty in each view-text correspondence. Different weightings guide the model to focus on different semantic information, enhancing the model's ability to comprehend the correspondence of images and texts. We also design an optimized image-text matching strategy by normalizing the similarity matrix to improve model performance. Experimental results on the Flicker30k and MS-COCO datasets demonstrate that UAMVSE outperforms state-of-the-art models.

翻译：图像-文本检索的关键挑战在于有效利用语义信息来衡量视觉与语言数据之间的相似性。然而，使用实例级二元标签（即每张图像与单个文本配对）无法捕捉不同语义单元之间的多重对应关系，导致多模态语义理解中存在不确定性。尽管近期研究通过更复杂的模型结构或预训练技术捕捉了细粒度信息，但很少有研究直接对对应关系的不确定性建模以充分利用二元标签。为解决这一问题，我们提出一个不确定性感知的多视角视觉语义嵌入（UAMVSE）框架，该框架将整体的图像-文本匹配分解为多个视角-文本匹配。我们的框架引入一个不确定性感知损失函数（UALoss），通过自适应建模每个视角-文本对应关系中的不确定性来计算各视角-文本损失的权重。不同的权重引导模型关注不同的语义信息，从而增强模型理解图像与文本对应关系的能力。我们还通过归一化相似度矩阵设计了一种优化的图像-文本匹配策略以提升模型性能。在Flicker30k和MS-COCO数据集上的实验结果表明，UAMVSE的性能优于现有的最先进模型。