The core of cross-modal matching is to accurately measure the similarity between different modalities in a unified representation space. However, compared to textual descriptions of a certain perspective, the visual modality has more semantic variations. So, images are usually associated with multiple textual captions in databases. Although popular symmetric embedding methods have explored numerous modal interaction approaches, they often learn toward increasing the average expression probability of multiple semantic variations within image embeddings. Consequently, information entropy in embeddings is increased, resulting in redundancy and decreased accuracy. In this work, we propose a Dynamic Visual Semantic Sub-Embeddings framework (DVSE) to reduce the information entropy. Specifically, we obtain a set of heterogeneous visual sub-embeddings through dynamic orthogonal constraint loss. To encourage the generated candidate embeddings to capture various semantic variations, we construct a mixed distribution and employ a variance-aware weighting loss to assign different weights to the optimization process. In addition, we develop a Fast Re-ranking strategy (FR) to efficiently evaluate the retrieval results and enhance the performance. We compare the performance with existing set-based method using four image feature encoders and two text feature encoders on three benchmark datasets: MSCOCO, Flickr30K and CUB Captions. We also show the role of different components by ablation studies and perform a sensitivity analysis of the hyperparameters. The qualitative analysis of visualized bidirectional retrieval and attention maps further demonstrates the ability of our method to encode semantic variations.

翻译：跨模态匹配的核心在于统一表示空间中精确衡量不同模态间的相似性。然而，相较于特定视角的文字描述，视觉模态包含更丰富的语义变化。因此，图像在数据库中通常与多个文本描述相关联。尽管流行的对称嵌入方法已探索多种模态交互策略，但其优化方向往往倾向于提升图像嵌入中多重语义变化的平均表达概率，导致嵌入中的信息熵升高，进而引发冗余并降低匹配精度。针对这一问题，本文提出动态视觉语义子嵌入框架（DVSE）以降低信息熵。具体而言，我们通过动态正交约束损失获取一组异构的视觉子嵌入。为促使生成的候选嵌入捕获不同语义变化，我们构建了混合分布，并采用方差感知加权损失为优化过程分配差异化权重。此外，我们提出快速重排序策略（FR）以高效评估检索结果并增强性能。我们在MSCOCO、Flickr30K和CUB Captions三个基准数据集上，采用四种图像特征编码器和两种文本特征编码器，将本方法与现有基于集合的方法进行性能对比。通过消融实验揭示各组件的作用，并对超参数进行敏感性分析。可视化双向检索结果与注意力图的定性分析进一步验证了本方法在编码语义变化方面的有效性。