Contrastive learning has become a fundamental approach in both uni-modal and multi-modal frameworks. This learning paradigm pulls positive pairs of samples closer while pushing negatives apart. In the uni-modal setting (e.g., image-based learning), previous research has shown that the strength of these forces can be controlled through the temperature parameter. In this work, we propose Multi-Modal Temperature and Margin Schedules (MM-TS), extending the concept of uni-modal temperature scheduling to multi-modal contrastive learning. Our method dynamically adjusts the temperature in the contrastive loss during training, modulating the attraction and repulsion forces in the multi-modal setting. Additionally, recognizing that standard multi-modal datasets often follow imbalanced, long-tail distributions, we adapt the temperature based on the local distribution of each training sample. Specifically, samples from dense clusters are assigned a higher temperature to better preserve their semantic structure. Furthermore, we demonstrate that temperature scheduling can be effectively integrated within a max-margin framework, thereby unifying the two predominant approaches in multi-modal contrastive learning: InfoNCE loss and max-margin objective. We evaluate our approach on four widely used image- and video-language datasets, Flickr30K, MSCOCO, EPIC-KITCHENS-100, and YouCook2, and show that our dynamic temperature and margin schedules improve performance and lead to new state-of-the-art results in the field.

翻译：对比学习已成为单模态与多模态框架中的基础性方法。该学习范式通过拉近正样本对的距离同时推远负样本对来实现表征学习。在单模态场景（例如基于图像的学习）中，先前研究表明，可通过温度参数控制这些吸引与排斥力的强度。本文提出多模态温度与间隔调度策略（MM-TS），将单模态温度调度的概念扩展至多模态对比学习。我们的方法在训练过程中动态调整对比损失中的温度参数，从而调节多模态设置下的吸引与排斥力。此外，考虑到标准多模态数据集通常呈现不平衡的长尾分布，我们依据每个训练样本的局部分布自适应调整温度：具体而言，对来自密集簇的样本分配较高温度，以更好地保持其语义结构。进一步地，我们证明了温度调度可有效整合至最大间隔框架中，从而统一多模态对比学习中两种主流方法——InfoNCE损失与最大间隔目标。我们在四个广泛使用的图像-语言及视频-语言数据集（Flickr30K、MSCOCO、EPIC-KITCHENS-100和YouCook2）上评估所提方法，结果表明动态温度与间隔调度策略能提升性能，并在该领域取得新的最先进结果。