Long-tailed distributions frequently emerge in real-world data, where a large number of minority categories contain a limited number of samples. Such imbalance issue considerably impairs the performance of standard supervised learning algorithms, which are mainly designed for balanced training sets. Recent investigations have revealed that supervised contrastive learning exhibits promising potential in alleviating the data imbalance. However, the performance of supervised contrastive learning is plagued by an inherent challenge: it necessitates sufficiently large batches of training data to construct contrastive pairs that cover all categories, yet this requirement is difficult to meet in the context of class-imbalanced data. To overcome this obstacle, we propose a novel probabilistic contrastive (ProCo) learning algorithm that estimates the data distribution of the samples from each class in the feature space, and samples contrastive pairs accordingly. In fact, estimating the distributions of all classes using features in a small batch, particularly for imbalanced data, is not feasible. Our key idea is to introduce a reasonable and simple assumption that the normalized features in contrastive learning follow a mixture of von Mises-Fisher (vMF) distributions on unit space, which brings two-fold benefits. First, the distribution parameters can be estimated using only the first sample moment, which can be efficiently computed in an online manner across different batches. Second, based on the estimated distribution, the vMF distribution allows us to sample an infinite number of contrastive pairs and derive a closed form of the expected contrastive loss for efficient optimization. Our code is available at https://github.com/LeapLabTHU/ProCo.

翻译：长尾分布经常出现在现实世界数据中，其中大量少数类别包含有限数量的样本。这种不平衡问题严重损害了主要为平衡训练集设计的标准监督学习算法的性能。近期研究表明，监督对比学习在缓解数据不平衡方面展现出巨大潜力。然而，监督对比学习的性能受到一个固有挑战的困扰：它需要足够大的训练数据批次来构建覆盖所有类别的对比对，但在类别不平衡数据的背景下这一要求难以满足。为克服这一障碍，我们提出了一种新颖的概率对比学习（ProCo）算法，该算法在特征空间中估计每个类别样本的数据分布，并据此采样对比对。事实上，在小型批次中利用特征估计所有类别的分布是不可行的，尤其对于不平衡数据。我们的核心思想是引入一个合理且简单的假设：对比学习中的归一化特征在单位空间上服从von Mises-Fisher (vMF)分布的混合，这带来了两方面益处。首先，分布参数仅需利用第一样本矩即可估计，该矩可以在不同批次之间以在线方式高效计算。其次，基于估计的分布，vMF分布允许我们采样无限数量的对比对，并推导出期望对比损失的闭式解以实现高效优化。我们的代码可在https://github.com/LeapLabTHU/ProCo获取。