Long-tailed distributions frequently emerge in real-world data, where a large number of minority categories contain a limited number of samples. Such imbalance issue considerably impairs the performance of standard supervised learning algorithms, which are mainly designed for balanced training sets. Recent investigations have revealed that supervised contrastive learning exhibits promising potential in alleviating the data imbalance. However, the performance of supervised contrastive learning is plagued by an inherent challenge: it necessitates sufficiently large batches of training data to construct contrastive pairs that cover all categories, yet this requirement is difficult to meet in the context of class-imbalanced data. To overcome this obstacle, we propose a novel probabilistic contrastive (ProCo) learning algorithm that estimates the data distribution of the samples from each class in the feature space, and samples contrastive pairs accordingly. In fact, estimating the distributions of all classes using features in a small batch, particularly for imbalanced data, is not feasible. Our key idea is to introduce a reasonable and simple assumption that the normalized features in contrastive learning follow a mixture of von Mises-Fisher (vMF) distributions on unit space, which brings two-fold benefits. First, the distribution parameters can be estimated using only the first sample moment, which can be efficiently computed in an online manner across different batches. Second, based on the estimated distribution, the vMF distribution allows us to sample an infinite number of contrastive pairs and derive a closed form of the expected contrastive loss for efficient optimization. Our code is available at https://github.com/LeapLabTHU/ProCo.

翻译：长尾分布广泛存在于实际数据中，其中大量少数类别包含有限样本。这种不平衡问题严重损害了主要针对平衡训练集设计的标准监督学习算法的性能。近期研究表明，监督对比学习在缓解数据不平衡方面展现出潜力，但其性能受困于一个固有挑战：需要足够大的训练批次来构建覆盖所有类别的对比对，而这一要求在类别不平衡数据中难以实现。为克服这一障碍，我们提出一种新型概率对比学习（ProCo）算法，该算法可估计特征空间中每类样本的数据分布并据此采样对比对。实际上，在小批次中利用特征估计所有类别的分布（尤其针对不平衡数据）是难以实现的。我们的核心思想是引入一个合理且简单的假设：对比学习中的归一化特征服从单位球面上的混合冯·米塞斯-费希尔（vMF）分布，这带来双重优势。第一，分布参数仅需使用一阶样本矩即可估计，且可在不同批次间以在线方式高效计算；第二，基于估计的分布，vMF分布允许我们采样无限数量的对比对，并推导出期望对比损失的闭式解以实现高效优化。代码已开源：https://github.com/LeapLabTHU/ProCo。