We study the problem of designing hard negative sampling distributions for unsupervised contrastive representation learning. We propose and analyze a novel min-max framework that seeks a representation which minimizes the maximum (worst-case) generalized contrastive learning loss over all couplings (joint distributions between positive and negative samples subject to marginal constraints) and prove that the resulting min-max optimum representation will be degenerate. This provides the first theoretical justification for incorporating additional regularization constraints on the couplings. We re-interpret the min-max problem through the lens of Optimal Transport (OT) theory and utilize regularized transport couplings to control the degree of hardness of negative examples. Through experiments we demonstrate that the negative samples generated from our designed negative distribution are more similar to the anchor than those generated from the baseline negative distribution. We also demonstrate that entropic regularization yields negative sampling distributions with parametric form similar to that in a recent state-of-the-art negative sampling design and has similar performance in multiple datasets. Utilizing the uncovered connection with OT, we propose a new ground cost for designing the negative distribution and show improved performance of the learned representation on downstream tasks compared to the representation learned when using squared Euclidean cost.

翻译：我们研究为无监督对比表示学习设计硬负采样分布的问题。我们提出并分析了一种新颖的最小-最大框架，该框架寻求一种表示，该表示在所有耦合（受边际约束的正负样本之间的联合分布）上最小化最大（最坏情况）广义对比学习损失，并证明由此产生的最小-最大最优表示将是退化的。这为首个理论依据提供了将额外正则化约束纳入耦合的必要性。我们通过最优传输（OT）理论的视角重新解释最小-最大问题，并利用正则化传输耦合来控制负例的困难程度。通过实验，我们证明了从我们设计的负分布中生成的负样本比从基线负分布中生成的样本更接近锚点。我们还证明，熵正则化产生的负采样分布具有与近期最先进的负采样设计相似的参数形式，并在多个数据集上取得了相近的性能。利用与OT的内在联系，我们提出了一种新的基础代价函数用于设计负分布，并展示了所学表示在下游任务上相比使用平方欧几里得代价学习的表示具有更优性能。