In-Batch contrastive learning is a state-of-the-art self-supervised method that brings semantically-similar instances close while pushing dissimilar instances apart within a mini-batch. Its key to success is the negative sharing strategy, in which every instance serves as a negative for the others within the mini-batch. Recent studies aim to improve performance by sampling hard negatives \textit{within the current mini-batch}, whose quality is bounded by the mini-batch itself. In this work, we propose to improve contrastive learning by sampling mini-batches from the input data. We present BatchSampler\footnote{The code is available at \url{https://github.com/THUDM/BatchSampler}} to sample mini-batches of hard-to-distinguish (i.e., hard and true negatives to each other) instances. To make each mini-batch have fewer false negatives, we design the proximity graph of randomly-selected instances. To form the mini-batch, we leverage random walk with restart on the proximity graph to help sample hard-to-distinguish instances. BatchSampler is a simple and general technique that can be directly plugged into existing contrastive learning models in vision, language, and graphs. Extensive experiments on datasets of three modalities show that BatchSampler can consistently improve the performance of powerful contrastive models, as shown by significant improvements of SimCLR on ImageNet-100, SimCSE on STS (language), and GraphCL and MVGRL on graph datasets.

翻译：批内对比学习是一种先进的自我监督方法，其通过在小批量内拉近语义相似的实例、推远不相似实例来实现表征学习。该方法成功的关键在于负例共享策略——每个实例在批量内同时作为其他实例的负例。近期研究致力于通过在当前批内采样困难负例来提升性能，但其质量受限于批量本身。本文提出通过从输入数据中采样小批量来改进对比学习，并设计BatchSampler\footnote{代码见\url{https://github.com/THUDM/BatchSampler}}来采样难以区分（即互为困难负例与真实负例）的实例。为减少每个批量中的假负例，我们基于随机选取的实例构建邻近图；进而利用重启随机游走从图中采样难以区分的实例形成批量。BatchSampler作为通用轻量技术，可直接嵌入视觉、语言及图领域的现有对比学习模型。在三类模态数据集上的大量实验表明，该采样器能持续提升强对比模型的性能：在ImageNet-100上显著增强SimCLR，在STS文本数据上改进SimCSE，在图数据集上优化GraphCL与MVGRL。