Many Contrastive Learning (CL) methods train their models to be invariant to different "views" of an image input for which a good data augmentation pipeline is crucial. While considerable efforts were directed towards improving pre-text tasks, architectures, or robustness (e.g., Siamese networks or teacher-softmax centering), the majority of these methods remain strongly reliant on the random sampling of operations within the image augmentation pipeline, such as the random resized crop or color distortion operation. In this paper, we argue that the role of the view generation and its effect on performance has so far received insufficient attention. To address this, we propose an easy, learning-free, yet powerful Hard View Selection (HVS) strategy designed to extend the random view generation to expose the pretrained model to harder samples during CL training. It encompasses the following iterative steps: 1) randomly sample multiple views and create pairs of two views, 2) run forward passes for each view pair on the currently trained model, 3) adversarially select the pair yielding the worst loss, and 4) run the backward pass with the selected pair. In our empirical analysis we show that under the hood, HVS increases task difficulty by controlling the Intersection over Union of views during pretraining. With only 300-epoch pretraining, HVS is able to closely rival the 800-epoch DINO baseline which remains very favorable even when factoring in the slowdown induced by the additional forwards of HVS. Additionally, HVS consistently achieves accuracy improvements on ImageNet between 0.4% and 1.9% on linear evaluation and similar improvements on transfer tasks across multiple CL methods, such as DINO, SimSiam, and SimCLR.

翻译：许多对比学习（CL）方法通过训练模型对图像输入的不同“视图”保持不变性，而良好的数据增强管道对此至关重要。尽管大量研究致力于改进预文本任务、架构或鲁棒性（例如，孪生网络或教师-softmax中心化），但大多数方法仍高度依赖于图像增强管道中操作的随机采样，如随机裁剪或颜色失真操作。本文认为，视图生成的作用及其对性能的影响至今尚未得到充分关注。为此，我们提出一种简单、无学习过程且高效的硬视图选择（HVS）策略，旨在扩展随机视图生成过程，使预训练模型在CL训练中暴露于更难的样本。该策略包括以下迭代步骤：1）随机采样多个视图并生成两两视图对；2）对当前训练模型上的每个视图对执行前向传播；3）对抗性地选择产生最差损失的视图对；4）利用所选视图对执行反向传播。我们的实证分析表明，HVS通过控制预训练期间视图的交并比（IoU）来增加任务难度。仅需300轮预训练，HVS即可接近800轮DINO基线的性能，即便考虑HVS额外前向传播带来的减速，这一结果依然非常有利。此外，HVS在线性评估中始终在ImageNet上提升0.4%至1.9%的准确率，并在多种CL方法（如DINO、SimSiam和SimCLR）的迁移任务上实现类似改进。