Many Contrastive Learning (CL) methods train their models to be invariant to different "views" of an image input for which a good data augmentation pipeline is crucial. While considerable efforts were directed towards improving pre-text tasks, architectures, or robustness (e.g., Siamese networks or teacher-softmax centering), the majority of these methods remain strongly reliant on the random sampling of operations within the image augmentation pipeline, such as the random resized crop or color distortion operation. In this paper, we argue that the role of the view generation and its effect on performance has so far received insufficient attention. To address this, we propose an easy, learning-free, yet powerful Hard View Selection (HVS) strategy designed to extend the random view generation to expose the pretrained model to harder samples during CL training. It encompasses the following iterative steps: 1) randomly sample multiple views and create pairs of two views, 2) run forward passes for each view pair on the currently trained model, 3) adversarially select the pair yielding the worst loss, and 4) run the backward pass with the selected pair. In our empirical analysis we show that under the hood, HVS increases task difficulty by controlling the Intersection over Union of views during pretraining. With only 300-epoch pretraining, HVS is able to closely rival the 800-epoch DINO baseline which remains very favorable even when factoring in the slowdown induced by the additional forwards of HVS. Additionally, HVS consistently achieves accuracy improvements on ImageNet between 0.55% and 1.9% on linear evaluation and similar improvements on transfer tasks across multiple CL methods, such as DINO, SimSiam, and SimCLR.

翻译：许多对比学习方法通过训练模型对不同“视图”的输入图像保持不变性，其中良好的数据增强流程至关重要。尽管大量研究致力于改进预文本任务、架构或鲁棒性（如孪生网络或教师-软最大化中心化），大多数方法仍高度依赖图像增强流程中操作的随机采样，例如随机裁剪或颜色失真操作。本文认为，视图生成的作用及其对性能的影响迄今未得到足够关注。为此，我们提出一种简单、无需学习且高效的硬视图选择策略，旨在扩展随机视图生成，以在对比学习训练中向预训练模型暴露更难的样本。该策略包含以下迭代步骤：1）随机采样多个视图并构建视图对；2）对当前训练模型的每个视图对执行前向传播；3）对抗性选择损失最差的视图对；4）使用所选视图对执行反向传播。实证分析表明，HVS通过控制预训练过程中视图的交并比来增加任务难度。仅需300轮预训练，HVS即可接近800轮DINO基线的性能，即使考虑HVS额外前向传播导致的减速，该结果仍具有竞争力。此外，HVS在ImageNet线性评估上持续提升0.55%至1.9%的准确率，并在DINO、SimSiam和SimCLR等多种对比学习方法的迁移任务中实现类似改进。