Many Self-Supervised Learning (SSL) methods train their models to be invariant to different "views" of an image input for which a good data augmentation pipeline is crucial. While considerable efforts were directed towards improving pre-text tasks, architectures, or robustness (e.g., Siamese networks or teacher-softmax centering), the majority of these methods remain strongly reliant on the random sampling of operations within the image augmentation pipeline, such as the random resized crop or color distortion operation. In this paper, we argue that the role of the view generation and its effect on performance has so far received insufficient attention. To address this, we propose an easy, learning-free, yet powerful Hard View Selection (HVS) strategy designed to extend the random view generation to expose the pretrained model to harder samples during SSL training. It encompasses the following iterative steps: 1) randomly sample multiple views and create pairs of two views, 2) run forward passes for each view pair on the currently trained model, 3) adversarially select the pair yielding the worst loss, and 4) run the backward pass with the selected pair. In our empirical analysis we show that under the hood, HVS increases task difficulty by controlling the Intersection over Union of views during pretraining. With only 300-epoch pretraining, HVS is able to closely rival the 800-epoch DINO baseline which remains very favorable even when factoring in the slowdown induced by the additional forwards of HVS. Additionally, HVS consistently achieves accuracy improvements on ImageNet between 0.4% and 1.9% on linear evaluation and similar improvements on transfer tasks across multiple SSL methods, such as DINO, SimSiam, iBOT, and SimCLR.

翻译：许多自监督学习方法通过让模型对图像输入的不同“视图”保持不变性进行训练，其中良好的数据增强流程至关重要。尽管已有大量研究致力于改进预文本任务、模型架构或鲁棒性（例如孪生网络或教师-softmax中心化），但多数方法仍高度依赖图像增强流程中操作的随机采样（如随机裁剪或颜色失真）。本文认为，视图生成的作用及其对性能的影响至今尚未得到足够重视。为此，我们提出一种简单、无需学习且高效的困难视图选择（Hard View Selection，HVS）策略，旨在扩展随机视图生成，使预训练模型在自监督训练中暴露于更困难的样本。该策略包含以下迭代步骤：1）随机采样多个视图并生成视图对；2）对每个视图对在当前模型上执行前向传播；3）通过对抗方式选择损失最差的视图对；4）使用所选视图对执行反向传播。实验分析表明，HVS通过控制预训练期间视图的交并比来提升任务难度。仅需300轮预训练，HVS即可接近800轮DINO基线的性能，即使考虑HVS额外前向传播带来的速度下降，该结果仍极具竞争力。此外，在ImageNet线性评估上，HVS可在多个自监督方法（如DINO、SimSiam、iBOT和SimCLR）上持续提升0.4%至1.9%的准确率，并在迁移任务上取得类似改进。