Many Self-Supervised Learning (SSL) methods train their models to be invariant to different "views" of an image input for which a good data augmentation pipeline is crucial. While considerable efforts were directed towards improving pre-text tasks, architectures, or robustness (e.g., Siamese networks or teacher-softmax centering), the majority of these methods remain strongly reliant on the random sampling of operations within the image augmentation pipeline, such as the random resized crop or color distortion operation. In this paper, we argue that the role of the view generation and its effect on performance has so far received insufficient attention. To address this, we propose an easy, learning-free, yet powerful Hard View Selection (HVS) strategy designed to extend the random view generation to expose the pretrained model to harder samples during SSL training. It encompasses the following iterative steps: 1) randomly sample multiple views and create pairs of two views, 2) run forward passes for each view pair on the currently trained model, 3) adversarially select the pair yielding the worst loss, and 4) run the backward pass with the selected pair. In our empirical analysis we show that under the hood, HVS increases task difficulty by controlling the Intersection over Union of views during pretraining. With only 300-epoch pretraining, HVS is able to closely rival the 800-epoch DINO baseline which remains very favorable even when factoring in the slowdown induced by the additional forwards of HVS. Additionally, HVS consistently achieves accuracy improvements on ImageNet between 0.4% and 1.9% on linear evaluation and similar improvements on transfer tasks across multiple SSL methods, such as DINO, SimSiam, iBOT, and SimCLR.

翻译：许多自监督学习（SSL）方法训练模型对不同图像输入的"视图"保持不变性，其中高质量的数据增强流程至关重要。尽管研究者们在改进预文本任务、架构或鲁棒性（例如孪生网络或教师softmax中心化）方面投入了大量努力，但大多数方法仍强烈依赖于图像增强流程中操作的随机采样，例如随机裁剪或颜色失真操作。本文认为，视图生成的作用及其对性能的影响至今未得到充分关注。为此，我们提出一种简单、无需学习的硬视图选择（HVS）策略，旨在扩展随机视图生成过程，使SSL训练中的预训练模型接触更困难的样本。该策略包含以下迭代步骤：1）随机采样多个视图并生成视图对；2）对当前训练模型的每个视图对执行前向传播；3）对抗性选择产生最差损失的视图对；4）使用所选视图对执行反向传播。实证分析表明，HVS通过控制预训练过程中视图的交并比来增加任务难度。仅需300轮预训练，HVS就能与800轮DINO基线性能相当，即使在考虑HVS额外前向传播带来的减速后，这一优势依然显著。此外，在ImageNet线性评估中，HVS在不同SSL方法（如DINO、SimSiam、iBOT和SimCLR）上持续提升0.4%至1.9%的准确率，并在迁移任务中表现出类似改进。