Contrastive self-supervised learning methods famously produce high quality transferable representations by learning invariances to different data augmentations. Invariances established during pre-training can be interpreted as strong inductive biases. However these may or may not be helpful, depending on if they match the invariance requirements of downstream tasks or not. This has led to several attempts to learn task-specific invariances during pre-training, however, these methods are highly compute intensive and tedious to train. We introduce the notion of amortised invariance learning for contrastive self supervision. In the pre-training stage, we parameterize the feature extractor by differentiable invariance hyper-parameters that control the invariances encoded by the representation. Then, for any downstream task, both linear readout and task-specific invariance requirements can be efficiently and effectively learned by gradient-descent. We evaluate the notion of amortised invariances for contrastive learning over two different modalities: vision and audio, on two widely-used contrastive learning methods in vision: SimCLR and MoCo-v2 with popular architectures like ResNets and Vision Transformers, and SimCLR with ResNet-18 for audio. We show that our amortised features provide a reliable way to learn diverse downstream tasks with different invariance requirements, while using a single feature and avoiding task-specific pre-training. This provides an exciting perspective that opens up new horizons in the field of general purpose representation learning.

翻译：对比自监督学习方法通过学习对不同数据增强的不变性，以产生高质量的可迁移表示而闻名。预训练过程中建立的不变性可被视作强归纳偏置。然而，这些偏置可能有益，也可能无益，具体取决于它们是否与下游任务的不变性需求相匹配。这引发了若干在预训练期间学习任务特定不变性的尝试，但这些方法计算密集且训练繁琐。我们提出用于对比自监督学习的摊销不变性学习概念。在预训练阶段，我们通过可微分的不变性超参数对特征提取器进行参数化，这些超参数控制表示所编码的不变性。随后，对于任何下游任务，线性读出和任务特定的不变性需求均可通过梯度下降高效且有效地学习。我们针对两种不同模态（视觉和音频）评估了对比学习的摊销不变性概念：在视觉领域采用两种广泛使用的对比学习方法（SimCLR和MoCo-v2）与主流架构（如ResNets和Vision Transformers），在音频领域采用SimCLR与ResNet-18。结果表明，我们的摊销特征提供了一种可靠方式，可使用单一特征学习具有不同不变性需求的多样化下游任务，同时避免了任务特定的预训练。这为通用表示学习领域开辟了令人兴奋的新视野。