Self supervision and natural language supervision have emerged as two exciting ways to train general purpose image encoders which excel at a variety of downstream tasks. Recent works such as M3AE and SLIP have suggested that these approaches can be effectively combined, but most notably their results use small pre-training datasets (<50M samples) and don't effectively reflect the large-scale regime (>100M examples) that is commonly used for these approaches. Here we investigate whether a similar approach can be effective when trained with a much larger amount of data. We find that a combination of two state of the art approaches: masked auto-encoders, MAE and contrastive language image pre-training, CLIP provides a benefit over CLIP when trained on a corpus of 11.3M image-text pairs, but little to no benefit (as evaluated on a suite of common vision tasks) over CLIP when trained on a large corpus of 1.4B images. Our work provides some much needed clarity into the effectiveness (or lack thereof) of self supervision for large-scale image-text training.

翻译：自监督和自然语言监督已成为训练通用图像编码器的两种激动人心的方式，这些编码器在各种下游任务中表现出色。近期如M3AE和SLIP等研究表明，这些方法可以有效结合，但值得注意的是，它们的结果使用了小型预训练数据集（<5000万样本），并未有效反映这些方法通常使用的大规模场景（>1亿样本）。本文研究了在更大数据量训练下类似方法是否有效。我们发现，将两种先进方法——掩码自编码器（MAE）和对比语言-图像预训练（CLIP）结合，在1130万图像-文本对语料上训练时较CLIP有提升，但在14亿图像的大规模语料上训练时，相较于CLIP几乎无增益（在一系列常见视觉任务上评估）。我们的工作为大规模图像-文本训练中自监督的有效性（或无效性）提供了亟需的清晰认识。