Both masked image modeling (MIM) and natural language supervision have facilitated the progress of transferable visual pre-training. In this work, we seek the synergy between two paradigms and study the emerging properties when MIM meets natural language supervision. To this end, we present a novel masked visual Reconstruction In Language semantic Space (RILS) pre-training framework, in which sentence representations, encoded by the text encoder, serve as prototypes to transform the vision-only signals into patch-sentence probabilities as semantically meaningful MIM reconstruction targets. The vision models can therefore capture useful components with structured information by predicting proper semantic of masked tokens. Better visual representations could, in turn, improve the text encoder via the image-text alignment objective, which is essential for the effective MIM target transformation. Extensive experimental results demonstrate that our method not only enjoys the best of previous MIM and CLIP but also achieves further improvements on various tasks due to their mutual benefits. RILS exhibits advanced transferability on downstream classification, detection, and segmentation, especially for low-shot regimes. Code will be made available at https://github.com/hustvl/RILS.

翻译：掩码图像建模（MIM）与自然语言监督共同促进了可迁移视觉预训练的发展。本文探索两种范式的协同作用，研究MIM与自然语言监督相遇时涌现的特性。为此，我们提出一种新颖的掩码视觉重建框架RILS（Reconstruction In Language Semantic Space）。该框架将文本编码器编码的句子表示作为原型，将纯视觉信号转化为补丁-句子概率，作为具有语义意义的MIM重建目标。通过预测掩码token的适当语义，视觉模型能够捕获具有结构化信息的有用组件。更优的视觉表征又能通过图像-文本对齐目标反过来改进文本编码器，这对实现有效的MIM目标转换至关重要。大量实验结果表明，我们的方法不仅兼具MIM和CLIP的最佳特性，更因二者的相互增益而在各类任务上取得进一步提升。RILS在下游分类、检测和分割任务中展现出优越的可迁移性，尤其在低样本场景下表现突出。代码将开源至https://github.com/hustvl/RILS。