We present MOFI, a new vision foundation model designed to learn image representations from noisy entity annotated images. MOFI differs from previous work in two key aspects: ($i$) pre-training data, and ($ii$) training recipe. Regarding data, we introduce a new approach to automatically assign entity labels to images from noisy image-text pairs. Our approach involves employing a named entity recognition model to extract entities from the alt-text, and then using a CLIP model to select the correct entities as labels of the paired image. The approach is simple, does not require costly human annotation, and can be readily scaled up to billions of image-text pairs mined from the web. Through this method, we have created Image-to-Entities (I2E), a new large-scale dataset with 1 billion images and 2 million distinct entities, covering rich visual concepts in the wild. Building upon the I2E dataset, we study different training recipes, including supervised pre-training, contrastive pre-training, and multi-task learning. For constrastive pre-training, we treat entity names as free-form text, and further enrich them with entity descriptions. Experiments show that supervised pre-training with large-scale fine-grained entity labels is highly effective for image retrieval tasks, and multi-task training further improves the performance. The final MOFI model achieves 86.66% mAP on the challenging GPR1200 dataset, surpassing the previous state-of-the-art performance of 72.19% from OpenAI's CLIP model. Further experiments on zero-shot and linear probe image classification also show that MOFI outperforms a CLIP model trained on the original image-text data, demonstrating the effectiveness of the I2E dataset in learning strong image representations.

翻译：我们提出MOFI，一种从含噪实体标注图像中学习图像表示的新型视觉基础模型。MOFI在以下两个关键方面与以往工作有所不同：（i）预训练数据，以及（ii）训练策略。在数据方面，我们提出一种新方法，能够从含噪图像-文本对中自动为图像分配实体标签。该方法首先使用命名实体识别模型从替代文本（alt-text）中提取实体，然后利用CLIP模型选择正确的实体作为配对图像的标签。该方法简单易行，无需昂贵的人工标注，并可轻松扩展至从网络挖掘的数十亿图像-文本对。通过该方法，我们构建了Image-to-Entities（I2E）数据集——一个包含10亿张图像和200万不同实体的全新大规模数据集，覆盖了丰富的开放域视觉概念。基于I2E数据集，我们研究了不同的训练策略，包括监督预训练、对比预训练和多任务学习。在对比预训练中，我们将实体名称视为自由文本，并进一步利用实体描述进行丰富。实验表明，采用大规模细粒度实体标签的监督预训练对图像检索任务非常有效，而多任务训练则能进一步提升性能。最终，MOFI模型在具有挑战性的GPR1200数据集上实现了86.66%的平均精度（mAP），超越了OpenAI的CLIP模型此前72.19%的最优性能。此外，在零样本和线性探针图像分类任务中的实验也显示，MOFI优于在原始图像-文本数据上训练的CLIP模型，证实了I2E数据集在学习强图像表示方面的有效性。