We present MOFI, a new vision foundation model designed to learn image representations from noisy entity annotated images. MOFI differs from previous work in two key aspects: ($i$) pre-training data, and ($ii$) training recipe. Regarding data, we introduce a new approach to automatically assign entity labels to images from noisy image-text pairs. Our approach involves employing a named entity recognition model to extract entities from the alt-text, and then using a CLIP model to select the correct entities as labels of the paired image. The approach is simple, does not require costly human annotation, and can be readily scaled up to billions of image-text pairs mined from the web. Through this method, we have created Image-to-Entities (I2E), a new large-scale dataset with 1 billion images and 2 million distinct entities, covering rich visual concepts in the wild. Building upon the I2E dataset, we study different training recipes, including supervised pre-training, contrastive pre-training, and multi-task learning. For constrastive pre-training, we treat entity names as free-form text, and further enrich them with entity descriptions. Experiments show that supervised pre-training with large-scale fine-grained entity labels is highly effective for image retrieval tasks, and multi-task training further improves the performance. The final MOFI model achieves 86.66% mAP on the challenging GPR1200 dataset, surpassing the previous state-of-the-art performance of 72.19% from OpenAI's CLIP model. Further experiments on zero-shot and linear probe image classification also show that MOFI outperforms a CLIP model trained on the original image-text data, demonstrating the effectiveness of the I2E dataset in learning strong image representations.

翻译：我们提出MOFI，一种旨在从含噪实体标注图像中学习图像表示的新型视觉基础模型。MOFI在以下两个关键方面与先前工作不同：（i）预训练数据，以及（ii）训练方案。关于数据方面，我们引入了一种新方法，可自动从含噪图文对中为图像分配实体标签。该方法采用命名实体识别模型从替代文本中提取实体，然后使用CLIP模型选择正确的实体作为配对图像的标签。该方法简单，无需昂贵的人工标注，并可轻松扩展至从互联网挖掘的数十亿图文对。通过此方法，我们创建了Image-to-Entities (I2E)，一个包含10亿张图像和200万个不同实体的新大规模数据集，覆盖了丰富的开放世界视觉概念。基于I2E数据集，我们研究了不同的训练方案，包括监督预训练、对比预训练和多任务学习。对于对比预训练，我们将实体名称视为自由格式文本，并进一步用实体描述进行丰富。实验表明，使用大规模细粒度实体标签进行监督预训练对图像检索任务极为有效，而多任务训练则进一步提升了性能。最终MOFI模型在具有挑战性的GPR1200数据集上达到86.66%的mAP，超越了先前OpenAI CLIP模型72.19%的最佳性能。在零样本和线性探测图像分类上的进一步实验也表明，MOFI优于在原始图文数据上训练的CLIP模型，证明了I2E数据集在学习强图像表示方面的有效性。