This paper presents Tag2Text, a vision language pre-training (VLP) framework, which introduces image tagging into vision-language models to guide the learning of visual-linguistic features. In contrast to prior works which utilize object tags either manually labeled or automatically detected with an off-the-shelf detector with limited performance, our approach explicitly learns an image tagger using tags parsed from image-paired text and thus provides a strong semantic guidance to vision-language models. In this way, Tag2Text can utilize large-scale annotation-free image tags in accordance with image-text pairs, and provides more diverse tag categories beyond objects. As a result, Tag2Text demonstrates the ability of a foundational image tagging model, with superior zero-shot performance even comparable to fully supervised models. Moreover, by leveraging the tagging guidance, Tag2Text effectively enhances the performance of vision-language models on both generation-based and alignment-based tasks. Across a wide range of downstream benchmarks, Tag2Text achieves state-of-the-art results with similar model sizes and data scales, demonstrating the efficacy of the proposed tagging guidance. Code, demo and pre-trained models are available at \url{https://github.com/xinyu1205/recognize-anything}.

翻译：本文提出Tag2Text，一种视觉语言预训练框架，通过将图像标记引入视觉语言模型以引导视觉-语言特征的学习。与先前利用手动标注或借助性能有限的现成检测器自动获取对象标签的方法不同，本方法明确学习一个图像标记器，利用从图像配对文本中解析的标签生成标记，从而为视觉语言模型提供强语义引导。通过这种方式，Tag2Text能够利用与图像-文本对一致的大规模无标注图像标记，并提供超越对象类别的更多样化标签类别。实验表明，Tag2Text展示了基础图像标记模型的能力，其零样本性能甚至可与全监督模型相媲美。此外，借助标记引导，Tag2Text有效提升了视觉语言模型在生成型和对齐型任务上的表现。在多种下游基准测试中，Tag2Text在相似模型规模和数据规模下取得了最先进的结果，验证了所提标记引导的有效性。代码、演示和预训练模型可从\url{https://github.com/xinyu1205/recognize-anything}获取。