This paper presents PaLI-3, a smaller, faster, and stronger vision language model (VLM) that compares favorably to similar models that are 10x larger. As part of arriving at this strong performance, we compare Vision Transformer (ViT) models pretrained using classification objectives to contrastively (SigLIP) pretrained ones. We find that, while slightly underperforming on standard image classification benchmarks, SigLIP-based PaLI shows superior performance across various multimodal benchmarks, especially on localization and visually-situated text understanding. We scale the SigLIP image encoder up to 2 billion parameters, and achieves a new state-of-the-art on multilingual cross-modal retrieval. We hope that PaLI-3, at only 5B parameters, rekindles research on fundamental pieces of complex VLMs, and could fuel a new generation of scaled-up models.

翻译：本文提出PaLI-3——一种更小、更快且更强的视觉语言模型（VLM），其性能与同类大10倍的模型相比毫不逊色。在实现这一强大性能的过程中，我们对比了使用分类目标预训练的Vision Transformer（ViT）模型与基于对比学习（SigLIP）预训练的模型。研究发现，尽管SigLIP在标准图像分类基准上表现略逊一筹，但基于SigLIP的PaLI在多种多模态基准测试中展现出卓越性能，尤其在定位和视觉文本理解任务上表现突出。我们将SigLIP图像编码器扩展至20亿参数，并在多语言跨模态检索任务上达到新的最优水平。我们期望仅拥有50亿参数的PaLI-3能够重燃对复杂VLM基础组件的研究，并为新一代规模化模型的研发注入动力。