Image-language learning has made unprecedented progress in visual understanding. These developments have come at high costs, as contemporary vision-language models require large model scales and amounts of data. We here propose a much easier recipe for image-language learning, which produces effective models, outperforming bigger and more expensive ones, often trained on orders of magnitude larger datasets. Our key finding is the joint learning of a compact vision and language representation, which adaptively and iteratively fuses the multi-modal features. This results in a more effective image-language learning, greatly lowering the FLOPs by combining and reducing the number of tokens for both text and images, e.g. a 33\% reduction in FLOPs is achieved, compared to baseline fusion techniques used by popular image-language models, while improving performance. This also allows the model to scale without a large increase in FLOPs or memory. In addition, we propose adaptive pre-training data sampling which improves the data efficiency. The proposed approach achieves competitive performance compared to much larger models, and does so with significantly less data and FLOPs. With only 40M training examples and with 39 GFLOPs our lightweight model outperforms many times larger state-of-the-art models of 2-20x more FLOPs and using bigger datasets some of which with close to 1B training examples.

翻译：图像-语言学习在视觉理解领域取得了前所未有的进展。然而，这些发展伴随着高昂的成本，当今的视觉-语言模型需要庞大的模型规模和海量数据。本文提出一种更简洁的图像-语言学习方案，能够产生有效的模型，且性能优于规模更大、成本更高的模型（这些模型通常基于数量级更大的数据集进行训练）。关键发现在于联合学习紧凑的视觉与语言表征，以自适应且迭代的方式融合多模态特征。这实现了更高效的图像-语言学习，通过合并并减少文本与图像的令牌数量，大幅降低浮点运算次数（FLOPs）。例如，与主流图像-语言模型的基线融合技术相比，本方法在提升性能的同时实现了33%的FLOPs缩减。此外，该方案还使模型能够在不显著增加FLOPs或内存需求的前提下进行扩展。我们进一步提出自适应预训练数据采样方法，以提升数据效率。所提方法在显著降低数据量和FLOPs的情况下，达到了与更大模型相媲美的竞争性能。仅凭4000万个训练样本和39 GFLOPs，我们的轻量级模型即可超越许多规模更大的最先进模型——这些模型所需FLOPs高达2-20倍，且部分使用接近于10亿的训练样本。