Image-language learning has made unprecedented progress in visual understanding. These developments have come at high costs, as contemporary vision-language models require large model scales and amounts of data. We here propose a much easier recipe for image-language learning, which produces effective models, outperforming bigger and more expensive ones, often trained on orders of magnitude larger datasets. Our key finding is the joint learning of a compact vision and language representation, which adaptively and iteratively fuses the multi-modal features. This results in a more effective image-language learning, greatly lowering the FLOPs by combining and reducing the number of tokens for both text and images, e.g. a 33\% reduction in FLOPs is achieved, compared to baseline fusion techniques used by popular image-language models, while improving performance. This also allows the model to scale without a large increase in FLOPs or memory. In addition, we propose adaptive pre-training data sampling which improves the data efficiency. The proposed approach achieves competitive performance compared to much larger models, and does so with significantly less data and FLOPs. With only 40M training examples and with 39 GFLOPs our lightweight model outperforms many times larger state-of-the-art models of 2-20x more FLOPs and using bigger datasets some of which with close to 1B training examples.

翻译：图像-语言学习在视觉理解领域取得了前所未有的进展。然而，这些进展依赖于高昂的成本——当代视觉语言模型需要庞大的模型规模和海量数据。本文提出一种更为简洁的图像-语言学习方法，能够生成有效模型，且性能超越规模更大、成本更高（通常基于数量级更大数据集训练）的模型。关键创新在于联合学习紧凑的视觉与语言表示，通过自适应迭代融合多模态特征。该方法显著提升了图像-语言学习效率，通过整合并缩减文本与图像标记数量大幅降低FLOPs，例如相比主流图像语言模型采用的基线融合技术，FLOPs可降低33%，同时性能得到提升。此特性使模型能够在FLOPs或内存无大幅增长的情况下进行扩展。此外，我们提出自适应预训练数据采样方法以提升数据效率。所提方法在显著减少数据和计算量的前提下，性能堪比规模更大的模型。仅需4000万训练样本和39 GFLOPs的轻量级模型，即可超越计算量需求高2-20倍、部分数据集规模接近10亿样本的现有先进模型。