Image Captioning is a traditional vision-and-language task that aims to generate the language description of an image. Recent studies focus on scaling up the model size and the number of training data, which significantly increase the cost of model training. Different to these heavy-cost models, we introduce a lightweight image captioning framework (I-Tuning), which contains a small number of trainable parameters. We design a novel I-Tuning cross-attention module to connect the non-trainable pre-trained language decoder GPT2 and vision encoder CLIP-ViT. Since most parameters are not required to be updated during training, our framework is lightweight and fast. Experimental results conducted on three image captioning benchmarks reveal that our framework achieves comparable or better performance than the large-scale baseline systems. But our models contain up to 10 times fewer trainable parameters and require much fewer data for training compared with state-of-the-art baselines.

翻译：图像描述是一项传统的视觉-语言任务，旨在生成图像的文本描述。近期研究聚焦于扩大模型规模及训练数据量，这显著增加了模型训练成本。不同于这些高成本模型，我们提出了一种轻量级图像描述框架（I-Tuning），其仅包含少量可训练参数。我们设计了一种新颖的I-Tuning交叉注意力模块，用于连接非可训练的预训练语言解码器GPT2与视觉编码器CLIP-ViT。由于大部分参数在训练过程中无需更新，本框架兼具轻量性与高效性。在三个图像描述基准数据集上的实验结果表明，我们的框架能够达到与大规模基线系统相当甚至更优的性能。但与当前最优基线方法相比，我们的模型可训练参数数量减少至十分之一，训练所需数据量也大幅降低。