Recent advances in image captioning have focused on scaling the data and model size, substantially increasing the cost of pre-training and finetuning. As an alternative to large models, we present SmallCap, which generates a caption conditioned on an input image and related captions retrieved from a datastore. Our model is lightweight and fast to train, as the only learned parameters are in newly introduced cross-attention layers between a pre-trained CLIP encoder and GPT-2 decoder. SmallCap can transfer to new domains without additional finetuning and can exploit large-scale data in a training-free fashion since the contents of the datastore can be readily replaced. Our experiments show that SmallCap, trained only on COCO, has competitive performance on this benchmark, and also transfers to other domains without retraining, solely through retrieval from target-domain data. Further improvement is achieved through the training-free exploitation of diverse human-labeled and web data, which proves to be effective for a range of domains, including the nocaps benchmark, designed to test generalization to unseen visual concepts.

翻译：近期图像描述生成的进展主要集中于扩大数据和模型规模，显著提升了预训练与微调的成本。作为大型模型的替代方案，我们提出了SmallCap，该方法基于输入图像及从数据存储中检索到的相关描述生成句子。我们的模型轻量且训练快速，唯一需要学习的参数是预训练CLIP编码器与GPT-2解码器之间新增的交叉注意力层。SmallCap无需额外微调即可迁移至新领域，且由于数据存储内容可便捷替换，能够以无训练方式利用大规模数据。实验表明，仅在COCO数据集上训练的SmallCap，在该基准测试上表现出竞争力，并且仅通过检索目标领域数据即可实现域迁移而无需重训练。通过无训练方式利用多样化人工标注数据及网络数据，进一步提升了性能——这对于包含nocaps基准测试在内的多个领域（该基准专为测试对未见视觉概念的泛化能力而设计）均证实有效。