Large-scale pre-trained multi-modal models (e.g., CLIP) demonstrate strong zero-shot transfer capability in many discriminative tasks. Their adaptation to zero-shot image-conditioned text generation tasks has drawn increasing interest. Prior arts approach to zero-shot captioning by either utilizing the existing large language models (e.g., GPT-2) or pre-training the encoder-decoder network in an end-to-end manner. In this work, we propose a simple framework, named DeCap, for zero-shot captioning. We introduce a lightweight visual-aware language decoder. This decoder is both data-efficient and computation-efficient: 1) it only requires the text data for training, easing the burden on the collection of paired data. 2) it does not require end-to-end training. When trained with text-only data, the decoder takes the text embedding extracted from the off-the-shelf CLIP encoder as a prefix embedding. The challenge is that the decoder is trained on the text corpus but at the inference stage, it needs to generate captions based on visual inputs. The modality gap issue is widely observed in multi-modal contrastive models that prevents us from directly taking the visual embedding as the prefix embedding. We propose a training-free mechanism to reduce the modality gap. We project the visual embedding into the CLIP text embedding space, while the projected embedding retains the information of the visual input. Taking the projected embedding as the prefix embedding, the decoder generates high-quality descriptions that match the visual input. The experiments show that DeCap outperforms other zero-shot captioning methods and unpaired captioning methods on the typical image captioning benchmarks, i.e., MSCOCO and NoCaps.

翻译：大规模预训练多模态模型（如CLIP）在众多判别任务中展现出强大的零样本迁移能力，其向零样本图像条件文本生成任务的适配性正引起日益关注。现有零样本描述生成方法或利用现有大语言模型（如GPT-2），或以端到端方式预训练编码器-解码器网络。本文提出名为DeCap的零样本描述生成简易框架，引入轻量级视觉感知语言解码器。该解码器兼具数据高效性与计算高效性：1）仅需文本数据训练，缓解成对数据采集负担；2）无需端到端训练。在仅使用文本数据训练时，解码器将来自现成CLIP编码器提取的文本嵌入作为前缀嵌入。面临挑战在于：解码器虽在文本语料上训练，但推理阶段需基于视觉输入生成描述。多模态对比模型中广泛存在的模态差距问题，阻碍了直接将视觉嵌入作为前缀嵌入。为此，我们提出无训练机制缩减模态差距：将视觉嵌入投影到CLIP文本嵌入空间，同时投影嵌入保留视觉输入信息。以投影嵌入作为前缀嵌入，解码器可生成与视觉输入匹配的高质量描述。实验表明，在典型图像描述生成基准数据集MSCOCO与NoCaps上，DeCap性能优于现有零样本及非配对描述生成方法。