Supervised visual captioning models typically require a large scale of images or videos paired with descriptions in a specific language (i.e., the vision-caption pairs) for training. However, collecting and labeling large-scale datasets is time-consuming and expensive for many scenarios and languages. Therefore, sufficient labeled pairs are usually not available. To deal with the label shortage problem, we present a simple yet effective zero-shot approach MultiCapCLIP that can generate visual captions for different scenarios and languages without any labeled vision-caption pairs of downstream datasets. In the training stage, MultiCapCLIP only requires text data for input. Then it conducts two main steps: 1) retrieving concept prompts that preserve the corresponding domain knowledge of new scenarios; 2) auto-encoding the prompts to learn writing styles to output captions in a desired language. In the testing stage, MultiCapCLIP instead takes visual data as input directly to retrieve the concept prompts to generate the final visual descriptions. The extensive experiments on image and video captioning across four benchmarks and four languages (i.e., English, Chinese, German, and French) confirm the effectiveness of our approach. Compared with state-of-the-art zero-shot and weakly-supervised methods, our method achieves 4.8% and 21.5% absolute improvements in terms of BLEU@4 and CIDEr metrics. Our code is available at https://github.com/yangbang18/MultiCapCLIP.

翻译：有监督的视觉描述模型通常需要大量图像或视频与特定语言描述的配对数据（即视觉-描述对）进行训练。然而，在许多场景和语言中，收集和标注大规模数据集既耗时又昂贵，导致往往无法获得充足的标注配对数据。针对标注短缺问题，我们提出一种简单而有效的零样本方法MultiCapCLIP，它无需下游数据集的任何标注视觉-描述对，即可为不同场景和语言生成视觉描述。在训练阶段，MultiCapCLIP仅需文本数据输入，然后执行两个主要步骤：1）检索能保留新场景相应领域知识的概念提示；2）对提示进行自编码以学习写作风格，从而输出目标语言的描述。在测试阶段，MultiCapCLIP直接以视觉数据为输入，检索概念提示以生成最终的视觉描述。在四个基准数据集和四种语言（英语、中文、德语和法语）上的图像与视频描述广泛实验证实了我们方法的有效性。与最先进的零样本和弱监督方法相比，我们的方法在BLEU@4和CIDEr指标上分别实现了4.8%和21.5%的绝对提升。代码已开源：https://github.com/yangbang18/MultiCapCLIP。