Recent advancements in image captioning have explored text-only training methods to overcome the limitations of paired image-text data. However, existing text-only training methods often overlook the modality gap between using text data during training and employing images during inference. To address this issue, we propose a novel approach called Image-like Retrieval, which aligns text features with visually relevant features to mitigate the modality gap. Our method further enhances the accuracy of generated captions by designing a Fusion Module that integrates retrieved captions with input features. Additionally, we introduce a Frequency-based Entity Filtering technique that significantly improves caption quality. We integrate these methods into a unified framework, which we refer to as IFCap ($\textbf{I}$mage-like Retrieval and $\textbf{F}$requency-based Entity Filtering for Zero-shot $\textbf{Cap}$tioning). Through extensive experimentation, our straightforward yet powerful approach has demonstrated its efficacy, outperforming the state-of-the-art methods by a significant margin in both image captioning and video captioning compared to zero-shot captioning based on text-only training.

翻译：近年来，图像描述生成领域已开始探索纯文本训练方法，以克服成对图像-文本数据的局限性。然而，现有的纯文本训练方法往往忽视了训练阶段使用文本数据与推理阶段使用图像数据之间的模态差异。为解决这一问题，我们提出了一种称为类图像检索的新方法，该方法通过将文本特征与视觉相关特征对齐来缓解模态差异。我们进一步设计了一个融合模块，将检索到的描述与输入特征进行整合，从而提升了生成描述的准确性。此外，我们引入了一种基于频率的实体过滤技术，显著改善了描述质量。我们将这些方法集成到一个统一框架中，称之为IFCap（面向零样本描述的类图像检索与基于频率的实体过滤）。通过大量实验，我们这种简洁而强大的方法证明了其有效性，在基于纯文本训练的零样本描述任务中，无论是图像描述还是视频描述，其性能均显著超越了现有最优方法。