Current captioning approaches tend to generate correct but "generic" descriptions that lack real-world knowledge, e.g., named entities and contextual information. Considering that Vision-Language Pre-Training (VLP) models master massive such knowledge from large-scale web-harvested data, it is promising to utilize the generalizability of VLP models to incorporate knowledge into image descriptions. However, using VLP models faces challenges: zero-shot inference suffers from knowledge hallucination that leads to low-quality descriptions, but the generic bias in downstream task fine-tuning hinders the VLP model from expressing knowledge. To address these concerns, we propose a simple yet effective method called Knowledge-guided Replay (K-Replay), which enables the retention of pre-training knowledge during fine-tuning. Our approach consists of two parts: (1) a knowledge prediction task on automatically collected replay exemplars to continuously awaken the VLP model's memory about knowledge, thus preventing the model from collapsing into the generic pattern; (2) a knowledge distillation constraint to improve the faithfulness of generated descriptions hence alleviating the knowledge hallucination. To evaluate knowledge-enhanced descriptions, we construct a novel captioning benchmark KnowCap, containing knowledge of landmarks, famous brands, special foods and movie characters. Experimental results show that our approach effectively incorporates knowledge into descriptions, outperforming strong VLP baseline by 20.9 points (78.7->99.6) in CIDEr score and 20.5 percentage points (34.0%->54.5%) in knowledge recognition accuracy. Our code and data is available at https://github.com/njucckevin/KnowCap.

翻译：当前图像描述方法往往生成正确但“通用”的描述，缺乏真实世界知识，如命名实体和上下文信息。考虑到视觉-语言预训练（VLP）模型从大规模网络采集数据中掌握了大量此类知识，利用VLP模型的泛化能力将知识融入图像描述具有广阔前景。然而，使用VLP模型面临挑战：零样本推理会产生知识幻觉，导致低质量描述；但下游任务微调中的通用偏差会阻碍VLP模型表达知识。为解决这些问题，我们提出一种简单而有效的方法——知识引导回放（K-Replay），能够在微调过程中保留预训练知识。该方法包含两部分：（1）在自动收集的回放样本上进行知识预测任务，持续唤醒VLP模型对知识的记忆，从而防止模型陷入通用模式；（2）采用知识蒸馏约束提高生成描述的忠实度，进而缓解知识幻觉。为评估知识增强型描述，我们构建了新型描述基准KnowCap，包含地标、知名品牌、特色美食和电影角色等知识。实验结果表明，我们的方法有效将知识融入描述，在CIDEr分数上超越强大VLP基线20.9个百分点（78.7→99.6），知识识别准确率提升20.5个百分点（34.0%→54.5%）。代码与数据开源地址：https://github.com/njucckevin/KnowCap。