Image captioning is one of the straightforward tasks that can take advantage of large-scale web-crawled data which provides rich knowledge about the visual world for a captioning model. However, since web-crawled data contains image-text pairs that are aligned at different levels, the inherent noises (e.g., misaligned pairs) make it difficult to learn a precise captioning model. While the filtering strategy can effectively remove noisy data, it leads to a decrease in learnable knowledge and sometimes brings about a new problem of data deficiency. To take the best of both worlds, we propose a Noise-aware Captioning (NoC) framework, which learns rich knowledge from the whole web-crawled data while being less affected by the noises. This is achieved by the proposed alignment-level-controllable captioner, which is learned using alignment levels of the image-text pairs as a control signal during training. The alignment-level-conditioned training allows the model to generate high-quality captions by simply setting the control signal to the desired alignment level at inference time. An in-depth analysis shows the effectiveness of our framework in handling noise. With two tasks of zero-shot captioning and text-to-image retrieval using generated captions (i.e., self-retrieval), we also demonstrate our model can produce high-quality captions in terms of descriptiveness and distinctiveness. The code is available at \url{https://github.com/kakaobrain/noc}.

翻译：图像描述是一项直接受益于大规模网络爬取数据的任务，此类数据为描述模型提供了丰富的视觉世界知识。然而，由于网络爬取数据包含不同对齐程度的图文对，其中的固有噪声（例如错位对）使得难以学习到精确的描述模型。虽然过滤策略可有效移除噪声数据，但这会导致可学习知识减少，有时还会引发数据不足的新问题。为兼顾两者优势，我们提出噪声感知描述（NoC）框架，该框架能从整个网络爬取数据中学习丰富知识，同时减轻噪声影响。这一目标通过所提出的对齐级别可控描述器实现——该描述器在训练过程中利用图文对的对齐程度作为控制信号进行学习。基于对齐级别的条件训练使模型在推理时仅需将控制信号设为目标对齐程度，即可生成高质量描述。深入分析表明，我们的框架在处理噪声方面具有有效性。通过零样本描述和利用生成描述进行文本到图像检索（即自检索）两项任务，我们也证明了模型能够生成兼具描述性和区分性的高质量描述。代码开源于：\url{https://github.com/kakaobrain/noc}。