Image captioning is conventionally formulated as the task of generating captions for images that match the distribution of reference image-caption pairs. However, reference captions in standard captioning datasets are short and may not uniquely identify the images they describe. These problems are further exacerbated when models are trained directly on image-alt text pairs collected from the internet. In this work, we show that it is possible to generate more specific captions with minimal changes to the training process. We implement classifier-free guidance for an autoregressive captioning model by fine-tuning it to estimate both conditional and unconditional distributions over captions. The guidance scale applied at decoding controls a trade-off between maximizing $p(\mathrm{caption}|\mathrm{image})$ and $p(\mathrm{image}|\mathrm{caption})$. Compared to standard greedy decoding, decoding with a guidance scale of 2 substantially improves reference-free metrics such as CLIPScore (0.808 vs. 0.775) and caption$\to$image retrieval performance in the CLIP embedding space (recall@1 44.6% vs. 26.5%), but worsens standard reference-based captioning metrics (e.g., CIDEr 78.6 vs 126.1). We further explore the use of language models to guide the decoding process, obtaining small improvements over the Pareto frontier of reference-free vs. reference-based captioning metrics that arises from classifier-free guidance, and substantially improving the quality of captions generated from a model trained only on minimally curated web data.

翻译：图像描述通常被定义为为图像生成与参考图像-描述对分布相匹配的描述任务。然而，标准描述数据集中的参考描述通常较短，且可能无法唯一标识其描述的图像。当模型直接使用从互联网收集的图像-替代文本对进行训练时，这些问题会进一步加剧。在本研究中，我们证明可以通过最小化训练过程的改动来生成更具描述性的标题。我们通过微调自回归描述模型，使其同时估计描述的条件分布和无条件分布，从而实现无分类器引导。解码时应用的引导尺度控制了最大化$p(\text{描述}|\text{图像})$与$p(\text{图像}|\text{描述})$之间的权衡。与标准贪婪解码相比，使用引导尺度为2的解码显著提升了无参考指标（如CLIPScore从0.775提升至0.808）以及CLIP嵌入空间中的描述→图像检索性能（recall@1从26.5%提升至44.6%），但降低了基于参考的标准描述指标（例如CIDEr从126.1降至78.6）。我们进一步探索了利用语言模型引导解码过程，在无分类器引导产生的无参考与基于参考描述指标的帕累托前沿上获得了小幅改进，并显著提升了仅使用最小化整理的网络数据训练的模型所生成描述的质量。