Diffusion models have exhibited remarkable capabilities in text-to-image generation. However, their performance in image-to-text generation, specifically image captioning, has lagged behind Auto-Regressive (AR) models, casting doubt on their applicability for such tasks. In this work, we revisit diffusion models, highlighting their capacity for holistic context modeling and parallel decoding. With these benefits, diffusion models can alleviate the inherent limitations of AR methods, including their slow inference speed, error propagation, and unidirectional constraints. Furthermore, we identify the prior underperformance of diffusion models stemming from the absence of an effective latent space for image-text alignment, and the discrepancy between continuous diffusion processes and discrete textual data. In response, we introduce a novel architecture, LaDiC, which utilizes a split BERT to create a dedicated latent space for captions and integrates a regularization module to manage varying text lengths. Our framework also includes a diffuser for semantic image-to-text conversion and a Back&Refine technique to enhance token interactivity during inference. LaDiC achieves state-of-the-art performance for diffusion-based methods on the MS COCO dataset with 38.2 BLEU@4 and 126.2 CIDEr, demonstrating exceptional performance without pre-training or ancillary modules. This indicates strong competitiveness with AR models, revealing the previously untapped potential of diffusion models in image-to-text generation.

翻译：摘要：扩散模型在文本到图像生成任务中已展现出卓越能力，但其在图像到文本生成（尤其是图像描述）中的表现仍落后于自回归模型，这对其在此类任务中的适用性提出了质疑。本文重新审视扩散模型，强调其全局上下文建模与并行解码的能力。凭借这些优势，扩散模型可缓解自回归方法固有的局限性，包括推理速度慢、错误传播及单向约束等问题。此外，我们识别出扩散模型先前性能不足的根源：缺乏有效的图像-文本对齐潜在空间，以及连续扩散过程与离散文本数据之间的差异。为此，我们提出一种新型架构LaDiC，该架构利用分裂BERT为描述生成专用潜在空间，并集成正则化模块以处理不同文本长度。我们的框架还包含一个用于语义图像到文本转换的扩散器，以及一种Back&Refine技术，以增强推理过程中的令牌交互性。LaDiC在MS COCO数据集上以38.2 BLEU@4和126.2 CIDEr的指标取得了基于扩散方法的最优性能，且无需预训练或辅助模块即展现出卓越表现。这表明其与自回归模型具有强劲竞争力，揭示了扩散模型在图像到文本生成中此前未被发掘的潜力。