The recently rising markup-to-image generation poses greater challenges as compared to natural image generation, due to its low tolerance for errors as well as the complex sequence and context correlations between markup and rendered image. This paper proposes a novel model named "Contrast-augmented Diffusion Model with Fine-grained Sequence Alignment" (FSA-CDM), which introduces contrastive positive/negative samples into the diffusion model to boost performance for markup-to-image generation. Technically, we design a fine-grained cross-modal alignment module to well explore the sequence similarity between the two modalities for learning robust feature representations. To improve the generalization ability, we propose a contrast-augmented diffusion model to explicitly explore positive and negative samples by maximizing a novel contrastive variational objective, which is mathematically inferred to provide a tighter bound for the model's optimization. Moreover, the context-aware cross attention module is developed to capture the contextual information within markup language during the denoising process, yielding better noise prediction results. Extensive experiments are conducted on four benchmark datasets from different domains, and the experimental results demonstrate the effectiveness of the proposed components in FSA-CDM, significantly exceeding state-of-the-art performance by about 2%-12% DTW improvements. The code will be released at https://github.com/zgj77/FSACDM.

翻译：近年来兴起的标记转图像生成任务相较于自然图像生成更具挑战性，其难点在于对错误的极低容忍度以及标记与渲染图像之间复杂的序列和上下文关联。本文提出一种名为"对比增强扩散模型与细粒度序列对齐"（FSA-CDM）的新模型，该模型将对比正/负样本引入扩散模型，以提升标记转图像生成的性能。在技术层面，我们设计了一个细粒度跨模态对齐模块，以深入探索两种模态间的序列相似性，从而学习鲁棒的特征表示。为提高泛化能力，我们提出一种对比增强扩散模型，通过最大化新颖的对比变分目标（该目标经数学推导可为模型优化提供更紧致的界）显式地探索正负样本。此外，我们开发了上下文感知交叉注意力模块，以在去噪过程中捕获标记语言内的上下文信息，从而得到更优的噪声预测结果。我们在来自不同领域的四个基准数据集上进行了大量实验，实验结果表明了FSA-CDM中各组件的有效性，在DTW指标上显著超越现有最优方法约2%-12%。代码将发布在https://github.com/zgj77/FSACDM。