Diffusion model, a new generative modelling paradigm, has achieved great success in image, audio, and video generation. However, considering the discrete categorical nature of text, it is not trivial to extend continuous diffusion models to natural language, and text diffusion models are less studied. Sequence-to-sequence text generation is one of the essential natural language processing topics. In this work, we apply diffusion models to approach sequence-to-sequence text generation, and explore whether the superiority generation performance of diffusion model can transfer to natural language domain. We propose SeqDiffuSeq, a text diffusion model for sequence-to-sequence generation. SeqDiffuSeq uses an encoder-decoder Transformers architecture to model denoising function. In order to improve generation quality, SeqDiffuSeq combines the self-conditioning technique and a newly proposed adaptive noise schedule technique. The adaptive noise schedule has the difficulty of denoising evenly distributed across time steps, and considers exclusive noise schedules for tokens at different positional order. Experiment results illustrate the good performance on sequence-to-sequence generation in terms of text quality and inference time.

翻译：扩散模型作为一种新型生成建模范式，在图像、音频和视频生成领域取得了巨大成功。然而，考虑到文本的离散类别特性，将连续扩散模型扩展到自然语言领域并非易事，因此文本扩散模型的研究相对较少。序列到序列文本生成是自然语言处理的核心课题之一。本研究将扩散模型应用于序列到序列文本生成，探究扩散模型优异的生成性能能否迁移至自然语言领域。我们提出SeqDiffuSeq——一种面向序列到序列生成的文本扩散模型。SeqDiffuSeq采用编码器-解码器Transformer架构对去噪函数进行建模。为提升生成质量，SeqDiffuSeq结合了自条件化技术及新提出的自适应噪声调度策略。该自适应噪声调度使去噪难度在时间步上均匀分布，并针对不同位置顺序的令牌采用专属噪声调度方案。实验结果表明，该方法在文本质量和推理时间方面均展现出优异的序列到序列生成性能。