Diffusion models have emerged as a powerful paradigm for generation, obtaining strong performance in various domains with continuous-valued inputs. Despite the promises of fully non-autoregressive text generation, applying diffusion models to natural language remains challenging due to its discrete nature. In this work, we propose Text-to-text Self-conditioned Simplex Diffusion (TESS), a text diffusion model that is fully non-autoregressive, employs a new form of self-conditioning, and applies the diffusion process on the logit simplex space rather than the typical learned embedding space. Through extensive experiments on natural language understanding and generation tasks including summarization, text simplification, paraphrase generation, and question generation, we demonstrate that TESS outperforms state-of-the-art non-autoregressive models and is competitive with pretrained autoregressive sequence-to-sequence models.

翻译：扩散模型已成为一种强大的生成范式，在连续值输入的多个领域取得了优异表现。尽管全非自回归文本生成前景广阔，但由于自然语言的离散特性，将扩散模型应用于该领域仍具挑战性。本文提出文本到文本的自条件单纯形扩散（TESS）——一种全非自回归文本扩散模型，它采用新型自条件机制，并在logit单纯形空间（而非典型的嵌入学习空间）上应用扩散过程。通过涵盖摘要生成、文本简化、释义生成及问题生成等自然语言理解与生成任务的广泛实验，我们证明TESS性能超越现有最先进非自回归模型，且与预训练自回归序列到序列模型具有竞争力。