In this paper we propose a new generative model of text, Step-unrolled Denoising Autoencoder (SUNDAE), that does not rely on autoregressive models. Similarly to denoising diffusion techniques, SUNDAE is repeatedly applied on a sequence of tokens, starting from random inputs and improving them each time until convergence. We present a simple new improvement operator that converges in fewer iterations than diffusion methods, while qualitatively producing better samples on natural language datasets. SUNDAE achieves state-of-the-art results (among non-autoregressive methods) on the WMT'14 English-to-German translation task and good qualitative results on unconditional language modeling on the Colossal Cleaned Common Crawl dataset and a dataset of Python code from GitHub. The non-autoregressive nature of SUNDAE opens up possibilities beyond left-to-right prompted generation, by filling in arbitrary blank patterns in a template.

翻译：在本文中,我们提出了一个不依赖自动递减模型的新的文本基因模型(SUNDAE),该模型不依赖自动递减模型。类似地,在去除扩散技术的同时,SUNDAE反复在一系列符号上应用,从随机输入开始,每次改进,直到趋同。我们提出了一个简单的新的改进操作器,其迭代比扩散方法的迭代要少,同时在质量上产生更好的自然语言数据集样本。SUNDAE实现了WMT'14英文至德文翻译工作的最新结果,以及在GitHub的无条件语言模型化通用粗略拼图样数据集和Python代码数据集方面的良好质量结果。SUNDAE的非侵略性质通过在模板中填充任意空白模式,在左向右生成之外开辟了各种可能性。