Autoregressive models for text sometimes generate repetitive and low-quality output because errors accumulate during the steps of generation. This issue is often attributed to exposure bias - the difference between how a model is trained, and how it is used during inference. Denoising diffusion models provide an alternative approach in which a model can revisit and revise its output. However, they can be computationally expensive and prior efforts on text have led to models that produce less fluent output compared to autoregressive models, especially for longer text and paragraphs. In this paper, we propose PLANNER, a model that combines latent semantic diffusion with autoregressive generation, to generate fluent text while exercising global control over paragraphs. The model achieves this by combining an autoregressive "decoding" module with a "planning" module that uses latent diffusion to generate semantic paragraph embeddings in a coarse-to-fine manner. The proposed method is evaluated on various conditional generation tasks, and results on semantic generation, text completion and summarization show its effectiveness in generating high-quality long-form text in an efficient manner.

翻译：自回归文本模型在生成过程中因误差累积，有时会产生重复且低质量的输出。这一问题通常归因于暴露偏差——即模型训练方式与推理阶段使用方式之间的差异。去噪扩散模型提供了一种替代方法，使模型能够重新审视并修正其输出。然而，这类模型计算成本高昂，且此前针对文本领域的应用导致生成的文本（尤其是长文本和段落）流畅性低于自回归模型。本文提出PLANNER模型，该模型将潜在语义扩散与自回归生成相结合，在实现段落全局控制的同时生成流畅文本。该模型通过结合自回归“解码”模块与“规划”模块实现上述目标——其中规划模块利用潜在扩散以由粗到精的方式生成语义段落嵌入。我们在多种条件生成任务上对所提方法进行了评估，语义生成、文本补全和摘要任务上的结果表明，该方法能高效生成高质量的长文文本。