This survey paper provides a comprehensive review of the use of diffusion models in natural language processing (NLP). Diffusion models are a class of mathematical models that aim to capture the diffusion of information or signals across a network or manifold. In NLP, diffusion models have been used in a variety of applications, such as natural language generation, sentiment analysis, topic modeling, and machine translation. This paper discusses the different formulations of diffusion models used in NLP, their strengths and limitations, and their applications. We also perform a thorough comparison between diffusion models and alternative generative models, specifically highlighting the autoregressive (AR) models, while also examining how diverse architectures incorporate the Transformer in conjunction with diffusion models. Compared to AR models, diffusion models have significant advantages for parallel generation, text interpolation, token-level controls such as syntactic structures and semantic contents, and robustness. Exploring further permutations of integrating Transformers into diffusion models would be a valuable pursuit. Also, the development of multimodal diffusion models and large-scale diffusion language models with notable capabilities for few-shot learning would be important directions for the future advance of diffusion models in NLP.

翻译：本综述论文全面回顾了扩散模型在自然语言处理（NLP）中的应用。扩散模型是一类旨在捕捉信息或信号在网络或流形上扩散过程的数学模型。在NLP领域，扩散模型已被应用于多种任务，包括自然语言生成、情感分析、主题建模以及机器翻译。本文讨论了NLP中使用的不同扩散模型形式化方法、其优势与局限性，以及具体应用。我们还对扩散模型与其他生成模型进行了深入比较，特别强调了自回归（AR）模型，同时考察了不同架构如何将Transformer与扩散模型相结合。与AR模型相比，扩散模型在并行生成、文本插值、句法结构与语义内容等词元级控制能力以及鲁棒性方面具有显著优势。探索Transformer与扩散模型整合的更广泛排列组合将是一项有意义的研究方向。此外，发展多模态扩散模型以及具备显著小样本学习能力的大规模扩散语言模型，将是扩散模型在NLP领域未来发展的重要方向。