Despite the potential benefits of Diffusion Models for NLP applications, publicly available implementations, trained models, or reproducible training procedures currently need to be publicly available. We present the Democratized Diffusion Language Model (DDLM), based on the Continuous Diffusion for Categorical Data (CDCD) framework, to address these challenges. We propose a simplified training procedure for DDLM using the C4 dataset and perform an in-depth analysis of the trained model's behavior. Furthermore, we introduce a novel early-exiting strategy for faster sampling with models trained with score interpolation. Since no previous works aimed at solving downstream tasks with pre-trained Diffusion LM (e.g., classification tasks), we experimented with GLUE Benchmark to study the ability of DDLM to transfer knowledge. With this paper, we propose available training and evaluation pipelines to other researchers and pre-trained DDLM models, which could be used in future research with Diffusion LMs.

翻译：尽管扩散模型在自然语言处理应用中具有潜在优势，但目前仍缺乏公开可用的实现、预训练模型或可复现的训练流程。我们提出了基于分类数据连续扩散（CDCD）框架的民主化扩散语言模型（DDLM），以应对这些挑战。我们利用C4数据集为DDLM设计了一套简化训练流程，并对训练后模型的行为进行了深入分析。此外，我们引入了一种新颖的早期退出策略，用于加速基于分数插值训练的模型采样过程。由于此前尚无研究探索利用预训练扩散语言模型（如分类任务）解决下游任务，我们基于GLUE基准进行了实验，以研究DDLM的知识迁移能力。通过本文，我们向其他研究者提供了可复现的训练与评估流程，以及预训练DDLM模型，这些成果可应用于未来扩散语言模型的相关研究。