Language models based on discrete diffusion have attracted widespread interest for their potential to provide faster generation than autoregressive models. Despite their promise, these models typically produce samples whose quality sharply degrades in the few-step regime, preventing a dramatic speedup in practice. Here, we show that language models based on continuous flows over one-hot token embeddings can outperform discrete diffusion in both quality and speed. Importantly, our continuous formulation defines a unique flow map that can be learned directly for efficient few-step inference, a structure we show is unavailable to discrete methods. In this setting, we show that both the flow and its associated flow map can be learned with simple cross-entropy objectives that respect the simplex geometry of the data, and we identify three distinct choices for flow map distillation whose performance we compare in practice. Using these insights, we build a flow language model (FLM), a continuous flow that matches state-of-the-art discrete diffusion baselines on the One Billion Words (LM1B) and OpenWebText (OWT) datasets. We then distill FLM into a flow map language model (FMLM), whose one-step generation exceeds the 8-step quality of recent few-step discrete diffusion language models. Our work challenges the widely-held hypothesis that discrete noising processes are necessary for generative modeling over discrete modalities and paves the way toward accelerated language modeling at scale. Code is available at https://github.com/david3684/flm.

翻译：基于离散扩散的语言模型因其在生成速度上超越自回归模型的潜力而受到广泛关注。然而，尽管前景可期，这类模型在少步生成机制下生成的样本质量会急剧下降，从而在实际应用中难以实现显著的加速效果。本文证明，基于独热词元嵌入的连续流语言模型在质量和速度上均可超越离散扩散模型。关键在于，我们的连续形式定义了一种独特的流映射，该映射可直接学习以实现高效的少步推理——这一结构特性是离散方法所不具备的。在此框架下，我们证明了流及其对应的流映射均可通过遵循数据单纯形几何结构的简单交叉熵目标进行学习，并确定了三种不同的流映射蒸馏策略，在实际中对其性能进行了对比。基于这些发现，我们构建了流语言模型（FLM），这是一种连续流模型，在十亿词（LM1B）和OpenWebText（OWT）数据集上达到了与最先进离散扩散基线相当的性能。随后，我们将FLM蒸馏为流映射语言模型（FMLM），其单步生成质量超越了近期少步离散扩散语言模型的八步生成质量。本研究挑战了"离散模态生成必须依赖离散噪声过程"的广泛假设，为大规模加速语言建模开辟了新路径。代码已开源至 https://github.com/david3684/flm。