Non-monotonic sequence generation methods, such as masked diffusion models, provide a flexible alternative to left-to-right autoregressive modeling by allowing tokens to be generated in non-fixed and prescribed orders. Despite their practical advantages, most existing non-monotonic models are order-agnostic and rely on a fixed-length grid, limiting their ability to support variable-length generation and adaptive insertion order. In this work, we introduce a probabilistic framework for learning insertion order in variable-length insertion models. We formalize a bijective correspondence between insertion trajectories and permutations, which enables an exact reparameterization of the data likelihood as a sum over permutations. Building on this result, we propose the Insertion Process (IP), a stochastic generative model that jointly learns where to insert, what to insert, and when to terminate, trained via permutation-based variational inference. Unlike prior fixed-canvas approaches, IP natively supports variable-length generation and learns data-driven preferences over insertion orders. Experiments on goal-conditioned planning and molecular string generation demonstrate that learning insertion order improves both modeling quality and generalization in domains without a canonical left-to-right structure.

翻译：非单调序列生成方法（如掩码扩散模型）通过允许以非固定且非预设的顺序生成令牌，为从左到右的自回归建模提供了灵活的替代方案。尽管具有实际优势，但现有的大多数非单调模型是顺序无关的，并依赖于固定长度的网格，这限制了它们支持可变长度生成和自适应插入顺序的能力。在这项工作中，我们引入了一个概率框架，用于在可变长度插入模型中学习插入顺序。我们形式化了插入轨迹与排列之间的双射对应关系，从而能够将数据似然精确地重新参数化为对排列的求和。基于这一结果，我们提出了插入过程（IP），这是一种随机生成模型，能够联合学习在哪里插入、插入什么以及何时终止，并通过基于排列的变分推断进行训练。与先前的固定画布方法不同，IP原生支持可变长度生成，并学习数据驱动的插入顺序偏好。在目标条件规划和分子字符串生成上的实验表明，学习插入顺序提高了建模质量和泛化能力，尤其是在没有规范从左到右结构的领域中。