Autoregressive models (ARMs) have become the workhorse for sequence generation tasks, since many problems can be modeled as next-token prediction. While there appears to be a natural ordering for text (i.e., left-to-right), for many data types, such as graphs, the canonical ordering is less obvious. To address this problem, we introduce a variant of ARM that generates high-dimensional data using a probabilistic ordering that is sequentially inferred from data. This model incorporates a trainable probability distribution, referred to as an \emph{order-policy}, that dynamically decides the autoregressive order in a state-dependent manner. To train the model, we introduce a variational lower bound on the exact log-likelihood, which we optimize with stochastic gradient estimation. We demonstrate experimentally that our method can learn meaningful autoregressive orderings in image and graph generation. On the challenging domain of molecular graph generation, we achieve state-of-the-art results on the QM9 and ZINC250k benchmarks, evaluated using the Fr\'{e}chet ChemNet Distance (FCD).

翻译：自回归模型（ARMs）已成为序列生成任务的主流方法，因为许多问题可以建模为下一标记预测。虽然文本似乎存在自然顺序（即从左到右），但对于许多数据类型（如图形），规范顺序则不那么明显。为解决此问题，我们引入了一种自回归模型的变体，该模型使用从数据中顺序推断的概率排序来生成高维数据。该模型包含一个可训练的概率分布（称为顺序策略），以状态依赖的方式动态决定自回归顺序。为训练该模型，我们引入了精确对数似然的变分下界，并通过随机梯度估计进行优化。实验表明，我们的方法能够在图像和图形生成中学习有意义的自回归排序。在分子图生成这一挑战性领域，我们在QM9和ZINC250k基准测试中取得了最先进的结果，该结果通过Fr\'{e}chet ChemNet距离（FCD）进行评估。