In many domains, autoregressive models can achieve low log-likelihood on the task of predicting the next observation. However, this maximum-likelihood (MLE) objective does not necessarily match a downstream use-case of autoregressively generating high-quality sequences. The MLE objective weights sequences proportionally to their frequency under the data distribution, with no guidance for the model's behaviour out of distribution (OOD): leading to compounding error during autoregressive generation. In order to address this compounding error problem, we formulate sequence generation as an imitation learning (IL) problem. This allows us to minimize a variety of divergences between the distribution of sequences generated by an autoregressive model and sequences from a dataset, including divergences with weight on OOD generated sequences. The IL framework also allows us to incorporate backtracking by introducing a backspace action into the generation process. This further mitigates the compounding error problem by allowing the model to revert a sampled token if it takes the sequence OOD. Our resulting method, SequenceMatch, can be implemented without adversarial training or major architectural changes. We identify the SequenceMatch-$\chi^2$ divergence as a more suitable training objective for autoregressive models which are used for generation. We show that empirically, SequenceMatch training leads to improvements over MLE on text generation with language models.

翻译：在许多领域中，自回归模型在预测下一观测值的任务上能够实现较低的负对数似然。然而，这种最大似然估计（MLE）目标并不一定与自回归生成高质量序列的下游应用相匹配。MLE目标根据序列在数据分布下的频率分配权重，但未对模型在分布外（OOD）的行为提供指导，导致自回归生成过程中的累积误差。为解决这一累积误差问题，我们将序列生成建模为模仿学习（IL）问题。这使得我们能够最小化自回归模型生成的序列分布与数据集中序列分布之间的多种散度，包括对OOD生成序列赋予权重的散度。IL框架还允许我们通过向生成过程引入退格动作来融入回溯机制，从而进一步缓解累积误差问题——当模型生成导致序列处于OOD状态时，可回退已采样的令牌。我们提出的方法“序列匹配”（SequenceMatch）无需对抗训练或重大架构改动即可实现。我们确定了序列匹配-χ²散度作为更适合用于生成的序列模型训练目标。实验证明，相比MLE训练，序列匹配训练在语言模型的文本生成任务中能够带来性能提升。