Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions. This limits tractable querying of this knowledge to start-to-end autoregressive sampling. However, many tasks of interest -- including sequence continuation, infilling, and other forms of constrained generation -- involve sampling from intractable posterior distributions. We address this limitation by using amortized Bayesian inference to sample from these intractable posteriors. Such amortization is algorithmically achieved by fine-tuning LLMs via diversity-seeking reinforcement learning algorithms: generative flow networks (GFlowNets). We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training and reward-maximizing policy optimization. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem and demonstrate that our approach enables data-efficient adaptation of LLMs to tasks that require multi-step rationalization and tool use.

翻译：自回归大型语言模型通过下一个词元的条件分布压缩其训练数据中的知识。这限制了该知识的可处理查询方式，仅支持从起始到结束的自回归采样。然而，许多感兴趣的任务（包括序列续写、填充及其他形式的受控生成）涉及从难以处理的后验分布中采样。我们通过使用摊销贝叶斯推理从这些难以处理的后验分布中采样来应对这一限制。这种摊销在算法上通过使用追求多样性的强化学习算法——生成流网络（GFlowNets）——对大型语言模型进行微调来实现。我们实证证明，这种分布匹配范式的大型语言模型微调可以作为最大似然训练和奖励最大化策略优化的有效替代方案。作为一项重要应用，我们将链式思维推理解释为潜在变量建模问题，并证明我们的方法能够以数据高效的方式使大型语言模型适应需要多步骤推理和工具使用的任务。