Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions. This limits tractable querying of this knowledge to start-to-end autoregressive sampling. However, many tasks of interest -- including sequence continuation, infilling, and other forms of constrained generation -- involve sampling from intractable posterior distributions. We address this limitation by using amortized Bayesian inference to sample from these intractable posteriors. Such amortization is algorithmically achieved by fine-tuning LLMs via diversity-seeking reinforcement learning algorithms: generative flow networks (GFlowNets). We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training and reward-maximizing policy optimization. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem and demonstrate that our approach enables data-efficient adaptation of LLMs to tasks that require multi-step rationalization and tool use.

翻译：自回归大规模语言模型（LLMs）通过下一个词元的条件分布来压缩训练数据中的知识。这限制了从起点到终点的自回归采样来可处理地查询这些知识。然而，许多感兴趣的任务——包括序列续写、填充以及其他形式的约束生成——涉及从难处理的后验分布中采样。我们通过使用分摊贝叶斯推理来从这些难处理后验中采样，从而解决了这一局限性。这种分摊在算法上通过使用追求多样性的强化学习算法（生成流网络，GFlowNets）对LLMs进行微调来实现。我们实证表明，这种LLM微调的分布匹配范式可以作为最大似然训练和奖励最大化策略优化的有效替代方案。作为一个重要应用，我们将思维链推理解释为潜变量建模问题，并证明我们的方法能够使LLMs以数据高效的方式适应需要多步推理和工具使用的任务。