Recent improvements in conditional generative modeling have made it possible to generate high-quality images from language descriptions alone. We investigate whether these methods can directly address the problem of sequential decision-making. We view decision-making not through the lens of reinforcement learning (RL), but rather through conditional generative modeling. To our surprise, we find that our formulation leads to policies that can outperform existing offline RL approaches across standard benchmarks. By modeling a policy as a return-conditional diffusion model, we illustrate how we may circumvent the need for dynamic programming and subsequently eliminate many of the complexities that come with traditional offline RL. We further demonstrate the advantages of modeling policies as conditional diffusion models by considering two other conditioning variables: constraints and skills. Conditioning on a single constraint or skill during training leads to behaviors at test-time that can satisfy several constraints together or demonstrate a composition of skills. Our results illustrate that conditional generative modeling is a powerful tool for decision-making.

翻译：摘要：近期条件生成建模的改进使得仅凭语言描述即可生成高质量图像成为可能。我们探究了这些方法是否可直接解决序列决策问题。我们并非通过强化学习（RL）的视角，而是借助条件生成建模来审视决策问题。令人惊讶的是，我们发现这种公式化方法所产生的策略在标准基准测试中可超越现有离线RL方法。通过将策略建模为回报条件扩散模型，我们阐释了如何规避动态规划的需求，进而消除了传统离线RL中伴随的许多复杂性。我们还通过考虑另外两个条件变量——约束条件与技能——进一步展示了将策略建模为条件扩散模型优势：在训练阶段对单一约束或技能进行条件化，使测试时的行为能够同时满足多项约束或展现技能的组合。我们的结果表明，条件生成建模是解决决策问题的强大工具。