Aiming to produce reinforcement learning (RL) policies that are human-interpretable and can generalize better to novel scenarios, Trivedi et al. (2021) present a method (LEAPS) that first learns a program embedding space to continuously parameterize diverse programs from a pre-generated program dataset, and then searches for a task-solving program in the learned program embedding space when given a task. Despite encouraging results, the program policies that LEAPS can produce are limited by the distribution of the program dataset. Furthermore, during searching, LEAPS evaluates each candidate program solely based on its return, failing to precisely reward correct parts of programs and penalize incorrect parts. To address these issues, we propose to learn a meta-policy that composes a series of programs sampled from the learned program embedding space. By composing programs, our proposed method can produce program policies that describe out-of-distributionally complex behaviors and directly assign credits to programs that induce desired behaviors. We design and conduct extensive experiments in the Karel domain. The experimental results show that our proposed framework outperforms baselines. The ablation studies confirm the limitations of LEAPS and justify our design choices.

翻译：摘要：为生成人类可解释且能更好泛化至新场景的强化学习策略，Trivedi等（2021）提出了一种方法（LEAPS），该方法首先学习一个程序嵌入空间，以连续参数化来自预生成程序数据集中的多样化程序，然后在给定任务时，在已学习的程序嵌入空间中搜索能够解决该任务的程序。尽管取得了令人鼓舞的结果，但LEAPS所能生成的程序策略仍受限于程序数据集的分布。此外，在搜索过程中，LEAPS仅基于候选程序的回报对其评估，未能准确奖励程序中的正确部分并惩罚错误部分。为解决这些问题，我们提出学习一种元策略，该策略能够组合从已学习程序嵌入空间中采样得到的一系列程序。通过组合程序，我们所提出的方法可以生成描述分布外复杂行为的程序策略，并直接为引发期望行为的程序分配奖励。我们在Karel领域中设计并开展了大量实验。实验结果表明，我们提出的框架优于基线方法。消融研究证实了LEAPS的局限性，并验证了我们的设计选择。