Meta reinforcement learning (meta-RL) methods such as RL$^2$ have emerged as promising approaches for learning data-efficient RL algorithms tailored to a given task distribution. However, these RL algorithms struggle with long-horizon tasks and out-of-distribution tasks since they rely on recurrent neural networks to process the sequence of experiences instead of summarizing them into general RL components such as value functions. Moreover, even transformers have a practical limit to the length of histories they can efficiently reason about before training and inference costs become prohibitive. In contrast, traditional RL algorithms are data-inefficient since they do not leverage domain knowledge, but they do converge to an optimal policy as more data becomes available. In this paper, we propose RL$^3$, a principled hybrid approach that combines traditional RL and meta-RL by incorporating task-specific action-values learned through traditional RL as an input to the meta-RL neural network. We show that RL$^3$ earns greater cumulative reward on long-horizon and out-of-distribution tasks compared to RL$^2$, while maintaining the efficiency of the latter in the short term. Experiments are conducted on both custom and benchmark discrete domains from the meta-RL literature that exhibit a range of short-term, long-term, and complex dependencies.

翻译：摘要：诸如RL$^2$之类的元强化学习方法已成为一种有前景的途径，能够学习针对特定任务分布的数据高效型强化学习算法。然而，这些强化学习算法在长时域任务和分布外任务中表现不佳，原因在于它们依赖循环神经网络处理经验序列，而非将其总结为通用的强化学习组件（如值函数）。此外，即使使用Transformer架构，在训练和推理成本变得过高之前，其能够高效推理的历史序列长度也存在实际限制。相比之下，传统强化学习算法虽因未利用领域知识而数据效率低下，但会随着数据量增加收敛至最优策略。本文提出RL$^3$这一原则性的混合方法，通过将传统强化学习学习到的任务特定动作值作为元强化学习神经网络的输入，结合传统强化学习与元强化学习。实验表明，RL$^3$在长时域和分布外任务中相比RL$^2$能获得更高的累积奖励，同时保持后者在短期任务中的效率。我们在元强化学习文献中的自定义及基准离散域上进行实验，这些域展现了从短期、长期到复杂依赖关系的多样化特性。