A zero-shot RL agent is an agent that can solve any RL task in a given environment, instantly with no additional planning or learning, after an initial reward-free learning phase. This marks a shift from the reward-centric RL paradigm towards "controllable" agents that can follow arbitrary instructions in an environment. Current RL agents can solve families of related tasks at best, or require planning anew for each task. Strategies for approximate zero-shot RL ave been suggested using successor features (SFs) [BBQ+ 18] or forward-backward (FB) representations [TO21], but testing has been limited. After clarifying the relationships between these schemes, we introduce improved losses and new SF models, and test the viability of zero-shot RL schemes systematically on tasks from the Unsupervised RL benchmark [LYL+21]. To disentangle universal representation learning from exploration, we work in an offline setting and repeat the tests on several existing replay buffers. SFs appear to suffer from the choice of the elementary state features. SFs with Laplacian eigenfunctions do well, while SFs based on auto-encoders, inverse curiosity, transition models, low-rank transition matrix, contrastive learning, or diversity (APS), perform unconsistently. In contrast, FB representations jointly learn the elementary and successor features from a single, principled criterion. They perform best and consistently across the board, reaching 85% of supervised RL performance with a good replay buffer, in a zero-shot manner.

翻译：零样本强化学习智能体是一种能够在初始无奖励学习阶段后，无需额外规划或学习，即时解决给定环境中任何强化学习任务的智能体。这标志着从以奖励为中心的强化学习范式向可在环境中遵循任意指令的“可控”智能体的转变。当前的强化学习智能体最多只能解决相关任务族，或需要为每个任务重新规划。已有研究提出利用后继特征（SFs）[BBQ+ 18]或前向-后向（FB）表示[TO21]来实现近似零样本强化学习的策略，但测试有限。在厘清这些方案之间的关系后，我们引入了改进的损失函数和新的SF模型，并系统地测试了零样本强化学习方案在无监督强化学习基准[LYL+21]任务上的可行性。为将通用表示学习与探索相分离，我们采用离线设置，并在多个现有经验回放缓冲池上重复测试。SFs似乎受限于基本状态特征的选择；基于拉普拉斯特征函数的SFs表现良好，而基于自编码器、逆向好奇心、转移模型、低秩转移矩阵、对比学习或多样性（APS）的SFs表现不一致。相比之下，FB表示通过单一、原则性的准则联合学习基本特征和后继特征，其结果在所有测试中一致最优，在优质回放缓冲池下以零样本方式达到监督强化学习性能的85%。