Generalization in partially observed markov decision processes (POMDPs) is critical for successful applications of visual reinforcement learning (VRL) in real scenarios. A widely used idea is to learn task-relevant representations that encode task-relevant information of common features in POMDPs, i.e., rewards and transition dynamics. As transition dynamics in the latent state space -- which are task-relevant and invariant to visual distractions -- are unknown to the agents, existing methods alternatively use transition dynamics in the observation space to extract task-relevant information in transition dynamics. However, such transition dynamics in the observation space involve task-irrelevant visual distractions, degrading the generalization performance of VRL methods. To tackle this problem, we propose the reward sequence distribution conditioned on the starting observation and the predefined subsequent action sequence (RSD-OA). The appealing features of RSD-OA include that: (1) RSD-OA is invariant to visual distractions, as it is conditioned on the predefined subsequent action sequence without task-irrelevant information from transition dynamics, and (2) the reward sequence captures long-term task-relevant information in both rewards and transition dynamics. Experiments demonstrate that our representation learning approach based on RSD-OA significantly improves the generalization performance on unseen environments, outperforming several state-of-the-arts on DeepMind Control tasks with visual distractions.

翻译：在部分可观测马尔可夫决策过程（POMDPs）中，泛化性对于视觉强化学习（VRL）在真实场景中的成功应用至关重要。一种广泛采用的方法是学习任务相关表示，该表示编码POMDP中共同特征的任务相关信息，即奖励和转移动力学。由于潜在状态空间中的转移动力学（与任务相关且对视觉干扰保持不变）对智能体是未知的，现有方法转而利用观测空间中的转移动力学来提取转移动力学中的任务相关信息。然而，观测空间中的此类转移动力学包含与任务无关的视觉干扰，从而降低了VRL方法的泛化性能。为解决这一问题，我们提出了基于起始观测和预定义后续动作序列的条件化奖励序列分布（RSD-OA）。RSD-OA的显著特点包括：（1）RSD-OA对视觉干扰保持不变，因为它以预定义的后续动作序列为条件，不包含转移动力学中的任务无关信息；（2）奖励序列同时捕捉了奖励和转移动力学中的长期任务相关信息。实验表明，基于RSD-OA的表示学习方法显著提升了在未见环境中的泛化性能，在包含视觉干扰的DeepMind Control任务上优于多种当前最先进方法。