Learning policies which are robust to changes in the environment are critical for real world deployment of Reinforcement Learning agents. They are also necessary for achieving good generalization across environment shifts. We focus on bisimulation metrics, which provide a powerful means for abstracting task relevant components of the observation and learning a succinct representation space for training the agent using reinforcement learning. In this work, we extend the bisimulation framework to also account for context dependent observation shifts. Specifically, we focus on the simulator based learning setting and use alternate observations to learn a representation space which is invariant to observation shifts using a novel bisimulation based objective. This allows us to deploy the agent to varying observation settings during test time and generalize to unseen scenarios. We further provide novel theoretical bounds for simulator fidelity and performance transfer guarantees for using a learnt policy to unseen shifts. Empirical analysis on the high-dimensional image based control domains demonstrates the efficacy of our method.

翻译：学习对环境变化具有鲁棒性的策略对于强化学习智能体在实际部署中至关重要，也是实现跨环境偏移良好泛化的必要条件。我们聚焦于双模拟度量——该度量提供了一种强大手段，可抽象出观测中与任务相关的成分，并学习用于训练智能体的简洁表征空间。本研究将双模拟框架扩展至考虑上下文相关的观测偏移：具体而言，我们基于模拟器学习设定，利用交替观测通过一种新型双模拟目标函数学习对观测偏移保持不变的表征空间。这使得智能体在测试阶段能够适应不同观测场景，并泛化至未见场景。我们进一步为模拟器保真度和使用已学习策略迁移至未知偏移的性能保障提供了新的理论边界。基于高维图像控制域的实验分析证实了本方法的有效性。