Reinforcement Learning (RL) environments can produce training data with spurious correlations between features due to the amount of training data or its limited feature coverage. This can lead to RL agents encoding these misleading correlations in their latent representation, preventing the agent from generalising if the correlation changes within the environment or when deployed in the real world. Disentangled representations can improve robustness, but existing disentanglement techniques that minimise mutual information between features require independent features, thus they cannot disentangle correlated features. We propose an auxiliary task for RL algorithms that learns a disentangled representation of high-dimensional observations with correlated features by minimising the conditional mutual information between features in the representation. We demonstrate experimentally, using continuous control tasks, that our approach improves generalisation under correlation shifts, as well as improving the training performance of RL algorithms in the presence of correlated features.

翻译：强化学习（RL）环境可能因训练数据量或有限的特征覆盖范围，产生带有特征间虚假相关性的训练数据。这会导致RL智能体在其潜在表征中编码这些误导性相关性，当环境内或部署到现实世界时相关性发生变化，智能体将无法泛化。解耦表征可提升鲁棒性，但现有通过最小化特征间互信息实现的解耦技术要求特征相互独立，因此无法解耦相关特征。我们提出一种RL算法辅助任务，通过最小化表征中特征间的条件互信息，对具有相关性特征的高维观测进行解耦表征学习。基于连续控制任务的实验表明，我们的方法能改善相关性偏移下的泛化能力，并提升RL算法在存在相关特征时的训练性能。