Reinforcement learning (RL) is used in various robotic applications. RL enables agents to learn tasks autonomously by interacting with the environment. The more critical the tasks are, the higher the demand for the robustness of the RL systems. Causal RL combines RL and causal inference to make RL more robust. Causal RL agents use a causal representation to capture the invariant causal mechanisms that can be transferred from one task to another. Currently, there is limited research in Causal RL, and existing solutions are usually not complete or feasible for real-world applications. In this work, we propose CausalCF, the first complete Causal RL solution incorporating ideas from Causal Curiosity and CoPhy. Causal Curiosity provides an approach for using interventions, and CoPhy is modified to enable the RL agent to perform counterfactuals. Causal Curiosity has been applied to robotic grasping and manipulation tasks in CausalWorld. CausalWorld provides a realistic simulation environment based on the TriFinger robot. We apply CausalCF to complex robotic tasks and show that it improves the RL agent's robustness using CausalWorld.

翻译：强化学习被广泛应用于各类机器人应用中。强化学习使智能体能够通过与环境的交互自主习得任务。任务越关键，对强化学习系统鲁棒性的要求就越高。因果强化学习将强化学习与因果推断相结合以提升其鲁棒性。因果强化学习智能体利用因果表征捕捉可在不同任务间迁移的不变因果机制。目前，因果强化学习领域的研究尚不充分，现有方案通常不完整或难以应用于实际场景。本文提出CausalCF——首个融合Causal Curiosity与CoPhy思想的完整因果强化学习方案。Causal Curiosity提供了干预方法的应用框架，而CoPhy经改进后使强化学习智能体能够执行反事实推理。Causal Curiosity已应用于CausalWorld中的机器人抓取与操作任务。CausalWorld提供了基于TriFinger机器人的逼真仿真环境。我们将CausalCF应用于复杂机器人任务，实验表明该方法通过CausalWorld有效提升了强化学习智能体的鲁棒性。