Despite numerous successes in Deep Reinforcement Learning (DRL), the learned policies are not interpretable. Moreover, since DRL does not exploit symbolic relational representations, it has difficulties in coping with structural changes in its environment (such as increasing the number of objects). Relational Reinforcement Learning, on the other hand, inherits the relational representations from symbolic planning to learn reusable policies. However, it has so far been unable to scale up and exploit the power of deep neural networks. We propose Deep Explainable Relational Reinforcement Learning (DERRL), a framework that exploits the best of both -- neural and symbolic worlds. By resorting to a neuro-symbolic approach, DERRL combines relational representations and constraints from symbolic planning with deep learning to extract interpretable policies. These policies are in the form of logical rules that explain how each decision (or action) is arrived at. Through several experiments, in setups like the Countdown Game, Blocks World, Gridworld, and Traffic, we show that the policies learned by DERRL can be applied to different configurations and contexts, hence generalizing to environmental modifications.

翻译：尽管深度强化学习在诸多领域取得了成功，但学习到的策略并不具备可解释性。此外，由于深度强化学习未利用符号关系表示，它在应对环境的结构性变化（如物体数量增加）时存在困难。而关系强化学习则继承了符号规划中的关系表示来学习可重用策略，但至今仍难以扩展规模并充分利用深度神经网络的能力。我们提出了深度可解释关系强化学习（DERRL），这是一个融合神经与符号世界各自优势的框架。通过采用神经符号方法，DERRL将符号规划中的关系表示与约束同深度学习相结合，以提取可解释的策略。这些策略以逻辑规则的形式呈现，清晰解释了每个决策（或动作）的推导过程。在倒计时游戏、积木世界、网格世界和交通等场景的多项实验中，我们证明DERRL学习到的策略可应用于不同的配置与上下文，从而实现对环境变化的泛化。