Deep reinforcement learning (DeepRL) agents surpass human-level performance in many tasks. However, the direct mapping from states to actions makes it hard to interpret the rationale behind the decision-making of the agents. In contrast to previous a-posteriori methods for visualizing DeepRL policies, in this work, we propose to equip the DeepRL model with an innate visualization ability. Our proposed agent, named region-sensitive Rainbow (RS-Rainbow), is an end-to-end trainable network based on the original Rainbow, a powerful deep Q-network agent. It learns important regions in the input domain via an attention module. At inference time, after each forward pass, we can visualize regions that are most important to decision-making by backpropagating gradients from the attention module to the input frames. The incorporation of our proposed module not only improves model interpretability, but leads to performance improvement. Extensive experiments on games from the Atari 2600 suite demonstrate the effectiveness of RS-Rainbow.

翻译：深度强化学习（DeepRL）智能体在许多任务中超越了人类水平的表现。然而，从状态到动作的直接映射使得解释智能体决策背后的原理变得困难。与以往用于可视化深度强化学习策略的事后方法不同，本文我们提出为深度强化学习模型赋予内在的可视化能力。我们提出的智能体，名为区域敏感彩虹（RS-Rainbow），是基于原始彩虹（Rainbow）——一个强大的深度Q网络智能体——的端到端可训练网络。它通过注意力模块学习输入域中的重要区域。在推理时，每次前向传播后，我们可以通过将梯度从注意力模块反向传播到输入帧，来可视化对决策最重要的区域。我们所提出的模块不仅提升了模型的可解释性，还带来了性能提升。在Atari 2600套件游戏上的大量实验证明了RS-Rainbow的有效性。