We propose a novel algorithmic framework for distributional reinforcement learning, based on learning finite-dimensional mean embeddings of return distributions. We derive several new algorithms for dynamic programming and temporal-difference learning based on this framework, provide asymptotic convergence theory, and examine the empirical performance of the algorithms on a suite of tabular tasks. Further, we show that this approach can be straightforwardly combined with deep reinforcement learning, and obtain a new deep RL agent that improves over baseline distributional approaches on the Arcade Learning Environment.

翻译：我们提出了一种新颖的分布强化学习算法框架，其核心是学习回报分布的有限维均值嵌入。基于该框架，我们推导出若干新的动态规划与时序差分学习算法，提供了渐近收敛性理论分析，并在系列表格型任务上检验了算法的实证性能。进一步研究表明，该方法可简便地与深度强化学习相结合，由此得到的深度强化学习智能体在Arcade学习环境中相较于基准分布方法取得了性能提升。