The distributional reinforcement learning (RL) approach advocates for representing the complete probability distribution of the random return instead of only modelling its expectation. A distributional RL algorithm may be characterised by two main components, namely the representation of the distribution together with its parameterisation and the probability metric defining the loss. The present research work considers the unconstrained monotonic neural network (UMNN) architecture, a universal approximator of continuous monotonic functions which is particularly well suited for modelling different representations of a distribution. This property enables the efficient decoupling of the effect of the function approximator class from that of the probability metric. The research paper firstly introduces a methodology for learning different representations of the random return distribution (PDF, CDF and QF). Secondly, a novel distributional RL algorithm named unconstrained monotonic deep Q-network (UMDQN) is presented. To the authors' knowledge, it is the first distributional RL method supporting the learning of three, valid and continuous representations of the random return distribution. Lastly, in light of this new algorithm, an empirical comparison is performed between three probability quasi-metrics, namely the Kullback-Leibler divergence, Cramer distance, and Wasserstein distance. The results highlight the main strengths and weaknesses associated with each probability metric together with an important limitation of the Wasserstein distance.

翻译：分布强化学习（RL）方法主张表征随机回报的完整概率分布，而非仅建模其期望。分布强化学习算法的特征可归结为两个主要组成部分：分布的表示及其参数化方式，以及定义损失的概率度量。本研究采用无约束单调神经网络（UMNN）架构——一种连续单调函数的通用逼近器，特别适合建模分布的不同表示形式。该特性能够有效解耦函数逼近器类别与概率度量的影响。本文首先提出一种学习随机回报分布不同表示（PDF、CDF和QF）的方法论，继而引入名为无约束单调深度Q网络（UMDQN）的新型分布强化学习算法。据作者所知，这是首个支持学习随机回报分布三种有效的连续表示形式的分布RL方法。最后，基于该新算法，对三种概率拟度量——Kullback-Leibler散度、Cramer距离和Wasserstein距离——进行了实证比较。研究结果揭示了每种概率度量的主要优劣特性，并指出Wasserstein距离的重要局限性。