Many applications in Reinforcement Learning (RL) usually have noise or stochasticity present in the environment. Beyond their impact on learning, these uncertainties lead the exact same policy to perform differently, i.e. yield different return, from one roll-out to another. Common evaluation procedures in RL summarise the consequent return distributions using solely the expected return, which does not account for the spread of the distribution. Our work defines this spread as the policy reproducibility: the ability of a policy to obtain similar performance when rolled out many times, a crucial property in some real-world applications. We highlight that existing procedures that only use the expected return are limited on two fronts: first an infinite number of return distributions with a wide range of performance-reproducibility trade-offs can have the same expected return, limiting its effectiveness when used for comparing policies; second, the expected return metric does not leave any room for practitioners to choose the best trade-off value for considered applications. In this work, we address these limitations by recommending the use of Lower Confidence Bound, a metric taken from Bayesian optimisation that provides the user with a preference parameter to choose a desired performance-reproducibility trade-off. We also formalise and quantify policy reproducibility, and demonstrate the benefit of our metrics using extensive experiments of popular RL algorithms on common uncertain RL tasks.

翻译：在强化学习（RL）的众多应用中，环境通常存在噪声或随机性。除了对学习过程产生影响外，这些不确定性会导致完全相同的策略在多次执行中产生不同结果，即获得不同的回报。RL中常见的评估流程仅使用期望回报来概括相应的回报分布，而并未考虑分布的离散程度。我们的研究将这种离散程度定义为策略的可复现性：即策略在多次执行时获得相似性能的能力，这是某些实际应用中的关键属性。我们指出，现有仅使用期望回报的评估流程存在两方面局限：首先，存在无限种具有不同性能-可复现性权衡的回报分布，其期望回报可能完全相同，这限制了其在策略比较中的有效性；其次，期望回报指标无法为从业者提供空间，以针对具体应用选择最佳权衡值。针对这些局限，本研究推荐使用置信下界这一源自贝叶斯优化的指标，该指标通过提供偏好参数，允许用户自主选择期望的性能-可复现性权衡。此外，我们正式定义了策略可复现性并对其进行量化分析，通过在常见不确定性RL任务上对主流RL算法进行大量实验，验证了所提指标的优势。