While deep reinforcement learning has achieved tremendous successes in various applications, most existing works only focus on maximizing the expected value of total return and thus ignore its inherent stochasticity. Such stochasticity is also known as the aleatoric uncertainty and is closely related to the notion of risk. In this work, we make the first attempt to study risk-sensitive deep reinforcement learning under the average reward setting with the variance risk criteria. In particular, we focus on a variance-constrained policy optimization problem where the goal is to find a policy that maximizes the expected value of the long-run average reward, subject to a constraint that the long-run variance of the average reward is upper bounded by a threshold. Utilizing Lagrangian and Fenchel dualities, we transform the original problem into an unconstrained saddle-point policy optimization problem, and propose an actor-critic algorithm that iteratively and efficiently updates the policy, the Lagrange multiplier, and the Fenchel dual variable. When both the value and policy functions are represented by multi-layer overparameterized neural networks, we prove that our actor-critic algorithm generates a sequence of policies that finds a globally optimal policy at a sublinear rate. Further, We provide numerical studies of the proposed method using two real datasets to back up the theoretical results.

翻译：尽管深度强化学习在各类应用中取得了巨大成功，但现有工作大多仅关注总回报期望值的最大化，而忽略了其内在的随机性。这种随机性也被称为偶然不确定性，与风险概念密切相关。本文首次尝试研究平均奖励设定下基于方差风险准则的风险敏感深度强化学习。具体而言，我们聚焦于一个方差约束策略优化问题，其目标是在长期平均奖励的方差不超过阈值约束下，最大化长期平均奖励的期望值。利用拉格朗日对偶性和芬切尔对偶性，我们将原问题转化为无约束鞍点策略优化问题，并提出一种能够迭代高效更新策略、拉格朗日乘子和芬切尔对偶变量的演员-评论家算法。当值函数和策略函数均由多层过参数化神经网络表示时，我们证明该演员-评论家算法能以次线性速率生成全局最优策略序列。此外，我们利用两个真实数据集开展了数值实验，验证了理论结果。