We study fair multi-objective reinforcement learning in which an agent must learn a policy that simultaneously achieves high reward on multiple dimensions of a vector-valued reward. Motivated by the fair resource allocation literature, we model this as an expected welfare maximization problem, for some nonlinear fair welfare function of the vector of long-term cumulative rewards. One canonical example of such a function is the Nash Social Welfare, or geometric mean, the log transform of which is also known as the Proportional Fairness objective. We show that even approximately optimal optimization of the expected Nash Social Welfare is computationally intractable even in the tabular case. Nevertheless, we provide a novel adaptation of Q-learning that combines nonlinear scalarized learning updates and non-stationary action selection to learn effective policies for optimizing nonlinear welfare functions. We show that our algorithm is provably convergent, and we demonstrate experimentally that our approach outperforms techniques based on linear scalarization, mixtures of optimal linear scalarizations, or stationary action selection for the Nash Social Welfare Objective.

翻译：我们研究公平的多目标强化学习，其中智能体必须学习一种策略，以在向量值奖励的多个维度上同时实现高回报。受公平资源分配文献的启发，我们将此建模为期望福利最大化问题，其中福利函数是关于长期累积奖励向量的某种非线性公平福利函数。这类函数的一个典型例子是纳什社会福利（Nash Social Welfare），即几何均值，其对数变换也被称为比例公平目标。我们证明，即使在表格情形下，对期望纳什社会福利进行近似最优优化在计算上也是难解的。尽管如此，我们提供了一种Q学习的创新性适应方法，它将非线性标量化学习更新与非平稳动作选择相结合，以学习优化非线性福利函数的有效策略。我们证明了该算法是收敛的，并通过实验表明，在处理纳什社会福利目标时，我们的方法优于基于线性标量化、最优线性标量化混合或平稳动作选择的技术。