While machine learning can myopically reinforce social inequalities, it may also be used to dynamically seek equitable outcomes. In this paper, we formalize long-term fairness in the context of online reinforcement learning. This formulation can accommodate dynamical control objectives, such as driving equity inherent in the state of a population, that cannot be incorporated into static formulations of fairness. We demonstrate that this framing allows an algorithm to adapt to unknown dynamics by sacrificing short-term incentives to drive a classifier-population system towards more desirable equilibria. For the proposed setting, we develop an algorithm that adapts recent work in online learning. We prove that this algorithm achieves simultaneous probabilistic bounds on cumulative loss and cumulative violations of fairness (as statistical regularities between demographic groups). We compare our proposed algorithm to the repeated retraining of myopic classifiers, as a baseline, and to a deep reinforcement learning algorithm that lacks safety guarantees. Our experiments model human populations according to evolutionary game theory and integrate real-world datasets.

翻译：虽然机器学习可能短视地强化社会不平等，但它也可以被用来动态地追求公平结果。在本文中，我们在在线强化学习的背景下形式化了长期公平性。这一形式化能够容纳动态控制目标，例如驱动人口状态中固有的公平性，这些目标无法被纳入静态的公平性表述中。我们证明，这种框架允许算法通过牺牲短期激励来适应未知动态，从而推动分类器-人口系统朝向更理想的均衡状态。针对所提出的设定，我们开发了一种算法，该算法改编了在线学习领域的最新工作。我们证明，该算法在累积损失和公平性违规（作为人口组之间的统计规律）上实现了同时的概率界限。我们将所提出的算法与短视分类器的重复训练（作为基线）以及缺乏安全保证的深度强化学习算法进行了比较。我们的实验根据演化博弈论对人群进行建模，并整合了真实世界的数据集。