In reinforcement learning (RL), rewards of states are typically considered additive, and following the Markov assumption, they are $\textit{independent}$ of states visited previously. In many important applications, such as coverage control, experiment design and informative path planning, rewards naturally have diminishing returns, i.e., their value decreases in light of similar states visited previously. To tackle this, we propose $\textit{submodular RL}$ (SubRL), a paradigm which seeks to optimize more general, non-additive (and history-dependent) rewards modelled via submodular set functions which capture diminishing returns. Unfortunately, in general, even in tabular settings, we show that the resulting optimization problem is hard to approximate. On the other hand, motivated by the success of greedy algorithms in classical submodular optimization, we propose SubPO, a simple policy gradient-based algorithm for SubRL that handles non-additive rewards by greedily maximizing marginal gains. Indeed, under some assumptions on the underlying Markov Decision Process (MDP), SubPO recovers optimal constant factor approximations of submodular bandits. Moreover, we derive a natural policy gradient approach for locally optimizing SubRL instances even in large state- and action- spaces. We showcase the versatility of our approach by applying SubPO to several applications, such as biodiversity monitoring, Bayesian experiment design, informative path planning, and coverage maximization. Our results demonstrate sample efficiency, as well as scalability to high-dimensional state-action spaces.

翻译：在强化学习中，状态的奖励通常被认为是可加的，并且遵循马尔可夫假设，它们与先前访问过的状态$\textit{独立}$。在许多重要应用中，例如覆盖控制、实验设计和信息路径规划，奖励自然具有递减效应，即其值会因先前访问过相似状态而降低。为解决这一问题，我们提出$\textit{次模强化学习}$（SubRL），这是一种旨在优化更一般、非可加（且依赖历史）奖励的范式，该奖励通过捕获递减效应的次模集合函数进行建模。不幸的是，我们证明即使在表格设置中，一般情况下的相关优化问题也难以近似。另一方面，受经典次模优化中贪心算法成功经验的启发，我们提出SubPO，一种基于策略梯度的简单算法，通过贪心地最大化边际增益来处理非可加奖励。实际上，在关于底层马尔可夫决策过程（MDP）的某些假设下，SubPO能够恢复次模赌博机的最优常数因子近似。此外，我们推导出一种自然的策略梯度方法，用于在状态和动作空间很大的情况下局部优化SubRL实例。通过将SubPO应用于多个应用场景（如生物多样性监测、贝叶斯实验设计、信息路径规划和覆盖最大化），我们展示了方法的通用性。实验结果证明了其样本效率及对高维状态-动作空间的可扩展性。