Many strategic decision-making problems, such as environment design for warehouse robots, can be naturally formulated as bi-level reinforcement learning (RL), where a leader agent optimizes its objective while a follower solves a Markov decision process (MDP) conditioned on the leader's decisions. In many situations, a fundamental challenge arises when the leader cannot intervene in the follower's optimization process; it can only observe the optimization outcome. We address this decentralized setting by deriving the hypergradient of the leader's objective, i.e., the gradient of the leader's strategy that accounts for changes in the follower's optimal policy. Unlike prior hypergradient-based methods that require extensive data for repeated state visits or rely on gradient estimators whose complexity can increase substantially with the high-dimensional leader's decision space, we leverage the Boltzmann covariance trick to derive an alternative hypergradient formulation. This enables efficient hypergradient estimation solely from interaction samples, even when the leader's decision space is high-dimensional. Additionally, to our knowledge, this is the first method that enables hypergradient-based optimization for 2-player Markov games in decentralized settings. Experiments highlight the impact of hypergradient updates and demonstrate our method's effectiveness in both discrete and continuous state tasks.

翻译：许多战略决策问题，例如仓库机器人的环境设计，可以自然地表述为双层强化学习（RL），其中领导者智能体优化其目标，而跟随者在给定领导者决策的条件下求解一个马尔可夫决策过程（MDP）。在许多情况下，当领导者无法干预跟随者的优化过程，而只能观察优化结果时，一个根本性挑战便会出现。我们通过推导领导者目标的超梯度（即考虑跟随者最优策略变化时领导者策略的梯度）来解决这一去中心化设定。与先前基于超梯度的方法（这些方法需要大量数据以实现重复状态访问，或者依赖于其复杂度会随领导者高维决策空间而显著增加的梯度估计器）不同，我们利用玻尔兹曼协方差技巧推导出一种替代的超梯度公式。这使得仅通过交互样本即可高效估计超梯度，即使在领导者决策空间为高维时也是如此。此外，据我们所知，这是第一种能够在去中心化设定下为2-玩家马尔可夫博弈实现基于超梯度优化的方法。实验突出了超梯度更新的影响，并展示了我们的方法在离散和连续状态任务中的有效性。