In various applications, the optimal policy in a strategic decision-making problem depends both on the environmental configuration and exogenous events. For these settings, we introduce Bilevel Optimization with Contextual Markov Decision Processes (BO-CMDP), a stochastic bilevel decision-making model, where the lower level consists of solving a contextual Markov Decision Process (CMDP). BO-CMDP can be viewed as a Stackelberg Game where the leader and a random context beyond the leader's control together decide the setup of (many) MDPs that (potentially multiple) followers best respond to. This framework extends beyond traditional bilevel optimization and finds relevance in diverse fields such as model design for MDPs, tax design, reward shaping and dynamic mechanism design. We propose a stochastic Hyper Policy Gradient Descent (HPGD) algorithm to solve BO-CMDP, and demonstrate its convergence. Notably, HPGD only utilizes observations of the followers' trajectories. Therefore, it allows followers to use any training procedure and the leader to be agnostic of the specific algorithm used, which aligns with various real-world scenarios. We further consider the setting when the leader can influence the training of followers and propose an accelerated algorithm. We empirically demonstrate the performance of our algorithm.

翻译：在许多应用中，战略决策问题的最优策略既取决于环境配置，也取决于外生事件。针对这类场景，我们引入了具有上下文马尔可夫决策过程的双层优化（BO-CMDP），这是一个随机双层决策模型，其下层涉及求解一个上下文马尔可夫决策过程（CMDP）。BO-CMDP 可视为一个斯塔克尔伯格博弈，其中领导者与一个超出其控制范围的随机上下文共同决定（多个）MDP 的设置，而（潜在的多个）跟随者则对此做出最优响应。该框架超越了传统的双层优化，在 MDP 模型设计、税收设计、奖励塑形和动态机制设计等多个领域具有相关性。我们提出了一种随机超策略梯度下降（HPGD）算法来求解 BO-CMDP，并证明了其收敛性。值得注意的是，HPGD 仅利用跟随者轨迹的观测信息。因此，它允许跟随者使用任何训练过程，且领导者无需知晓所使用的具体算法，这与许多现实场景相符。我们进一步考虑了领导者能够影响跟随者训练的情况，并提出了一种加速算法。我们通过实验验证了算法的性能。