As increasingly capable agents are deployed, a central safety question is how to retain meaningful human control without modifying the underlying system. We study a minimal control interface where an agent chooses whether to act autonomously (play) or defer (ask), while a human simultaneously chooses whether to be permissive (trust) or to engage in oversight (oversee). If the agent defers, the human's choice determines the outcome, potentially leading to a corrective action or a system shutdown. We model this interaction as a two-player Markov Game. Our analysis focuses on cases where this game qualifies as a Markov Potential Game (MPG), a class of games where we can provide an alignment guarantee: under a structural assumption on the human's value function, any decision by the agent to act more autonomously that benefits itself cannot harm the human's value. We also analyze extensions to this MPG framework. Theoretically, this perspective provides conditions for a specific form of intrinsic alignment. If the reward structures of the human-agent game meet these conditions, we have a formal guarantee that the agent improving its own outcome will not harm the human's. Practically, this model motivates a transparent control layer with predictable incentives where the agent learns to defer when risky and act when safe, while its pretrained policy and the environment's reward structure remain untouched. Our gridworld simulation shows that through independent learning, the agent and human discover their optimal oversight roles. The agent learns to ask when uncertain and the human learns when to oversee, leading to an emergent collaboration that avoids safety violations introduced post-training. This demonstrates a practical method for making misaligned models safer after deployment.

翻译：随着能力日益增强的智能体被部署应用，一个核心的安全问题在于如何在不修改底层系统的前提下保持有效的人类控制。我们研究了一种最小化控制接口：智能体选择自主行动（执行）或请求协助（询问），而人类同时选择允许操作（信任）或介入监督（监督）。若智能体请求协助，人类的选择将决定结果，可能引发纠正性操作或系统停机。我们将这种交互建模为双人马尔可夫博弈。研究重点分析该博弈符合马尔可夫势博弈条件的情形——这类博弈可提供对齐性保证：在人类价值函数满足结构性假设的前提下，智能体任何提升自身收益的自主性增强决策都不会损害人类价值。我们同时拓展分析了该MPG框架的延伸形式。理论上，这一视角为特定形式的内在对齐提供了条件约束：若人机博弈的奖励结构满足这些条件，我们可获得形式化保证——智能体优化自身收益时不会损害人类利益。实践层面，该模型促成了一个具有可预测激励的透明控制层：智能体学会在风险场景中请求协助，在安全场景中自主行动，而其预训练策略与环境奖励结构保持不变。网格世界仿真表明，通过独立学习，智能体与人类能自主发现最优监督角色：智能体学会在不确定时询问，人类学会适时监督，从而形成一种避免训练后引入安全违规的涌现式协作机制。这为部署后未对齐模型的安全增强提供了一种实用方法。