Mutual adaptation is a central challenge in human-AI teaming, as humans naturally adjust their strategies in response to an AI agent's behavior. Existing approaches attempt to approximate human behavior by diversifying training partners; however, these partners are typically static and fail to capture the adaptive nature of human teammates. When agents are trained jointly in standard multi-agent settings, they often converge to opaque coordination strategies that work only with their co-trained partners, leading to poor generalization. To model adaptive human behavior, we formulate human-AI teaming as an Interactive Partially Observable Markov Decision Process (I-POMDP). We propose NestRL, a nested training regime that learns the solution to a finite-level I-POMDP by training agents at each level against adaptive agents from the level below. This exposes agents to adaptive behavior while preventing emergence of opaque coordination strategies. We provide theoretical analysis showing that NestRL agents avoid convergence to partner-specific strategies, and validate this empirically in the Overcooked domain against state-of-the-art baselines. NestRL achieves higher task performance with both unseen adaptive agents and real human teammates, while exhibiting significantly greater adaptability over the course of interaction.

翻译：互适性是人机协同中的核心挑战，因为人类会基于AI智能体的行为自然调整自身策略。现有方法试图通过多样化的训练伙伴来近似人类行为，但这些伙伴通常是静态的，无法捕捉人类队友的适应特性。当智能体在标准多智能体环境中联合训练时，它们常会收敛于仅能与共训伙伴协作的隐性协同策略，导致泛化能力低下。为建模人类适应性行为，我们将人机协同形式化为交互式部分可观测马尔可夫决策过程（I-POMDP）。我们提出NestRL——一种嵌套式训练范式，通过训练每一层级的智能体与低一层级的自适应智能体对抗，从而学习有限层级I-POMDP的解决方案。该方法既使智能体暴露于适应行为下，又防止了隐性协同策略的产生。理论分析表明，NestRL智能体可避免收敛至伙伴特异性策略，我们在Overcooked环境中针对最先进基线模型进行了实证验证。无论是与未见过的自适应智能体还是真实人类队友合作，NestRL均能获得更高的任务性能，并在交互过程中展现出显著增强的适应性。