Multi-agent reinforcement learning (MARL) offers a scalable alternative to exact game-theoretic analysis but suffers from non-stationarity and the need to maintain diverse populations of strategies that capture non-transitive interactions. Policy Space Response Oracles (PSRO) address these issues by iteratively expanding a restricted game with approximate best responses (BRs), yet per-agent BR training makes it prohibitively expensive in many-agent or simulator-expensive settings. We introduce Joint Experience Best Response (JBR), a drop-in modification to PSRO that collects trajectories once under the current meta-strategy profile and reuses this joint dataset to compute BRs for all agents simultaneously. This amortizes environment interaction and improves the sample efficiency of best-response computation. Because JBR converts BR computation into an offline RL problem, we propose three remedies for distribution-shift bias: (i) Conservative JBR with safe policy improvement, (ii) Exploration-Augmented JBR that perturbs data collection and admits theoretical guarantees, and (iii) Hybrid BR that interleaves JBR with periodic independent BR updates. Across benchmark multi-agent environments, Exploration-Augmented JBR achieves the best accuracy-efficiency trade-off, while Hybrid BR attains near-PSRO performance at a fraction of the sample cost. Overall, JBR makes PSRO substantially more practical for large-scale strategic learning while preserving equilibrium robustness.

翻译：多智能体强化学习（MARL）为精确博弈论分析提供了一种可扩展的替代方案，但面临着非平稳性以及需要维持捕捉非传递性交互的多样化策略种群的问题。策略空间响应预言机（PSRO）通过迭代扩展具有近似最优响应（BR）的受限博弈来解决这些问题，然而每个智能体的BR训练使其在多智能体或模拟器成本高昂的场景中代价过高。我们提出了联合经验最优响应（JBR），这是对PSRO的一种即插即用式改进，它在当前元策略配置下一次性收集轨迹，并重用该联合数据集同时计算所有智能体的BR。这分摊了环境交互成本，并提高了最优响应计算的样本效率。由于JBR将BR计算转化为离线强化学习问题，我们针对分布偏移偏差提出了三种解决方案：（i）采用安全策略改进的保守型JBR，（ii）通过扰动数据收集并具备理论保证的探索增强型JBR，以及（iii）将JBR与周期性独立BR更新交替进行的混合型BR。在基准多智能体环境中，探索增强型JBR实现了最佳的精度-效率权衡，而混合型BR以极低的样本成本达到了接近PSRO的性能。总体而言，JBR在保持均衡鲁棒性的同时，显著提升了PSRO在大规模策略学习中的实用性。