Multi-agent football poses an unsolved challenge in AI research. Existing work has focused on tackling simplified scenarios of the game, or else leveraging expert demonstrations. In this paper, we develop a multi-agent system to play the full 11 vs. 11 game mode, without demonstrations. This game mode contains aspects that present major challenges to modern reinforcement learning algorithms; multi-agent coordination, long-term planning, and non-transitivity. To address these challenges, we present TiZero; a self-evolving, multi-agent system that learns from scratch. TiZero introduces several innovations, including adaptive curriculum learning, a novel self-play strategy, and an objective that optimizes the policies of multiple agents jointly. Experimentally, it outperforms previous systems by a large margin on the Google Research Football environment, increasing win rates by over 30%. To demonstrate the generality of TiZero's innovations, they are assessed on several environments beyond football; Overcooked, Multi-agent Particle-Environment, Tic-Tac-Toe and Connect-Four.

翻译：多智能体足球是AI研究中一个尚未解决的挑战。现有工作主要聚焦于游戏的简化场景，或利用专家演示。本文提出一个无需演示即可应对完整11对11比赛模式的多智能体系统。该模式包含对现代强化学习算法构成重大挑战的多个方面：多智能体协调、长期规划及非传递性。为应对这些挑战，我们提出TiZero——一个从零开始学习的自进化多智能体系统。TiZero引入多项创新，包括自适应课程学习、新型自博弈策略，以及联合优化多个智能体策略的目标函数。实验表明，在Google Research Football环境中，它大幅超越先前系统，胜率提升超过30%。为验证TiZero创新的通用性，我们在足球之外的多个环境中进行评估：Overcooked、多智能体粒子环境、井字棋及四子棋。