As agentic AI becomes more widespread, agents with distinct and possibly conflicting goals will interact in complex ways. These multi-agent interactions pose a fundamental challenge, particularly in social dilemmas, where agents' individual incentives can undermine collective welfare. While reinforcement learning (RL) has been effective for aligning large language models (LLMs) in the single-agent regime, prior small-network results suggest that standard RL in multi-agent settings often converges to defecting, self-interested policies. We show the same effect in LLMs: despite cooperative priors, RL-trained LLM agents develop opportunistic behavior that can exploit even advanced closed-source models. To address this tendency of RL to converge to poor equilibria, we adapt a recent opponent-learning awareness algorithm, Advantage Alignment, to fine-tune LLMs toward multi-agent cooperation and non-exploitability. We then introduce a group-relative baseline that simplifies advantage computation in iterated games, enabling multi-agent training at LLM scale. We also contribute a novel social dilemma environment, Trust and Split, which requires natural language communication to achieve high collective welfare. Across a wide range of social dilemmas, policies learned with Advantage Alignment achieve higher collective payoffs while remaining robust against exploitation by greedy agents.

翻译：随着智能体人工智能的日益普及，具有不同且可能冲突目标的智能体将以复杂方式交互。这些多智能体交互构成了一个根本性挑战，尤其是在社会困境中，智能体的个体激励可能损害集体福利。虽然强化学习（RL）在单智能体场景中对齐大型语言模型（LLM）方面表现有效，但先前的小规模网络研究结果表明，标准RL在多智能体设置中通常收敛于背叛性的自利策略。我们在LLM中观察到相同效应：尽管存在合作先验，但经过RL训练的LLM智能体会发展出机会主义行为，甚至可能利用先进的闭源模型。为应对RL收敛至不良均衡的趋势，我们采用了一种近期提出的对手学习感知算法——优势对齐（Advantage Alignment），对LLM进行微调以促进多智能体合作与抗剥削性。随后，我们引入了一种群体相对基线，简化了迭代博弈中的优势计算，从而实现了LLM规模的多智能体训练。我们还贡献了一个新颖的社会困境环境“信任与分割”（Trust and Split），该环境需要自然语言沟通才能实现高集体福利。在广泛的社会困境场景中，通过优势对齐学习到的策略在保持对贪婪智能体剥削鲁棒性的同时，实现了更高的集体收益。