Credit assignment has remained a fundamental challenge in multi-agent reinforcement learning (MARL). Previous studies have primarily addressed this issue through value decomposition methods under the centralized training with decentralized execution paradigm, where neural networks are utilized to approximate the nonlinear relationship between individual Q-values and the global Q-value. Although these approaches have achieved considerable success in various benchmark tasks, they still suffer from several limitations, including imprecise attribution of contributions, limited interpretability, and poor scalability in high-dimensional state spaces. To address these challenges, we propose a novel algorithm, QLLM, which facilitates the automatic construction of credit assignment functions using large language models (LLMs). Specifically, the concept of TFCAF is introduced, wherein the credit allocation process is represented as a direct and expressive nonlinear functional formulation. A custom-designed coder-evaluator framework is further employed to guide the generation and verification of executable code by LLMs, significantly mitigating issues such as hallucination and shallow reasoning during inference. Furthermore, an IGM-Gating Mechanism enables QLLM to flexibly enforce or relax the monotonicity constraint depending on task demands, covering both IGM-compliant and non-monotonic scenarios. Extensive experiments conducted on several standard MARL benchmarks demonstrate that the proposed method consistently outperforms existing state-of-the-art baselines. Moreover, QLLM exhibits strong generalization capability and maintains compatibility with a wide range of MARL algorithms that utilize mixing networks, positioning it as a promising and versatile solution for complex multi-agent scenarios. The code is available at https://github.com/zhouyangjiang71-sys/QLLM.

翻译：信用分配一直是多智能体强化学习中的一个基本挑战。先前的研究主要通过在集中训练分散执行范式下的价值分解方法来解决这一问题，这些方法利用神经网络来近似个体Q值与全局Q值之间的非线性关系。尽管这些方法在各种基准任务中取得了显著成功，但仍存在若干局限性，包括贡献归属不精确、可解释性有限以及在状态空间高维时扩展性差等问题。为应对这些挑战，我们提出了一种新颖的算法QLLM，该算法利用大语言模型实现信用分配函数的自动构建。具体而言，我们引入了TFCAF概念，将信用分配过程表示为一种直接且富有表达力的非线性函数形式。进一步采用定制设计的编码器-评估器框架来引导大语言模型生成和验证可执行代码，显著缓解了推理过程中可能出现的幻觉和浅层推理等问题。此外，IGM门控机制使QLLM能够根据任务需求灵活地施加或放松单调性约束，覆盖了符合IGM要求和非单调性的多种场景。在多个标准多智能体强化学习基准上进行的广泛实验表明，所提方法始终优于现有的先进基线算法。此外，QLLM展现出强大的泛化能力，并与多种使用混合网络的多智能体强化学习算法保持兼容，使其成为复杂多智能体场景中一种极具前景且通用的解决方案。代码发布于https://github.com/zhouyangjiang71-sys/QLLM。