Collaborative multi-agent large language models (LLMs) can solve complex reasoning tasks by decomposing roles, but reinforcement learning for such systems is limited by credit assignment: shared terminal rewards obscure individual contributions and can encourage free-riding. We introduce two optimizer-agnostic credit assignment methods for converting joint outcomes into agent-specific learning signals. Counterfactual Credit for Policy Optimization (CCPO) estimates an agent's marginal contribution by comparing the realized joint outcome with a counterfactual outcome where that agent is removed. Self-Evaluated Credit for Policy Optimization (SEPO) uses constrained self- and peer-evaluations as a verifier-anchored credit signal while keeping the external task outcome dominant. Both operate at the reward-construction layer rather than as policy optimizers, producing role-specific rewards or advantages for GRPO, GSPO, or REINFORCE++. We instantiate these credit signals in a sequential Think--Solve setting and evaluate them on mathematical reasoning benchmarks. Results show that explicit credit assignment often improves dual-agent reasoning, especially on MATH500 and several out-of-distribution settings, while gains vary across models and datasets. Our code is available at: https://github.com/bhai114/ccpo.

翻译：协作式多智能体大语言模型（LLMs）通过角色分解可解决复杂推理任务，但此类系统的强化学习受限于信用分配问题：共享的终端奖励掩盖了个体贡献，可能助长搭便车行为。我们提出两种与优化器无关的信用分配方法，将联合结果转化为智能体专属的学习信号。策略优化的反事实信用（CCPO）通过比较实际联合结果与移除该智能体的反事实结果，估计智能体的边际贡献。策略优化的自评估信用（SEPO）使用受约束的自我评估与同伴评估作为基于验证器的锚定信用信号，同时保持外部任务结果的主导性。这两种方法在奖励构建层面而非策略优化器层面运作，可为GRPO、GSPO或REINFORCE++生成角色专属的奖励或优势函数。我们在序列化"思考-求解"场景中实例化这些信用信号，并在数学推理基准上评估其性能。结果表明，明确的信用分配通常能提升双智能体推理能力，尤其在MATH500及多个分布外场景中表现显著，但增益效果因模型和数据集而异。我们的代码已开源：https://github.com/bhai114/ccpo。