Multi-Agent Systems (MAS) built on Large Language Models (LLMs) require effective orchestration to coordinate specialized agents, yet training such orchestrators is hindered by limited supervision and high computational cost. We propose Orchestration Reward Modeling (OrchRM), a self-supervised framework for evaluating orchestration quality without human annotations. OrchRM leverages intermediate artifacts from multi-agent executions to construct win-lose pairs for Bradley-Terry reward model training. Unlike existing MAS test-time scaling and orchestrator training frameworks that rely on costly sub-agent rollouts, OrchRM operates directly at the orchestration level, enabling efficient and high-performing reward-guided orchestrator training and MAS test-time scaling. OrchRM improves training efficiency by up to 10x in token usage while improving MAS test-time scaling performance by up to 8% in accuracy. These gains consistently transfer across multiple domains, including mathematical reasoning, web-based question answering, and multi-hop reasoning, demonstrating orchestration-level reward modeling as a scalable direction for robust multi-agent orchestration. Code will be available at https://github.com/Wang-ML-Lab/OrchRM.

翻译：基于大语言模型(LLM)的多智能体系统(MAS)需要高效的编排机制来协调专业智能体，然而训练此类编排器面临监督信号有限和计算成本高昂的挑战。本文提出了编排奖励建模(OrchRM)——一种无需人工标注即可评估编排质量的自监督框架。OrchRM利用多智能体执行过程中的中间产物构建Bradley-Terry奖励模型训练的胜负对。与依赖昂贵子智能体展开的现有MAS测试时扩展及编排器训练框架不同，OrchRM直接在编排层运作，从而支持高效且高性能的奖励引导式编排器训练与MAS测试时扩展。OrchRM在令牌使用量上提升训练效率高达10倍，同时在准确率上使MAS测试时扩展性能提升最高8%。这些增益在数学推理、基于网页的问答和多跳推理等多个领域实现一致迁移，证明了编排级奖励建模作为鲁棒多智能体编排的可扩展方向的有效性。代码将发布于https://github.com/Wang-ML-Lab/OrchRM。