We study multi-domain LLM training in which two models, each stronger in a different domain, co-evolve by tutoring each other through on-policy feedback. Unlike one-way distillation or single-model fine-tuning, our goal is mutual Pareto improvement: each model improves across domains without losing its original strength. To this end, we propose On-Policy Co-Distillation (OPCoD), where each student's self-distillation is conditioned on its own correct rollout and feedback from its peer. To make feedback exchange effective, OPCoD uses cognizance-based gating to decide when to give feedback and feedback anchoring to ground feedback in the problem. On Science Q\&A tasks, OPCoD consistently outperforms baselines and achieves Pareto improvement across all evaluated domain pairs and students.

翻译：本研究探索多领域大语言模型协同训练范式，其中两个各自在不同领域具有优势的模型通过同策略反馈实现相互指导与共同进化。区别于单向蒸馏或单模型微调方法，我们的目标是实现互惠帕累托改进：每个模型在保持自身原有优势的同时，提升跨领域表现。为此，我们提出同策略共蒸馏（OPCoD）框架，该框架中每个学生的自蒸馏过程同时依赖其自身的正确决策轨迹与同伴的反馈信息。为提升反馈交换效率，OPCoD采用认知门控机制决定何时提供反馈，并运用反馈锚定技术将反馈与具体问题相关联。在科学问答任务上的实验表明，OPCoD在所有评估领域配对和学生模型中均持续优于基线方法，并成功实现了帕累托改进。