Large Language Models (LLMs) can justify or criticize their predictions through discussion with other models or humans, thereby enhancing their intrinsic understanding of instances. While proactive discussions enhance performance, this approach is currently limited to the inference phase. In this context, we posit a hypothesis: learning interactive discussions during training can improve understanding for the instances in the training step and proficiency in logical/critical thinking ability and verbalized expression of the model in the inference step. Our proposed SAIE training method involves both supportive and adversarial discussions between the learner and partner models. The learner model receives a remark from the partner through the discussion, and the parameters of the learner model are then updated based on this remark. That is, the teacher signal dynamically adjusts in response to the evolving model output throughout the training step. By bolstering the capacity for discussion and comprehension of instances, our experiments across datasets, including GSM8K, CommonsenseQA, and MMLU, reveal that models fine-tuned with our method consistently surpass those trained with standard fine-tuning techniques. Moreover, our approach demonstrates superior performance in multi-agent inference scenarios, boosting the models' reasoning abilities at the inference step.

翻译：大语言模型可通过与其他模型或人类讨论来论证或批判自身预测，从而增强对实例的内在理解。虽然主动式讨论能提升性能，但该方法目前局限于推理阶段。基于此，我们提出假设：在训练阶段学习交互式讨论，既能提升模型对训练实例的理解能力，又能强化模型在推理阶段的逻辑推理、批判性思维及语言表达能力。我们提出的SAIE训练方法包含学习模型与伙伴模型之间的支持性讨论与对抗性讨论。学习模型通过讨论接收伙伴模型的评论，并根据该评论更新自身参数——即教师信号会随训练过程中模型输出的动态变化而自适应调整。通过增强讨论能力与实例理解能力，我们在GSM8K、CommonsenseQA和MMLU等数据集上的实验表明，经本方法微调的模型始终优于采用标准微调技术的模型。此外，本方法在多智能体推理场景中展现出更优性能，显著提升了模型推理阶段的推理能力。