Evaluating the quality of multi-turn conversations is crucial for developing capable Large Language Models (LLMs), yet remains a significant challenge, often requiring costly human evaluation. Multi-turn reward models (RMs) offer a scalable alternative and can provide valuable signals for guiding LLM training. While recent work has advanced multi-turn \textit{training} techniques, effective automated \textit{evaluation} specifically for multi-turn interactions lags behind. We observe that standard preference datasets, typically contrasting responses based only on the final conversational turn, provide insufficient signal to capture the nuances of multi-turn interactions. Instead, we find that incorporating contrasts spanning \textit{multiple} turns is critical for building robust multi-turn RMs. Motivated by this finding, we propose \textbf{MU}lti-\textbf{S}tep \textbf{I}nstruction \textbf{C}ontrast (MUSIC), an unsupervised data augmentation strategy that synthesizes contrastive conversation pairs exhibiting differences across multiple turns. Leveraging MUSIC on the Skywork preference dataset, we train a multi-turn RM based on the Gemma-2-9B-Instruct model. Empirical results demonstrate that our MUSIC-augmented RM outperforms baseline methods, achieving higher alignment with judgments from advanced proprietary LLM judges on multi-turn conversations, crucially, without compromising performance on standard single-turn RM benchmarks.

翻译：评估多轮对话的质量对于开发能力强的大型语言模型（LLM）至关重要，但这仍然是一个重大挑战，通常需要昂贵的人工评估。多轮奖励模型（RM）提供了一种可扩展的替代方案，并能为指导LLM训练提供有价值的信号。尽管近期研究在多轮训练技术方面取得了进展，但专门针对多轮交互的有效自动化评估仍然滞后。我们观察到，标准的偏好数据集通常仅基于对话的最终轮次对比回复，这不足以捕捉多轮交互的细微差别。相反，我们发现，纳入跨越多个轮次的对比对于构建稳健的多轮RM至关重要。基于这一发现，我们提出了多步指令对比（MUSIC），这是一种无监督数据增强策略，能够合成在多轮对话中表现出差异的对比性对话对。利用MUSIC对Skywork偏好数据集进行处理，我们基于Gemma-2-9B-Instruct模型训练了一个多轮RM。实证结果表明，我们通过MUSIC增强的RM优于基线方法，在多轮对话上与先进的专有LLM评判者的判断实现了更高的一致性，关键的是，这并未损害其在标准单轮RM基准测试上的性能。