Large language models (LLMs) are increasingly relied upon for complex multi-turn conversations across diverse real-world applications. However, existing benchmarks predominantly focus on single-turn evaluations, overlooking the models' capabilities in multi-turn interactions. To address this gap, we introduce MT-Eval, a comprehensive benchmark designed to evaluate multi-turn conversational abilities. By analyzing human-LLM conversations, we categorize interaction patterns into four types: recollection, expansion, refinement, and follow-up. We construct multi-turn queries for each category either by augmenting existing datasets or by creating new examples with GPT-4 to avoid data leakage. To study the factors impacting multi-turn abilities, we create single-turn versions of the 1170 multi-turn queries and compare performance. Our evaluation of 11 well-known LLMs shows that while closed-source models generally surpass open-source ones, certain open-source models exceed GPT-3.5-Turbo in specific tasks. We observe significant performance degradation in multi-turn settings compared to single-turn settings in most models, which is not correlated with the models' fundamental capabilities. Moreover, we identify the distance to relevant content and susceptibility to error propagation as the key factors influencing multi-turn performance. MT-Eval is released publicly to encourage future research towards more robust conversational models.

翻译：大语言模型（LLMs）在诸多实际应用场景中日益被依赖以执行复杂的多轮对话。然而，现有基准评测主要聚焦于单轮评估，忽视了模型在多轮交互中的能力。为填补这一空白，我们提出MT-Eval——一个旨在评估多轮对话能力的综合性基准。通过分析人类与LLM的对话，我们将交互模式归纳为四类：回忆、扩展、精炼和追问。我们通过扩展现有数据集或利用GPT-4创建新示例以避免数据泄露，为每个类别构建多轮查询。为探究影响多轮能力的因素，我们为1170个多轮查询构建了对应的单轮版本并进行性能比较。对11个知名LLM的评估表明，尽管闭源模型普遍优于开源模型，但部分开源模型在特定任务上超越了GPT-3.5-Turbo。我们观察到，相较于单轮设置，多数模型在多轮设置下性能显著下降，且这种下降与模型的基础能力无关。此外，我们识别出与相关内容间的距离以及对错误传播的敏感性是影响多轮性能的关键因素。MT-Eval已公开发布，以鼓励未来对更鲁棒对话模型的研究。