We present a scalable methodology for evaluating language models in multi-turn interactions, using a suite of collaborative games that require effective communication about private information. This enables an interactive scaling analysis, in which a fixed token budget is divided over a variable number of turns. We find that in many cases, language models are unable to use interactive collaboration to improve over the non-interactive baseline scenario in which one agent attempts to summarize its information and the other agent immediately acts -- despite substantial headroom. This suggests that state-of-the-art models still suffer from significant weaknesses in planning and executing multi-turn collaborative conversations. We analyze the linguistic features of these dialogues, assessing the roles of sycophancy, information density, and discourse coherence. While there is no single linguistic explanation for the collaborative weaknesses of contemporary language models, we note that humans achieve comparable task success at superior token efficiency by producing dialogues that are more coherent than those produced by most language models. The proactive management of private information is a defining feature of real-world communication, and we hope that MT-PingEval will drive further work towards improving this capability.

翻译：我们提出了一种可扩展的方法论，用于评估语言模型在多轮交互中的表现，该方法基于一系列需要就私有信息进行有效沟通的协作博弈。该方法支持交互式扩展分析，其中固定的令牌预算被分配至可变轮数的对话中。研究发现，在许多情况下，语言模型无法通过交互式协作超越非交互式基线场景（即一个智能体尝试总结其信息，另一智能体立即行动）的表现——尽管存在显著的提升空间。这表明当前最先进的模型在规划和执行多轮协作对话方面仍存在明显缺陷。我们分析了这些对话的语言特征，评估了迎合倾向、信息密度和话语连贯性所起的作用。虽然对于当代语言模型的协作缺陷尚无单一的语言学解释，但我们注意到，人类通过生成比大多数语言模型更连贯的对话，能以更高的令牌效率实现可比的任务成功率。主动管理私有信息是现实世界沟通的一个决定性特征，我们希望MT-PingEval能推动进一步的研究以提升模型的此项能力。