Despite the utility of Large Language Models (LLMs) across a wide range of tasks and scenarios, developing a method for reliably evaluating LLMs across varied contexts continues to be challenging. Modern evaluation approaches often use LLMs to assess responses generated by LLMs. However, the meta-evaluation conducted to assess the effectiveness of these LLMs as evaluators is typically constrained by the coverage of existing benchmarks or requires extensive human annotation. This underscores the urgency of methods for scalable meta-evaluation that can effectively, reliably, and efficiently evaluate the performance of LLMs as evaluators across diverse tasks and scenarios, particularly in potentially new, user-defined scenarios. To fill this gap, we propose ScaleEval, an agent-debate-assisted meta-evaluation framework that leverages the capabilities of multiple communicative LLM agents. This framework supports multi-round discussions to assist human annotators in discerning the most capable LLMs as evaluators, which significantly eases their workload in cases that used to require large-scale annotations during meta-evaluation. We release the code for our framework, which is publicly available at: \url{https://github.com/GAIR-NLP/scaleeval}.

翻译：尽管大型语言模型（LLMs）在广泛的任务和场景中展现了其实用性，但开发一种能够在不同背景下可靠评估LLMs的方法仍然具有挑战性。现代评估方法常利用LLMs来评估其他LLMs生成的响应。然而，用于评估这些LLM作为评估器有效性的元评估通常受限于现有基准的覆盖范围，或需要大量的人工标注。这凸显了对可扩展元评估方法的迫切需求，该方法能够有效、可靠且高效地评估LLMs作为评估器在不同任务和场景中的表现，尤其是在可能新兴的、用户自定义的场景中。为填补这一空白，我们提出了ScaleEval——一个基于智能体辩论辅助的元评估框架，利用多个具备通信能力的LLM智能体。该框架支持多轮讨论，以协助人工标注者甄别出最胜任的LLM评估器，从而在原本需要大规模标注的元评估案例中显著减轻其工作负担。我们已公开发布该框架的代码，地址为：\url{https://github.com/GAIR-NLP/scaleeval}。