The rapid development of Large Language Models (LLMs) has substantially expanded the range of tasks they can address. In the field of Natural Language Processing (NLP), researchers have shifted their focus from conventional NLP tasks (e.g., sequence tagging and parsing) towards tasks that revolve around aligning with human needs (e.g., brainstorming and email writing). This shift in task distribution imposes new requirements on evaluating these aligned models regarding generality (i.e., assessing performance across diverse scenarios), flexibility (i.e., examining under different protocols), and interpretability (i.e., scrutinizing models with explanations). In this paper, we propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios and accommodates diverse evaluation protocols (e.g., pairwise response comparison and single-response evaluation) with well-structured natural language critiques. To demonstrate the efficacy of our approach, we construct a new testbed covering 58 different scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models, by a large margin. We also provide detailed analysis and case studies to further reveal the potential of our method and make a variety of resources public at https://github.com/GAIR-NLP/auto-j.

翻译：大型语言模型（LLMs）的快速发展极大地扩展了它们所能处理的任务范围。在自然语言处理（NLP）领域，研究者的关注点已从传统NLP任务（如序列标注和句法分析）转向围绕与人类需求对齐的任务（如头脑风暴和邮件撰写）。这种任务分布的转变对评估这些对齐模型提出了新要求，涉及通用性（即评估跨不同场景的性能）、灵活性（即在不同协议下进行检查）和可解释性（即利用解释仔细审查模型）。在本文中，我们提出了一种拥有13B参数的生成式评估器Auto-J，旨在应对这些挑战。我们的模型基于真实世界海量场景中的用户查询和LLM生成的回答进行训练，并通过结构良好的自然语言评论文本适应多种评估协议（如成对回答比较和单回答评估）。为验证我们方法的有效性，我们构建了一个涵盖58种不同场景的新测试平台。实验结果显示，Auto-J以较大优势超越了一系列强竞争对手，包括开源和闭源模型。我们还提供了详细分析和案例研究，以进一步揭示我们方法的潜力，并在https://github.com/GAIR-NLP/auto-j上公开了多种资源。