The rapid development of Large Language Models (LLMs) has substantially expanded the range of tasks they can address. In the field of Natural Language Processing (NLP), researchers have shifted their focus from conventional NLP tasks (e.g., sequence tagging and parsing) towards tasks that revolve around aligning with human needs (e.g., brainstorming and email writing). This shift in task distribution imposes new requirements on evaluating these aligned models regarding generality (i.e., assessing performance across diverse scenarios), flexibility (i.e., examining under different protocols), and interpretability (i.e., scrutinizing models with explanations). In this paper, we propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios and accommodates diverse evaluation protocols (e.g., pairwise response comparison and single-response evaluation) with well-structured natural language critiques. To demonstrate the efficacy of our approach, we construct a new testbed covering 58 different scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models, by a large margin. We also provide detailed analysis and case studies to further reveal the potential of our method and make a variety of resources public at https://github.com/GAIR-NLP/auto-j.

翻译：大规模语言模型的快速发展极大地扩展了其可处理的任务范围。在自然语言处理领域，研究者已将关注点从传统NLP任务（如序列标注和句法分析）转向围绕人类需求对齐的任务（如头脑风暴和邮件撰写）。这种任务分布的转变对评估这些对齐模型提出了新要求，涉及通用性（即评估跨场景性能）、灵活性（即在不同协议下检验）和可解释性（即通过解释审查模型）。本文提出一个具有130亿参数的生成式裁判Auto-J，旨在应对这些挑战。我们的模型基于真实世界海量场景下的用户查询和LLM生成响应进行训练，并通过结构良好的自然语言评语支持多样化的评估协议（如成对响应比较和单响应评估）。为验证方法有效性，我们构建了覆盖58个不同场景的新测试平台。实验表明，Auto-J以显著优势超越了包括开源和闭源模型在内的一系列强基线。我们还通过详细分析与案例研究进一步揭示方法潜力，并在https://github.com/GAIR-NLP/auto-j公开了多种资源。