LLM-as-a-judge approaches have emerged as a scalable solution for evaluating model behaviors, yet they rely on evaluation criteria often created by a single individual, embedding that person's assumptions, priorities, and interpretive lens. In practice, defining such criteria is a collaborative and contested process involving multiple stakeholders with different values, interpretations, and priorities; an aspect largely unsupported by existing tools. To examine this problem in depth, we present a formative study examining how stakeholders collaboratively create, negotiate, and refine evaluation criteria for LLM-as-a-judge systems. Our findings reveal challenges in human oversight, including difficulties in establishing shared understanding, aligning values across stakeholders with different expertise and priorities, and translating nuanced human judgments into criteria that are interpretable and actionable for LLM judges. Based on these insights, we developed MultEval, a system that supports collaborative criteria by enabling multiple evaluators to surface and diagnose disagreements using consensus-building theory, iteratively revise criteria with attached examples and proposal history, and maintain transparency over how judgments are encoded into an automated evaluator. We further report a case study in which a team of domain experts used MultEval to collaboratively author criteria, illustrating how coordination and collaborative consensus-making shape criteria evolution.

翻译：LLM-as-a-Judge方法已成为评估模型行为的可扩展解决方案，但其依赖的评估标准往往由单个个体制定，嵌入了该个体的假设、优先级和解读视角。在实践中，这类标准的定义是一个涉及多个利益相关者（持有不同价值观、解读方式和优先级）的协作与争议过程，现有工具在很大程度上未能支持这一方面。为深入探究该问题，我们开展了一项形成性研究，考察利益相关者如何协作创建、协商并完善LLM-as-a-Judge系统的评估标准。研究结果揭示了人工监督中的挑战，包括难以建立共同理解、难以在不同专业背景和优先级的利益相关者之间协调价值观、难以将细腻的人类判断转化为LLM法官可解读且可操作的评估标准。基于这些洞见，我们开发了MultEval系统，该系统通过支持多位评估者利用共识构建理论发现并诊断分歧、结合附带的示例和修订历史迭代完善标准、以及保持判断如何编码至自动评估器的透明性，从而支持协作式标准制定。我们进一步报告了一项案例研究：由领域专家组成的团队使用MultEval协作编写标准，展示了协作与共识构建如何塑造标准的演进。