LLM-as-a-Judge has been widely applied to evaluate and compare different LLM alignmnet approaches (e.g., RLHF and DPO). However, concerns regarding its reliability have emerged, due to LLM judges' biases and inconsistent decision-making. Previous research has developed evaluation frameworks to assess reliability of LLM judges and their alignment with human preferences. However, the employed evaluation metrics often lack adequate explainability and fail to address LLM internal inconsistency. Additionally, existing studies inadequately explore the impact of various prompt templates when applying LLM-as-a-Judge methods, leading to potentially inconsistent comparisons between different alignment algorithms. In this work, we systematically evaluate LLM-as-a-Judge on alignment tasks by defining more theoretically interpretable evaluation metrics and explicitly mitigating LLM internal inconsistency from reliability metrics. We develop an open-source framework to evaluate, compare, and visualize the reliability and alignment of LLM judges, which facilitates practitioners to choose LLM judges for alignment tasks. In the experiments, we examine effects of diverse prompt templates on LLM-judge reliability and also demonstrate our developed framework by comparing various LLM judges on two common alignment datasets (i.e., TL;DR Summarization and HH-RLHF-Helpfulness). Our results indicate a significant impact of prompt templates on LLM judge performance, as well as a mediocre alignment level between the tested LLM judges and human evaluators.

翻译：LLM作为评判者已被广泛应用于评估和比较不同的LLM对齐方法（例如RLHF和DPO）。然而，由于LLM评判者存在的偏见和决策不一致性，其可靠性引发了担忧。先前的研究已开发出评估框架来衡量LLM评判者的可靠性及其与人类偏好的对齐程度。然而，所采用的评估指标往往缺乏足够的可解释性，且未能解决LLM内部不一致性问题。此外，现有研究对应用LLM作为评判者方法时各种提示模板的影响探索不足，导致不同对齐算法之间的比较可能存在不一致性。在本工作中，我们通过定义更具理论可解释性的评估指标，并明确从可靠性指标中缓解LLM内部不一致性，系统评估了LLM作为评判者在对齐任务中的表现。我们开发了一个开源框架，用于评估、比较和可视化LLM评判者的可靠性与对齐程度，以帮助实践者为对齐任务选择合适的LLM评判者。在实验中，我们检验了多样化提示模板对LLM评判者可靠性的影响，并通过在两个常见对齐数据集（即TL;DR摘要和HH-RLHF-Helpfulness）上比较多种LLM评判者，展示了我们开发的框架。结果表明，提示模板对LLM评判者性能有显著影响，且所测试的LLM评判者与人类评估者之间的对齐程度处于中等水平。