Evaluating large language models (LLMs) is important for understanding their capabilities, comparing competing systems, and supporting the deployment of reliable models in practice. For open-ended tasks, pairwise evaluation has become a popular paradigm, in which two responses to the same prompt are compared and the resulting judgments are aggregated into an overall ranking. A central challenge of this paradigm is intransitivity: the induced comparison outcomes may fail to support any coherent global ranking. For example, one may observe cyclic preferences such as $A \succ B \succ C \succ A$, or inconsistencies involving ties such as $A \equiv B\equiv C\neq A$. Such contradictions make the resulting leaderboard unstable and challenging to interpret. In this paper, we propose a prompt perturbation framework for improving the consistency of pairwise LLM evaluation. Our approach generates perturbed variants of each prompt, uses the resulting comparison graphs to identify and filter out structurally inconsistent comparison patterns, and then applies standard ranking methods to the filtered comparisons. A key feature of the proposed framework is that graph-level structural consistency is incorporated explicitly into the evaluation pipeline before ranking aggregation. This provides a simple and principled way to reduce cyclic inconsistencies and improve the reliability of LLM rankings.

翻译：评估大语言模型（LLMs）对于理解其能力、比较竞争系统以及支持可靠模型在实际中的部署至关重要。对于开放式任务，成对评估已成为一种流行范式：通过比较同一提示下的两个响应，将所得判断聚合为整体排名。该范式的核心挑战在于不可传递性：诱导出的比较结果可能无法支持任何一致的全局排名。例如，可能出现循环偏好（如$A \succ B \succ C \succ A$）或涉及平局的不一致性（如$A \equiv B\equiv C\neq A$）。此类矛盾使得最终排行榜不稳定且难以解释。本文提出一种基于提示扰动的框架，用于提升成对LLM评估的一致性。该方法生成每个提示的扰动变体，利用所得比较图识别并过滤结构不一致的比较模式，随后对过滤后的比较结果应用标准排序方法。该框架的关键特征在于：在排序聚合前的评估流程中显式引入图级结构一致性。这为减少循环不一致性并提升LLM排名的可靠性提供了一种简单且原则性的方法。