Despite the remarkable success of Large Language Models (LLMs), evaluating their outputs' quality regarding preference remains a critical challenge. While existing works usually leverage a strong LLM as the judge for comparing LLMs' response pairwisely, such a single-evaluator approach is vulnerable to cyclic preference, i.e., output A is better than B, B than C, but C is better than A, causing contradictory evaluation results. To address this, we introduce PGED (Preference Graph Ensemble and Denoising), a novel approach that leverages multiple model-based evaluators to construct preference graphs, and then ensembles and denoises these graphs for acyclic, non-contradictory evaluation results. We provide theoretical guarantees for our framework, demonstrating its efficacy in recovering the ground truth preference structure. Extensive experiments on ten benchmarks demonstrate PGED's superiority in three applications: 1) model ranking for evaluation, 2) response selection for test-time scaling, and 3) data selection for model fine-tuning. Notably, PGED combines small LLM evaluators (e.g., Llama3-8B, Mistral-7B, Qwen2-7B) to outperform strong ones (e.g., Qwen2-72B), showcasing its effectiveness in enhancing evaluation reliability and improving model performance.

翻译：尽管大型语言模型（LLM）取得了显著成功，但对其输出质量的偏好评估仍是一个关键挑战。现有研究通常采用一个强大的LLM作为评估器，对LLM的响应进行两两比较，但这种单一评估器方法容易产生循环偏好问题，即输出A优于B，B优于C，但C却优于A，导致评估结果相互矛盾。为解决这一问题，我们提出了PGED（偏好图集成与去噪），这是一种新颖的方法，利用多个基于模型的评估器构建偏好图，然后对这些图进行集成和去噪，从而获得无环且非矛盾的评估结果。我们为该框架提供了理论保证，证明了其在恢复真实偏好结构方面的有效性。在十个基准测试上的大量实验表明，PGED在三个应用场景中具有优越性：1）用于模型评估的模型排序，2）用于测试时扩展的响应选择，以及3）用于模型微调的数据选择。值得注意的是，PGED通过组合小型LLM评估器（如Llama3-8B、Mistral-7B、Qwen2-7B）能够超越强大的评估器（如Qwen2-72B），这展示了其在提升评估可靠性和改进模型性能方面的有效性。