Existing LLM-as-a-Judge systems suffer from three fundamental limitations: limited adaptivity to task- and domain-specific evaluation criteria, systematic biases driven by non-semantic cues such as position, length, format, and model provenance, and evaluation inconsistency that leads to contradictory judgments across different evaluation modes (e.g., pointwise versus pairwise). To address these issues, we propose FairJudge, an adaptive, debiased, and consistent LLM-as-a-Judge. Unlike prior approaches that treat the judge as a static evaluator, FairJudge models judging behavior itself as a learnable and regularized policy. From a data-centric perspective, we construct a high-information-density judging dataset that explicitly injects supervision signals aligned with evaluation behavior. Building on this dataset, we adopt a curriculum-style SFT-DPO-GRPO training paradigm that progressively aligns rubric adherence, bias mitigation, and cross-mode consistency, while avoiding catastrophic forgetting. Experimental results on multiple internal and public benchmarks show that FairJudge consistently improves agreement and F1, reduces non-semantic biases, and outperforms substantially larger instruction-tuned LLMs. All resources will be publicly released after acceptance to facilitate future research.

翻译：现有的LLM-as-a-Judge系统存在三个根本性局限：对任务和领域特定评估标准的适应能力有限；受位置、长度、格式和模型来源等非语义线索驱动的系统性偏差；以及评估不一致性导致在不同评估模式（例如逐点评估与成对评估）下产生矛盾的判断。为解决这些问题，我们提出FairJudge，一种自适应、去偏且一致的LLM-as-a-Judge评估系统。与先前将评估者视为静态评价器的方法不同，FairJudge将评估行为本身建模为可学习且可正则化的策略。从数据中心的视角出发，我们构建了一个高信息密度的评估数据集，该数据集显式注入了与评估行为对齐的监督信号。基于此数据集，我们采用课程式SFT-DPO-GRPO训练范式，逐步对齐评分标准遵循度、偏差缓解和跨模式一致性，同时避免灾难性遗忘。在多个内部和公开基准上的实验结果表明，FairJudge持续提升了评估一致性和F1分数，降低了非语义偏差，并且显著优于规模更大的指令微调大语言模型。所有资源将在论文录用后公开发布，以促进未来研究。