AI evaluation pipelines often produce conflicting judgments rather than clean labels. Yet existing methods largely collapse disagreement into a single hard label. As a result, item ambiguity, rater biases, and the position preferences of LLM judges are erased together: evaluation outcomes appear deterministic while concealing the underlying uncertainty, and may even systematically favor one response. This conflict is especially acute in pairwise LLM evaluation. We propose \textbf{BayesJudge}, an online Bayesian meta-evaluation layer for conflicting human-LLM judgment streams. BayesJudge treats the latent preference of each item as a latent variable and jointly models two sources of evaluator bias: rater-specific confusion matrices for human annotation bias, and an explicit order parameter for the position preference of LLM judges, thereby separating response quality from evaluator bias. We derive an exact online posterior recursion and adopt a scalable Rao-Blackwellized assumed-density sequential Monte Carlo (SMC) approximation for streaming inference. Consequently, each comparison yields not a single hard label but a posterior distribution over each response winning (as well as a tie or ambiguity state), together with posterior estimates of the bias parameters, making evaluation uncertainty explicit and quantifiable. On synthetic data, BayesJudge accurately recovers human annotation biases and LLM position biases online, and strictly outperforms variants that freeze either bias channel in held-out predictive log-likelihood. On the real-world SummEval dataset, BayesJudge detects systematic order effects in LLM judgments, distinguishes the behavioral signatures of experts and crowdworkers without rater metadata, and yields posterior uncertainty that reflects genuine human disagreement. Our code is available at https://github.com/Killerofthecard/BayesJudge.git

翻译：暂无翻译