Ensuring that large language models (LLMs) reflect diverse user values and preferences is crucial as their user bases expand globally. It is therefore encouraging to see the growing interest in LLM personalization within the research community. However, current works often rely on the LLM-as-a-Judge approach for evaluation without thoroughly examining its validity. In this paper, we investigate the reliability of LLM-as-a-Personalized-Judge, asking LLMs to judge user preferences based on personas. Our findings suggest that directly applying LLM-as-a-Personalized-Judge is less reliable than previously assumed, showing low and inconsistent agreement with human ground truth. The personas typically used are often overly simplistic, resulting in low predictive power. To address these issues, we introduce verbal uncertainty estimation into the LLM-as-a-Personalized-Judge pipeline, allowing the model to express low confidence on uncertain judgments. This adjustment leads to much higher agreement (above 80%) on high-certainty samples for binary tasks. Through human evaluation, we find that the LLM-as-a-Personalized-Judge achieves comparable performance to third-party humans evaluation and even surpasses human performance on high-certainty samples. Our work indicates that certainty-enhanced LLM-as-a-Personalized-Judge offers a promising direction for developing more reliable and scalable methods for evaluating LLM personalization.

翻译：随着大型语言模型（LLM）用户群在全球范围内的扩展，确保其反映多样化的用户价值观和偏好变得至关重要。因此，研究界对LLM个性化日益增长的兴趣令人鼓舞。然而，当前的研究工作通常依赖LLM-as-a-Judge（LLM作为评判者）方法进行评估，而未深入检验其有效性。本文研究了LLM-as-a-Personalized-Judge（LLM作为个性化评判者）的可靠性，即要求LLM基于用户画像（persona）来评判用户偏好。我们的研究结果表明，直接应用LLM-as-a-Personalized-Judge的可靠性低于先前假设，其与人类基准真值的一致性较低且不稳定。通常使用的用户画像往往过于简化，导致预测能力不足。为解决这些问题，我们在LLM-as-a-Personalized-Judge流程中引入了语言不确定性估计，使模型能够对不确定的判断表达低置信度。这一调整使得在二元任务的高确定性样本上达成的一致性大幅提高（超过80%）。通过人工评估，我们发现LLM-as-a-Personalized-Judge在性能上可与第三方人类评估相媲美，甚至在高确定性样本上超越了人类表现。我们的工作表明，增强确定性的LLM-as-a-Personalized-Judge为开发更可靠、可扩展的LLM个性化评估方法提供了一个有前景的方向。