As AI assistants serve millions of users daily, evaluating user experience (UX) beyond general model capability has become increasingly important. We present UXBench, the first user-centric benchmark grounded in real user feedback signals for evaluating preference alignment and dialogue generation. The benchmark consists of three interconnected tasks, UX Judge, UX Eval, and UX Recovery, with 7,400 test instances extracted from over 70K interaction logs of a mainstream Chinese AI assistant. The dataset closely reflects real user distributions, covering 8 scenarios, 83 domains, and diverse failure patterns that pose severe challenges. Extensive experiments on 26 frontier language models provide novel insights into how well models perceive user experience and how improvements in model capability contribute to better dialogue engagement. Through comprehensive analysis of model behavior and performance gaps, we show that user feedback prediction is a learnable capability, where a reward model trained from in-the-wild feedback signals can achieve well-calibrated accuracy. We further document the systematic biases of LLM-as-a-judge evaluation protocols and compare typical response strategies that directly affect user experience. UXBench establishes a new evaluation landscape and calls for greater attention to tailored UX optimization, contributing to a user-centric scaling law that shapes the success of AI assistants.

翻译：摘要：随着AI助手每天服务数百万用户，评估超越通用模型能力的用户体验变得越来越重要。我们提出了UXBench，这是首个基于真实用户反馈信号、以用户为中心的基准测试，用于评估偏好对齐与对话生成。该基准包含三项互联任务：UX判别、UX评估和UX恢复，测试实例来自主流中文AI助手超过7万条交互日志中抽取的7400个样本。数据集紧密反映真实用户分布，涵盖8类场景、83个领域及多种严重挑战的失败模式。在26个前沿语言模型上的大量实验提供了全新洞见，揭示了模型如何感知用户体验，以及模型能力提升如何促进更好的对话参与。通过对模型行为和性能差距的全面分析，我们表明用户反馈预测是一种可学习的能力，基于野外反馈信号训练的奖励模型能够实现良好校准的准确性。我们进一步记录了LLM-as-a-judge评估协议的系统性偏差，并比较了直接影响用户体验的典型响应策略。UXBench建立了新的评估框架，呼吁更多关注定制化的用户体验优化，为塑造AI助手成功的人本规模化定律做出贡献。