As AI assistants serve millions of users daily, evaluating user experience (UX) beyond general model capability has become increasingly important. We present UXBench, the first user-centric benchmark grounded in real user feedback signals for evaluating preference alignment and dialogue generation. The benchmark consists of three interconnected tasks, UX Judge, UX Eval, and UX Recovery, with 7,400 test instances extracted from over 70K interaction logs of a mainstream Chinese AI assistant. The dataset closely reflects real user distributions, covering 8 scenarios, 83 domains, and diverse failure patterns that pose severe challenges. Extensive experiments on 26 frontier language models provide novel insights into how well models perceive user experience and how improvements in model capability contribute to better dialogue engagement. Through comprehensive analysis of model behavior and performance gaps, we show that user feedback prediction is a learnable capability, where a reward model trained from in-the-wild feedback signals can achieve well-calibrated accuracy. We further document the systematic biases of LLM-as-a-judge evaluation protocols and compare typical response strategies that directly affect user experience. UXBench establishes a new evaluation landscape and calls for greater attention to tailored UX optimization, contributing to a user-centric scaling law that shapes the success of AI assistants.

翻译：随着AI助手每日服务数百万用户，除通用模型能力外，评估用户体验（UX）的重要性日益凸显。我们提出UXBench——首个基于真实用户反馈信号的用户导向基准，用于评估偏好对齐与对话生成。该基准包含三项关联任务：UX Judge、UX Eval和UX Recovery，测试实例来自主流中文AI助手超过7万条交互日志中提取的7400个样本。数据集高度还原真实用户分布，覆盖8个场景、83个领域及严重挑战性故障模式。基于26个前沿语言模型的全面实验揭示：模型对用户体验的感知程度与能力提升如何促进更优对话互动。通过模型行为与性能差距的深度分析，我们证明用户反馈预测是一种可习得能力——基于真实环境反馈信号训练的奖励模型可实现精准校准。同时，我们系统记录了LLM-as-a-judge评估协议的固有偏差，并对比直接影响用户体验的典型响应策略。UXBench开创了新的评估范式，呼吁对定制化体验优化的更多关注，为塑造AI助手成功的用户导向规模化定律奠定基础。