User-centric evaluation has become a key paradigm for assessing Conversational Recommender Systems (CRS), aiming to capture subjective qualities such as satisfaction, trust, and rapport. To enable scalable evaluation, recent work increasingly relies on third-party annotations of static dialogue logs by crowd workers or large language models. However, the reliability of this practice remains largely unexamined. In this paper, we present a large-scale empirical study investigating the reliability and structure of user-centric CRS evaluation on static dialogue transcripts. We collected 1,053 annotations from 124 crowd workers on 200 ReDial dialogues using the 18-dimensional CRS-Que framework. Using random-effects reliability models and correlation analysis, we quantify the stability of individual dimensions and their interdependencies. Our results show that utilitarian and outcome-oriented dimensions such as accuracy, usefulness, and satisfaction achieve moderate reliability under aggregation, whereas socially grounded constructs such as humanness and rapport are substantially less reliable. Furthermore, many dimensions collapse into a single global quality signal, revealing a strong halo effect in third-party judgments. These findings challenge the validity of single-annotator and LLM-based evaluation protocols and motivate the need for multi-rater aggregation and dimension reduction in offline CRS evaluation.

翻译：用户中心评估已成为评估对话式推荐系统（CRS）的关键范式，旨在捕捉满意度、信任度和亲和关系等主观质量维度。为实现可扩展的评估，近期研究日益依赖众包工作者或大语言模型对静态对话记录进行第三方标注。然而，这种实践方法的可靠性尚未得到充分检验。本文通过大规模实证研究，探讨了基于静态对话文本的用户中心CRS评估的可靠性及其结构维度。我们采用18维度的CRS-Que框架，收集了124名众包工作者对200个ReDial对话生成的1,053条标注。通过随机效应可靠性模型与相关性分析，我们量化了各维度的稳定性及其相互依存关系。研究结果表明：功利性与结果导向的维度（如准确性、实用性和满意度）在聚合后具有中等可靠性，而基于社会属性的构念（如拟人性和亲和关系）的可靠性显著较低。此外，多数维度会坍缩为单一全局质量信号，揭示了第三方评估中强烈的光环效应。这些发现对单标注者与基于大语言模型的评估协议的有效性提出了质疑，并表明离线CRS评估需要采用多评估者聚合与降维方法。