In highly competitive software markets, user experience (UX) evaluation is crucial for ensuring software quality and fostering long-term product success. Such UX evaluations typically combine quantitative metrics from standardized questionnaires with qualitative feedback collected through open-ended questions. While open-ended feedback offers valuable insights for improvement and helps explain quantitative results, analyzing large volumes of user comments is challenging and time-consuming. In this paper, we present techniques developed during a long-term UX measurement project at a major software company to efficiently process and interpret extensive volumes of user comments. To provide a high-level overview of the collected comments, we employ a supervised machine learning approach that assigns meaningful, pre-defined topic labels to each comment. Additionally, we demonstrate how generative AI (GenAI) can be leveraged to create concise and informative summaries of user feedback, facilitating effective communication of findings to the organization and especially upper management. Finally, we investigate whether the sentiment expressed in user comments can serve as an indicator for overall product satisfaction. Our results show that sentiment analysis alone does not reliably reflect user satisfaction. Instead, product satisfaction needs to be assessed explicitly in surveys to measure the user's perception of the product.

翻译：在竞争激烈的软件市场中，用户体验评估对于确保软件质量和促进产品的长期成功至关重要。此类评估通常结合标准化问卷的量化指标与开放式问题收集的质性反馈。虽然开放式反馈为改进提供了宝贵见解并有助于解释量化结果，但分析大量用户评论既具挑战性又耗时费力。本文介绍了在某大型软件公司长期用户体验测量项目中开发的技术，旨在高效处理和解读海量用户评论。为提供所收集评论的高层概览，我们采用监督式机器学习方法，为每条评论分配有意义、预定义的主题标签。此外，我们展示了如何利用生成式人工智能创建简洁且信息丰富的用户反馈摘要，从而促进向组织（尤其是高层管理层）有效传达研究发现。最后，我们探究了用户评论中表达的情感是否可作为整体产品满意度的指标。我们的结果表明，仅依赖情感分析并不能可靠地反映用户满意度；相反，必须在调查中明确评估产品满意度，以衡量用户对产品的感知。