User Satisfaction Modeling (USM) is one of the popular choices for task-oriented dialogue systems evaluation, where user satisfaction typically depends on whether the user's task goals were fulfilled by the system. Task-oriented dialogue systems use task schema, which is a set of task attributes, to encode the user's task goals. Existing studies on USM neglect explicitly modeling the user's task goals fulfillment using the task schema. In this paper, we propose SG-USM, a novel schema-guided user satisfaction modeling framework. It explicitly models the degree to which the user's preferences regarding the task attributes are fulfilled by the system for predicting the user's satisfaction level. SG-USM employs a pre-trained language model for encoding dialogue context and task attributes. Further, it employs a fulfillment representation layer for learning how many task attributes have been fulfilled in the dialogue, an importance predictor component for calculating the importance of task attributes. Finally, it predicts the user satisfaction based on task attribute fulfillment and task attribute importance. Experimental results on benchmark datasets (i.e. MWOZ, SGD, ReDial, and JDDC) show that SG-USM consistently outperforms competitive existing methods. Our extensive analysis demonstrates that SG-USM can improve the interpretability of user satisfaction modeling, has good scalability as it can effectively deal with unseen tasks and can also effectively work in low-resource settings by leveraging unlabeled data.

翻译：用户满意度建模（USM）是任务型对话系统评估中的常用方法之一，用户满意度通常取决于系统是否实现了用户的任务目标。任务型对话系统通过任务框架（即一组任务属性编码用户的任务目标。现有USM研究忽视了利用任务框架显式建模用户任务目标的实现程度。本文提出SG-USM，一种新颖的框架引导的用户满意度建模框架。该框架显式建模系统对用户任务属性偏好的实现程度，以预测用户满意度水平。SG-USM采用预训练语言模型编码对话上下文和任务属性，并通过实现表征层学习对话中已实现的任务属性数量，同时利用重要性预测组件计算各任务属性的重要性。最终，模型基于任务属性实现程度与重要性预测用户满意度。在基准数据集（MWOZ、SGD、ReDial和JDDC）上的实验结果表明，SG-USM始终优于其他竞争方法。广泛分析显示，SG-USM能提升用户满意度建模的可解释性，具有良好的可扩展性（可有效处理未知任务），并能在低资源场景下通过利用未标注数据实现高效运行。