Predicting the success of Conversational Task Assistants (CTA) can be critical to understand user behavior and act accordingly. In this paper, we propose TB-Rater, a Transformer model which combines conversational-flow features with user behavior features for predicting user ratings in a CTA scenario. In particular, we use real human-agent conversations and ratings collected in the Alexa TaskBot challenge, a novel multimodal and multi-turn conversational context. Our results show the advantages of modeling both the conversational-flow and behavioral aspects of the conversation in a single model for offline rating prediction. Additionally, an analysis of the CTA-specific behavioral features brings insights into this setting and can be used to bootstrap future systems.

翻译：摘要：对话式任务助手（CTA）的成功预测对于理解用户行为并采取相应措施至关重要。本文提出TB-Rater模型——一种融合对话流特征与用户行为特征的Transformer模型，用于预测CTA场景下的用户评分。具体而言，我们采用Alexa TaskBot挑战赛中收集的真实人机对话数据及评分，该数据集具有新颖的多模态和多轮对话上下文特征。实验结果表明，将对话的流动特征与行为特征在单一模型中联合建模，对离线评分预测具有显著优势。此外，对CTA特有行为特征的分析为该场景提供了深刻见解，可用于构建未来系统的初始基准。