The use of Large Language Models for dialogue systems is rising, presenting a new challenge: how do we assess users' chat experience in these systems? Leveraging Natural Language Processing (NLP)-powered dialog analyzers to create dialog indicators like Coherence and Emotion has the potential to predict the chat experience. In this paper, we proposed a conceptual model to explain the relationship between the dialog indicators and various factors related to the chat experience, such as users' intentions, affinity toward dialog agents, and prompts of the agents' characters. We evaluated the conceptual model using PLS-SEM with 120 participants and found it well fit. Our results suggest that dialog indicators can predict the chat experience and fully mediate the impact of prompts and user intentions. Additionally, users' affinity toward agents can partially explain these predictions. Our findings demonstrate the potential of using dialog indicators in predicting the chat experience. Through the conceptual model we propose, researchers can apply the dialog analyzers to generate dialog indicators to constantly monitor the dialog process and improve the user's chat experience accordingly.

翻译：大型语言模型在对话系统中的使用日益增多，这带来了新的挑战：如何评估用户在这些系统中的聊天体验？利用基于自然语言处理的对话分析器生成诸如连贯性和情感等对话指标，具有预测聊天体验的潜力。本文提出了一个概念模型，用以解释对话指标与聊天体验相关多种因素（如用户意图、对对话代理的亲和力以及代理角色提示）之间的关系。我们使用偏最小二乘结构方程模型对120名参与者进行了概念模型评估，发现模型拟合良好。结果表明，对话指标能够预测聊天体验，并完全中介提示和用户意图的影响。此外，用户对代理的亲和力能够部分解释这些预测。我们的发现证明了使用对话指标预测聊天体验的潜力。通过我们提出的概念模型，研究者可以应用对话分析器生成对话指标，持续监控对话过程，并据此改善用户的聊天体验。