In this paper, we proposed a conceptual model to predict the chat experience in a natural language generation dialog system. We evaluated the model with 120 participants with Partial Least Squares Structural Equation Modeling (PLS-SEM) and obtained an R-square (R2) with 0.541. The model considers various factors, including the prompts used for generation; coherence, sentiment, and similarity in the conversation; and users' perceived dialog agents' favorability. We then further explore the effectiveness of the subset of our proposed model. The results showed that users' favorability and coherence, sentiment, and similarity in the dialogue are positive predictors of users' chat experience. Moreover, we found users may prefer dialog agents with characteristics of Extroversion, Openness, Conscientiousness, Agreeableness, and Non-Neuroticism. Through our research, an adaptive dialog system might use collected data to infer factors in our model, predict the chat experience for users through these factors, and optimize it by adjusting prompts.

翻译：本文提出了一种概念模型，用于预测自然语言生成对话系统中的聊天体验。我们通过120名参与者使用偏最小二乘结构方程模型（PLS-SEM）评估该模型，得到了0.541的R平方值（R²）。该模型考虑了多种因素，包括用于生成的提示词、对话中的连贯性、情感和相似性，以及用户感知的对话智能体好感度。随后，我们进一步探索了所提模型子集的有效性。结果表明，用户好感度以及对话中的连贯性、情感和相似性是用户聊天体验的正向预测因子。此外，我们发现用户可能偏好具有外向性、开放性、尽责性、宜人性和非神经质特征的对话智能体。通过本研究，自适应对话系统可利用收集的数据推断模型中的因素，通过这些因素预测用户的聊天体验，并通过调整提示词来优化体验。