There has been great recent advancement in human-computer chat. However, proper evaluation currently requires human judgements that produce notoriously high-variance metrics due to their inherent subjectivity. Furthermore, there is little standardization in the methods and labels used for evaluation, with an overall lack of work to compare and assess the validity of various evaluation approaches. As a consequence, existing evaluation results likely leave an incomplete picture of the strengths and weaknesses of open-domain chatbots. We aim towards a dimensional evaluation of human-computer chat that can reliably measure several distinct aspects of chat quality. To this end, we present our novel human evaluation method that quantifies the rate of several quality-related chatbot behaviors. Our results demonstrate our method to be more suitable for dimensional chat evaluation than alternative likert-style or comparative methods. We then use our validated method and existing methods to evaluate four open-domain chat models from the recent literature.

翻译：近年来人机聊天技术取得了巨大进步。然而，由于评估时需依赖人工判断，其固有的主观性导致指标方差极高。此外，评估方法和标签缺乏标准化，且鲜有研究对不同评估方法的有效性进行比较与验证。因此，现有评估结果可能无法全面揭示开放域聊天机器人的优缺点。我们旨在建立一种能够可靠衡量聊天质量多个独立维度的人机对话维度评估方法。为此，我们提出了一种新颖的人工评估方法，用于量化多种与质量相关的聊天机器人行为的发生率。结果表明，相较于传统的李克特式或比较式方法，本方法更适用于维度聊天评估。随后，我们基于验证后的方法及现有方法，对近期文献中四种开放域聊天模型进行了评估。