As conversational models become increasingly available to the general public, users are engaging with this technology in social interactions. Such unprecedented interaction experiences may pose considerable social and psychological risks to the users unless the technology is properly controlled. This highlights the need for scalable and robust evaluation metrics for conversational chatbots. Existing evaluation metrics aim to automate offline user evaluation and approximate human judgment of pre-curated dialogs. However, they are limited in their ability to capture subjective perceptions of users who actually interact with the bots and might not generalize to real-world settings. To address this limitation, we propose an approach to approximate online human evaluation leveraging large language models (LLMs) from the GPT family. We introduce a new Dialog system Evaluation framework based on Prompting (DEP), which enables a fully automatic evaluation pipeline that replicates live user studies and achieves an impressive correlation with human judgment (up to Pearson r=0.95 on a system level). The DEP approach involves collecting synthetic chat logs of evaluated bots with an LLM in the other-play setting, where the LLM is carefully conditioned to follow a specific scenario. We further explore different prompting approaches to produce evaluation scores with the same LLM. The best performing prompts, which contain few-shot demonstrations and instructions, show outstanding performance on the tested dataset and demonstrate the ability to generalize to other dialog corpora.

翻译：随着会话模型日益向公众开放，用户正在社交互动中接触这项技术。若缺乏适当管控，这种前所未有的互动体验可能给用户带来严重的社会和心理风险。这凸显了对可扩展且稳健的聊天机器人评估指标的需求。现有评估指标旨在实现离线用户评估自动化，并近似专家对预策划对话的人工判断，但其捕捉与实际聊天机器人交互的用户主观感知的能力有限，且可能无法泛化至真实场景。为突破这一局限，我们提出一种利用GPT系列大语言模型（LLMs）近似在线人工评估的方法。我们引入了基于提示的对话系统评估框架（DEP），该框架能构建完全自动化的评估管线，复刻实时用户研究，并与人工判断达到显著相关性（系统层级皮尔逊相关系数高达r=0.95）。DEP方法涉及在异质玩家设定下，使被评估聊天机器人与经过精细场景条件约束的LLM生成合成聊天日志。我们进一步探索了多种提示方法，以利用相同LLM生成评估分数。包含少量示例演示与指令的最优提示，在测试数据集上展现出卓越性能，并验证了其向其他对话语料库泛化的能力。