Recently, the advent of large language models (LLMs) has revolutionized generative agents. Among them, Role-Playing Conversational Agents (RPCAs) attract considerable attention due to their ability to emotionally engage users. However, the absence of a comprehensive benchmark impedes progress in this field. To bridge this gap, we introduce CharacterEval, a Chinese benchmark for comprehensive RPCA assessment, complemented by a tailored high-quality dataset. The dataset comprises 1,785 multi-turn role-playing dialogues, encompassing 23,020 examples and featuring 77 characters derived from Chinese novels and scripts. It was carefully constructed, beginning with initial dialogue extraction via GPT-4, followed by rigorous human-led quality control, and enhanced with in-depth character profiles sourced from Baidu Baike. CharacterEval employs a multifaceted evaluation approach, encompassing thirteen targeted metrics on four dimensions. Comprehensive experiments on CharacterEval demonstrate that Chinese LLMs exhibit more promising capabilities than GPT-4 in Chinese role-playing conversation. Source code, data source and reward model will be publicly accessible at https://github.com/morecry/CharacterEval.

翻译：近期，大型语言模型（LLMs）的出现彻底革新了生成式代理技术。其中，角色扮演对话代理（RPCAs）凭借其情感化用户交互能力备受关注。然而，该领域因缺乏综合性基准而进展受阻。为弥补这一空白，我们提出CharacterEval——面向中文角色扮演对话代理的全面评估基准，并配套推出高质量定制数据集。该数据集包含1785组多轮角色扮演对话，涵盖23020个示例，涉及77个源自中文小说与剧本的角色。数据集构建流程严谨：首先通过GPT-4提取初始对话，经严格人工质控后，结合百度百科深度角色档案进行优化。CharacterEval采用多维评估方法，涵盖四大维度十三项针对性指标。基于该基准的全面实验表明，中文LLMs在中文角色扮演对话中展现出比GPT-4更优的性能。源代码、数据来源及奖励模型将于https://github.com/morecry/CharacterEval 公开。