The emergence of large-scale pretrained language models has revolutionized the capabilities of new AI application, especially in the realm of crafting chatbots with distinct personas. Given the "stimulus-response" nature of chatbots, this paper unveils an innovative open-ended interview-style approach for personality assessment on role-playing chatbots, which offers a richer comprehension of their intrinsic personalities. We conduct personality assessments on 32 role-playing chatbots created by the ChatHaruhi library, across both the Big Five and MBTI dimensions, and measure their alignment with human perception. Evaluation results underscore that modern role-playing chatbots based on LLMs can effectively portray personality traits of corresponding characters, with an alignment rate of 82.8% compared with human-perceived personalities. Besides, we also suggest potential strategies for shaping chatbots' personalities. Hence, this paper serves as a cornerstone study for role-playing chatbots that intersects computational linguistics and psychology. Our resources are available at https://github.com/LC1332/Chat-Haruhi-Suzumiya

翻译：大规模预训练语言模型的出现革新了新型人工智能应用的能力，特别是在构建具有鲜明个性的聊天机器人领域。基于聊天机器人的"刺激-响应"特性，本文提出了一种创新的开放式访谈式个性评估方法，用于评估角色扮演聊天机器人的内在个性特征，从而更深入地理解其本质性格。我们利用ChatHaruhi库构建的32个角色扮演聊天机器人，分别在大五人格和MBTI维度上进行个性评估，并衡量其与人类感知的一致性。评估结果表明，基于大型语言模型的现代角色扮演聊天机器人能够有效展现对应角色的个性特质，与人类感知个性的对齐率达到82.8%。此外，我们还提出了塑造聊天机器人个性的潜在策略。因此，本文作为连接计算语言学与心理学的角色扮演聊天机器人基础性研究，具有重要意义。相关资源可访问https://github.com/LC1332/Chat-Haruhi-Suzumiya获取。