The emergence of large-scale pretrained language models has revolutionized the capabilities of new AI application, especially in the realm of crafting chatbots with distinct personas. Given the "stimulus-response" nature of chatbots, this paper unveils an innovative open-ended interview-style approach for personality assessment on role-playing chatbots, which offers a richer comprehension of their intrinsic personalities. We conduct personality assessments on 32 role-playing chatbots created by the ChatHaruhi library, across both the Big Five and MBTI dimensions, and measure their alignment with human perception. Evaluation results underscore that modern role-playing chatbots based on LLMs can effectively portray personality traits of corresponding characters, with an alignment rate of 82.8% compared with human-perceived personalities. Besides, we also suggest potential strategies for shaping chatbots' personalities. Hence, this paper serves as a cornerstone study for role-playing chatbots that intersects computational linguistics and psychology. Our resources are available at https://github.com/LC1332/Chat-Haruhi-Suzumiya

翻译：大规模预训练语言模型的出现彻底革新了人工智能应用的能力，尤其在构建具有独特人格的聊天机器人领域。基于聊天机器人的"刺激-响应"特性，本文提出了一种创新的开放式访谈式评估方法，用于对角色扮演聊天机器人进行个性评估，从而更深入地理解其内在人格特质。我们利用ChatHaruhi库创建的32个角色扮演聊天机器人，在大五人格和MBTI两个维度上进行了个性评估，并测量了其与人类感知的一致性。评估结果表明，基于LLM的现代角色扮演聊天机器人能够有效展现相应角色的人格特质，与人类感知人格的一致率达到82.8%。此外，我们还提出了塑造聊天机器人个性的潜在策略。因此，本文作为一项跨计算语言学与心理学的基础性研究，为角色扮演聊天机器人领域提供了重要参考。我们的资源可在https://github.com/LC1332/Chat-Haruhi-Suzumiya获取。