Despite the many use cases for large language models (LLMs) in creating personalized chatbots, there has been limited research on evaluating the extent to which the behaviors of personalized LLMs accurately and consistently reflect specific personality traits. We consider studying the behavior of LLM-based agents which we refer to as LLM personas and present a case study with GPT-3.5 and GPT-4 to investigate whether LLMs can generate content that aligns with their assigned personality profiles. To this end, we simulate distinct LLM personas based on the Big Five personality model, have them complete the 44-item Big Five Inventory (BFI) personality test and a story writing task, and then assess their essays with automatic and human evaluations. Results show that LLM personas' self-reported BFI scores are consistent with their designated personality types, with large effect sizes observed across five traits. Additionally, LLM personas' writings have emerging representative linguistic patterns for personality traits when compared with a human writing corpus. Furthermore, human evaluation shows that humans can perceive some personality traits with an accuracy of up to 80%. Interestingly, the accuracy drops significantly when the annotators were informed of AI authorship.

翻译：尽管大语言模型（LLMs）在创建个性化聊天机器人方面具有众多应用场景，但针对个性化LLM的行为能否准确且一致地反映特定人格特质这一问题的评估研究仍十分有限。本文研究基于LLM的智能体（我们称之为"LLM人格化身"）的行为，并以GPT-3.5和GPT-4为案例，探究LLM能否生成与其设定人格特征相符的内容。为此，我们基于大五人格模型模拟了不同的LLM人格化身，令其完成包含44项的大五人格量表（BFI）测试和故事写作任务，随后通过自动评估与人工评估对生成文本进行分析。结果表明，LLM人格化身的自我报告BFI得分与其设定人格类型具有一致性，且在五项人格特质上均呈现出较大的效应量。此外，与人类写作语料库相比，LLM人格化身的写作内容中涌现出代表特定人格特质的典型语言模式。人工评估显示，人类能够以高达80%的准确率识别出部分人格特质。有趣的是，当标注者被告知内容由AI生成后，识别准确率显著下降。