Despite the many use cases for large language models (LLMs) in creating personalized chatbots, there has been limited research on evaluating the extent to which the behaviors of personalized LLMs accurately and consistently reflect specific personality traits. We consider studying the behavior of LLM-based agents which we refer to as LLM personas and present a case study with GPT-3.5 and GPT-4 to investigate whether LLMs can generate content that aligns with their assigned personality profiles. To this end, we simulate distinct LLM personas based on the Big Five personality model, have them complete the 44-item Big Five Inventory (BFI) personality test and a story writing task, and then assess their essays with automatic and human evaluations. Results show that LLM personas' self-reported BFI scores are consistent with their designated personality types, with large effect sizes observed across five traits. Additionally, LLM personas' writings have emerging representative linguistic patterns for personality traits when compared with a human writing corpus. Furthermore, human evaluation shows that humans can perceive some personality traits with an accuracy of up to 80\%. Interestingly, the accuracy drops significantly when the annotators were informed of the AI's authorship.

翻译：尽管大型语言模型（LLMs）在创建个性化聊天机器人方面具有诸多应用场景，但关于个性化LLM行为能否准确且一致地反映特定人格特质的研究仍十分有限。本研究聚焦于分析基于LLM的智能体（我们称之为LLM人格角色）的行为，并以GPT-3.5和GPT-4为案例，探究LLM能否生成与其设定人格特征相一致的内容。为此，我们基于“大五人格模型”模拟了不同的LLM人格角色，要求其完成44项大五人格量表（BFI）测试和故事写作任务，随后通过自动评估与人工评估两种方式对写作内容进行评测。结果表明：LLM人格角色的自报BFI分数与其设定人格类型高度一致，五个特质维度均呈现显著效应量；相较于人类写作语料库，LLM人格角色的写作中涌现出具有代表性的人格特质语言学模式；此外，人工评估显示人类能够以高达80%的准确率感知部分人格特质——值得注意的是，当标注者知晓文本由AI生成后，其感知准确率显著下降。