As large language models (LLM) evolve in their capabilities, various recent studies have tried to quantify their behavior using psychological tools created to study human behavior. One such example is the measurement of "personality" of LLMs using self-assessment personality tests developed to measure human personality. Yet almost none of these works verify the applicability of these tests on LLMs. In this paper, we analyze the reliability of LLM personality scores obtained from self-assessment personality tests using two simple experiments. We first introduce the property of prompt sensitivity, where three semantically equivalent prompts representing three intuitive ways of administering self-assessment tests on LLMs are used to measure the personality of the same LLM. We find that all three prompts lead to very different personality scores, a difference that is statistically significant for all traits in a large majority of scenarios. We then introduce the property of option-order symmetry for personality measurement of LLMs. Since most of the self-assessment tests exist in the form of multiple choice question (MCQ) questions, we argue that the scores should also be robust to not just the prompt template but also the order in which the options are presented. This test unsurprisingly reveals that the self-assessment test scores are not robust to the order of the options. These simple tests, done on ChatGPT and three Llama2 models of different sizes, show that self-assessment personality tests created for humans are unreliable measures of personality in LLMs.

翻译：随着大语言模型（LLM）能力的发展，近期多项研究尝试利用为研究人类行为而创建的心理工具来量化其行为。其中一个典型案例是使用为测量人类人格而开发的自我评估人格测试来评估LLM的"人格"。然而，这些研究几乎都没有验证此类测试对LLM的适用性。本文通过两个简单实验，分析了从自我评估人格测试中获得的LLM人格得分的可靠性。我们首先引入提示敏感性这一特性：通过三个语义等价但代表向LLM实施自我评估测试的三种直观方式的提示，来测量同一个LLM的人格。结果发现，这三个提示产生了截然不同的人格得分，在绝大多数场景中所有特质的差异均具有统计学显著性。随后，我们针对LLM人格测量引入选项顺序对称性特性。由于大多数自我评估测试以多选题（MCQ）形式呈现，我们认为这类测试的得分不仅应对提示模板具有鲁棒性，还应应对选项呈现顺序具有鲁棒性。这一测试不出所料地表明，自我评估测试得分对选项顺序并不鲁棒。这些在ChatGPT及三个不同规模的Llama2模型上开展的简单实验表明，为人类设计的自我评估人格测试无法可靠衡量LLM的人格特质。