Large Language Models (LLMs) especially ChatGPT have produced impressive results in various areas, but their potential human-like psychology is still largely unexplored. Existing works study the virtual personalities of LLMs but rarely explore the possibility of analyzing human personalities via LLMs. This paper presents a generic evaluation framework for LLMs to assess human personalities based on Myers Briggs Type Indicator (MBTI) tests. Specifically, we first devise unbiased prompts by randomly permuting options in MBTI questions and adopt the average testing result to encourage more impartial answer generation. Then, we propose to replace the subject in question statements to enable flexible queries and assessments on different subjects from LLMs. Finally, we re-formulate the question instructions in a manner of correctness evaluation to facilitate LLMs to generate clearer responses. The proposed framework enables LLMs to flexibly assess personalities of different groups of people. We further propose three evaluation metrics to measure the consistency, robustness, and fairness of assessment results from state-of-the-art LLMs including ChatGPT and InstructGPT. Our experiments reveal ChatGPT's ability to assess human personalities, and the average results demonstrate that it can achieve more consistent and fairer assessments in spite of lower robustness against prompt biases compared with InstructGPT.

翻译：大型语言模型（LLMs）特别是ChatGPT已在多个领域取得了显著成果，但其潜在类人心理能力仍未得到充分探索。现有研究主要探讨LLMs的虚拟人格，但鲜有研究通过LLMs分析人类个性的可能性。本文提出一个基于迈尔斯-布里格斯类型指标（MBTI）测试的通用评估框架，用于LLMs评估人类个性。具体而言，我们首先通过随机排列MBTI问题选项设计无偏提示，并采用平均测试结果以促进更公正的答案生成。随后，我们提出替换问题陈述中的主语，使LLMs能够灵活查询和评估不同主体。最后，我们以正确性评估方式重构问题指令，以促进LLMs生成更清晰的回应。该框架使LLMs能够灵活评估不同人群的个性。我们进一步提出三个评估指标，衡量ChatGPT和InstructGPT等先进LLMs评估结果的一致性、鲁棒性和公平性。实验揭示了ChatGPT评估人类个性的能力，平均结果表明，尽管其对提示偏见的鲁棒性低于InstructGPT，但ChatGPT能实现更一致且更公平的评估。