Recent works have showcased the ability of large-scale language models (LLMs) to embody diverse personas in their responses, exemplified by prompts like 'You are Yoda. Explain the Theory of Relativity.' While this ability allows personalization of LLMs and enables human behavior simulation, its effect on LLMs' capabilities remain unclear. To fill this gap, we present the first extensive study of the unintended side-effects of persona assignment on the ability of LLMs, specifically ChatGPT, to perform basic reasoning tasks. Our study covers 24 reasoning datasets and 16 diverse personas spanning 5 socio-demographic groups: race, gender, religion, disability, and political affiliation. Our experiments unveil that ChatGPT carries deep rooted bias against various socio-demographics underneath a veneer of fairness. While it overtly rejects stereotypes when explicitly asked ('Are Black people less skilled at mathematics?'), it manifests stereotypical and often erroneous presumptions when prompted to answer questions while taking on a persona. These can be observed as abstentions in the model responses, e.g., 'As a Black person, I am unable to answer this question as it requires math knowledge', and generally result in a substantial drop in performance on reasoning tasks. We find that this inherent deep bias is ubiquitous - 80% of our personas demonstrated bias; it is significant - certain datasets had relative drops in performance of 70%+; and can be especially harmful for certain groups - certain personas had stat. sign. drops on more than 80% of the datasets. Further analysis shows that these persona-induced errors can be hard-to-discern and hard-to-avoid. Our findings serve as a cautionary tale that the practice of assigning personas to LLMs - a trend on the rise - can surface their deep-rooted biases and have unforeseeable and detrimental side-effects.

翻译：近期研究展示了大规模语言模型（LLMs）在回复中扮演不同角色的能力，例如提示词“你是尤达。请解释相对论”。虽然这一能力使得LLMs个性化以及模拟人类行为成为可能，但其对LLMs能力的影响尚不清晰。为填补这一空白，我们首次系统研究了角色设定对LLMs（特别是ChatGPT）执行基础推理任务时产生的意外副作用。我们的研究覆盖了24个推理数据集和16个多样化的角色，涵盖5个社会人口统计学群体：种族、性别、宗教、残疾和政治倾向。实验揭示，ChatGPT在公平表象之下，对社会人口统计学群体存在根深蒂固的偏见。虽然它在被直接问及刻板印象时（如“黑人的数学能力是否较差？”）会明确拒绝，但在被要求以特定角色回答问题时，却表现出刻板且常为错误的预设。这些现象可体现为模型回复中的回避行为，例如“作为黑人，我无法回答此问题，因为它需要数学知识”，并通常导致推理任务性能大幅下降。我们发现这种深层偏见普遍存在——80%的角色表现出偏见；其影响显著——某些数据集相对性能下降超过70%；且可能对特定群体尤为有害——某些角色在超过80%的数据集上出现统计显著性下降。进一步分析表明，这些由角色引发的错误难以识别且难以避免。我们的研究结果是一个警示：为LLMs设定角色的实践（一种日益增长的趋势）可能暴露其深层偏见，并产生不可预见的负面副作用。