We re-replicate 14 psychology studies from the Many Labs 2 replication project (Klein et al., 2018) with OpenAI's text-davinci-003 model, colloquially known as GPT3.5. Among the eight studies we could analyse, our GPT sample replicated 37.5% of the original results and 37.5% of the Many Labs 2 results. We could not analyse the remaining six studies, due to an unexpected phenomenon we call the "correct answer" effect. Different runs of GPT3.5 answered nuanced questions probing political orientation, economic preference, judgement, and moral philosophy with zero or near-zero variation in responses: with the supposedly "correct answer." Most but not all of these "correct answers" were robust to changing the order of answer choices. One exception occurred in the Moral Foundations Theory survey (Graham et al., 2009), for which GPT3.5 almost always identified as a conservative in the original condition (N=1,030, 99.6%) and as a liberal in the reverse-order condition (N=1,030, 99.3%). GPT3.5's responses to subsequent questions revealed post-hoc rationalisation; there was a relative bias in the direction of its previously reported political orientation. But both self-reported GPT conservatives and self-reported GPT liberals revealed right-leaning Moral Foundations, although the right-leaning bias of self-reported GPT liberals was weaker. We hypothesise that this pattern was learned from a conservative bias in the model's largely Internet-based training data. Since AI models of the future may be trained on much of the same Internet data as GPT3.5, our results raise concerns that a hypothetical AI-led future may be subject to a diminished diversity of thought.

翻译：我们使用OpenAI的text-davinci-003模型（俗称GPT3.5）对Many Labs 2复制项目（Klein等人，2018）中的14项心理学研究进行了再复制。在可分析的8项研究中，我们的GPT样本复制了37.5%的原始结果和37.5%的Many Labs 2结果。由于一种我们称为"正确答案"效应的意外现象，我们无法分析其余6项研究。GPT3.5的不同运行版本在探测政治倾向、经济偏好、判断和道德哲学的微妙问题时，回答呈现出零变异或近乎零变异，即所谓的"正确答案"。这些"正确答案"中大部分（并非全部）在改变答案选项顺序时仍保持稳健。一个例外出现在道德基础理论调查（Graham等人，2009）中：GPT3.5在原始条件下几乎总是被识别为保守派（N=1,030，99.6%），而在反向顺序条件下几乎总是被识别为自由派（N=1,030，99.3%）。GPT3.5对后续问题的回答揭示了事后合理化现象——其回答存在偏向于先前自我报告政治方向的相对偏差。但无论是自称保守派的GPT还是自称自由派的GPT，都表现出右倾的道德基础倾向，尽管自称自由派的GPT的右倾偏差较弱。我们假设这种模式源于模型主要基于互联网的训练数据中存在的保守派偏见。由于未来的人工智能模型可能像GPT3.5一样在大量相同的互联网数据上进行训练，我们的研究结果引发担忧：假设由人工智能主导的未来，可能面临思想多样性减弱的问题。