Large language models (LLMs) are increasing in capability and popularity, propelling their application in new domains -- including as replacements for human participants in computational social science, user testing, annotation tasks, and more. In many settings, researchers seek to distribute their surveys to a sample of participants that are representative of the underlying human population of interest. This means in order to be a suitable replacement, LLMs will need to be able to capture the influence of positionality (i.e., relevance of social identities like gender and race). However, we show that there are two inherent limitations in the way current LLMs are trained that prevent this. We argue analytically for why LLMs are likely to both misportray and flatten the representations of demographic groups, then empirically show this on 4 LLMs through a series of human studies with 3200 participants across 16 demographic identities. We also discuss a third limitation about how identity prompts can essentialize identities. Throughout, we connect each limitation to a pernicious history that explains why it is harmful for marginalized demographic groups. Overall, we urge caution in use cases where LLMs are intended to replace human participants whose identities are relevant to the task at hand. At the same time, in cases where the goal is to supplement rather than replace (e.g., pilot studies), we provide inference-time techniques that we empirically demonstrate do reduce, but do not remove, these harms.

翻译：大型语言模型（LLMs）的能力和普及度日益提升，正推动其在新领域的应用——包括作为计算社会科学、用户测试、标注任务等领域中人类参与者的替代品。在许多研究场景中，学者期望将调查分发给能代表目标人群的参与者样本。这意味着要成为合适的替代品，LLMs必须能够捕捉立场性（即性别、种族等社会身份的相关性）的影响。然而，我们指出当前LLMs的训练方式存在两个固有局限，使其无法实现这一目标。我们通过理论分析论证了LLMs为何可能同时曲解并扁平化人口群体的表征，随后通过包含16种人口身份的3200名参与者参与的系列人类研究，在4个LLMs上进行了实证验证。我们还探讨了第三个局限：身份提示可能如何本质化身份认同。全文将每个局限性与一段有害的历史背景相联系，阐释这些局限为何会对边缘化人口群体造成伤害。总体而言，我们强烈建议在LLMs旨在替代任务相关身份的人类参与者的应用场景中保持谨慎。同时，在旨在补充而非替代的场景中（例如预研实验），我们提出了推理阶段的技术方法，并通过实证证明这些方法能够减轻——但无法完全消除——上述危害。