Recent work has aimed to capture nuances of human behavior by using LLMs to simulate responses from particular demographics in settings like social science experiments and public opinion surveys. However, there are currently no established ways to discuss or evaluate the quality of such LLM simulations. Moreover, there is growing concern that these LLM simulations are flattened caricatures of the personas that they aim to simulate, failing to capture the multidimensionality of people and perpetuating stereotypes. To bridge these gaps, we present CoMPosT, a framework to characterize LLM simulations using four dimensions: Context, Model, Persona, and Topic. We use this framework to measure open-ended LLM simulations' susceptibility to caricature, defined via two criteria: individuation and exaggeration. We evaluate the level of caricature in scenarios from existing work on LLM simulations. We find that for GPT-4, simulations of certain demographics (political and marginalized groups) and topics (general, uncontroversial) are highly susceptible to caricature.

翻译：近期研究旨在通过利用大语言模型模拟社会科学实验和公众舆论调查等场景中特定人群的响应，从而捕捉人类行为的细微差异。然而，目前尚未建立讨论或评估此类大语言模型模拟质量的标准化方法。此外，日益增长的担忧表明，这些模拟可能沦为所模拟角色扁平化的漫画式刻画，既未能体现人类的多维性，又加剧了刻板印象。为弥合这些空白，我们提出CoMPosT框架，通过四个维度（上下文、模型、角色、主题）对大语言模型模拟进行表征。我们运用该框架，依据个体化与夸张化两项标准定义漫画化倾向，以衡量开放式大语言模型模拟的脆弱性。我们评估了现有大语言模型模拟研究中场景的漫画化程度，发现对于GPT-4模型，特定人群（政治群体与边缘群体）及话题（通用型、非争议性话题）的模拟具有高度漫画化倾向。