Personalization of LLMs by sociodemographic subgroup often improves user experience, but can also introduce or amplify biases and unfair outcomes across groups. Prior work has employed so-called personas, sociodemographic user attributes conveyed to a model, to study bias in LLMs by relying on a single cue to prompt a persona, such as user names or explicit attribute mentions. This disregards LLM sensitivity to prompt variation and the rarity of some cues in real interactions (external validity). We compare six commonly used persona cues across seven open and proprietary LLMs on four writing and advice tasks. While cues are overall highly correlated, they produce substantial variance in responses across personas that can change findings on persona-induced differences and bias. We therefore caution against claims based on single persona cues, especially when they are overly explicit and have low external validity.

翻译：通过社会人口学子群体对大语言模型进行个性化处理通常能改善用户体验，但也可能引入或加剧群体间的偏见与不公。现有研究使用所谓“身份特征”——即向模型传递的社会人口学用户属性，通过单一线索（如用户姓名或显式属性提及）来提示身份特征以研究大语言模型中的偏见。这种做法忽视了LLM对提示变化的敏感性以及在真实交互中某些线索的罕见性（外部效度）。我们针对七种开源和专有LLM，在四项写作与建议任务中比较了六种常用身份特征线索。尽管各线索整体呈现高度相关性，但不同身份特征下的响应存在显著差异，这可能改变关于身份特征引发的差异与偏见的研究结论。因此，我们告诫避免基于单一身份特征线索得出结论，尤其是在线索过于显式且外部效度较低的情况下。