The rapid emergence of generative Language Models (LMs) has led to growing concern about the impacts that their unexamined adoption may have on the social well-being of diverse user groups. Meanwhile, LMs are increasingly being adopted in K-20 schools and one-on-one student settings with minimal investigation of potential harms associated with their deployment. Motivated in part by real-world/everyday use cases (e.g., an AI writing assistant) this paper explores the potential psychosocial harms of stories generated by five leading LMs in response to open-ended prompting. We extend findings of stereotyping harms analyzing a total of 150K 100-word stories related to student classroom interactions. Examining patterns in LM-generated character demographics and representational harms (i.e., erasure, subordination, and stereotyping) we highlight particularly egregious vignettes, illustrating the ways LM-generated outputs may influence the experiences of users with marginalized and minoritized identities, and emphasizing the need for a critical understanding of the psychosocial impacts of generative AI tools when deployed and utilized in diverse social contexts.

翻译：生成式语言模型(LMs)的迅速涌现引发了人们对其未经审慎采用可能对不同用户群体社会福祉产生影响的日益担忧。与此同时，LMs正越来越多地被应用于K-20学校和一对一学生场景中，而其部署相关的潜在危害却鲜有调查。本文部分基于真实世界/日常使用场景（如AI写作助手），探究了五种主流LMs在开放式提示下生成的故事所产生的潜在社会心理危害。我们扩展了对刻板印象危害的分析，共计考察了15万篇与课堂学生互动相关的100词故事。通过分析LM生成的虚构角色人口统计学特征及表征性危害（即抹除、从属化和刻板印象），我们重点揭示了尤为恶劣的案例片段，阐明LM生成输出可能如何影响边缘化及少数群体身份用户的体验，并强调在多元化社会情境中部署和使用生成式AI工具时，必须对其社会心理影响建立批判性认知。