Multilingual large language models have been increasingly popular for their proficiency in comprehending and generating text across various languages. Previous research has shown that the presence of stereotypes and biases in monolingual large language models can be attributed to the nature of their training data, which is collected from humans and reflects societal biases. Multilingual language models undergo the same training procedure as monolingual ones, albeit with training data sourced from various languages. This raises the question: do stereotypes present in one social context leak across languages within the model? In our work, we first define the term ``stereotype leakage'' and propose a framework for its measurement. With this framework, we investigate how stereotypical associations leak across four languages: English, Russian, Chinese, and Hindi. To quantify the stereotype leakage, we employ an approach from social psychology, measuring stereotypes via group-trait associations. We evaluate human stereotypes and stereotypical associations manifested in multilingual large language models such as mBERT, mT5, and ChatGPT. Our findings show a noticeable leakage of positive, negative, and non-polar associations across all languages. Notably, Hindi within multilingual models appears to be the most susceptible to influence from other languages, while Chinese is the least. Additionally, ChatGPT exhibits a better alignment with human scores than other models.

翻译：多语言大语言模型因其精通多种语言的文本理解与生成能力而日益流行。先前研究表明，单语大语言模型中存在的刻板印象和偏见可归因于其训练数据的性质——这些数据来自人类且反映了社会偏见。多语言语言模型遵循与单语模型相同的训练流程，但训练数据源自多种语言。这引发了一个问题：某一社会情境中的刻板印象是否会跨语言在模型内部泄露？在本工作中，我们首先定义术语“刻板印象泄露”，并提出衡量该现象的框架。基于该框架，我们探究刻板印象关联在英语、俄语、汉语和印地语四种语言间的跨语言泄露方式。为量化刻板印象泄露，我们采用社会心理学方法，通过群体-特质关联来评估刻板印象。我们评估了人类刻板印象以及多语言大语言模型（如mBERT、mT5和ChatGPT）中显现的刻板印象关联。研究结果表明，所有语言中均存在积极、消极及非极性关联的显著泄露。值得注意的是，多语言模型中印地语最易受其他语言影响，而汉语受影响程度最小。此外，相较于其他模型，ChatGPT与人类评分的一致性更高。