While multilingual language models can improve NLP performance on low-resource languages by leveraging higher-resource languages, they also reduce average performance on all languages (the 'curse of multilinguality'). Here we show another problem with multilingual models: grammatical structures in higher-resource languages bleed into lower-resource languages, a phenomenon we call grammatical structure bias. We show this bias via a novel method for comparing the fluency of multilingual models to the fluency of monolingual Spanish and Greek models: testing their preference for two carefully-chosen variable grammatical structures (optional pronoun-drop in Spanish and optional Subject-Verb ordering in Greek). We find that multilingual BERT is biased toward the English-like setting (explicit pronouns and Subject-Verb-Object ordering) as compared to our monolingual control language model. With our case studies, we hope to bring to light the fine-grained ways in which multilingual models can be biased,and encourage more linguistically-aware fluency evaluation.

翻译：虽然多语言语言模型可以通过利用高资源语言来提升低资源语言的自然语言处理性能，但它们也会降低所有语言的平均性能（“多语言诅咒”）。本文揭示了多语言模型的另一个问题：高资源语言的语法结构会渗透到低资源语言中，我们将这种现象称为语法结构偏差。我们通过一种新颖的方法证明了这种偏差——比较多语言模型与单语西班牙语和希腊语模型的流利程度：测试它们对两种精心选择的可变语法结构（西班牙语中的可选代词省略和希腊语中的可选主-动语序）的偏好。我们发现，与我们的单语对照语言模型相比，多语言BERT更倾向于类似英语的设置（显式代词和主-动-宾语序）。通过我们的案例研究，我们希望揭示多语言模型可能以何种精细方式存在偏差，并鼓励更具语言学意识的流利度评估。