Bias studies on multilingual models confirm the presence of gender-related stereotypes in masked models processing languages with high NLP resources. We expand on this line of research by introducing Filipino CrowS-Pairs and Filipino WinoQueer: benchmarks that assess both sexist and anti-queer biases in pretrained language models (PLMs) handling texts in Filipino, a low-resource language from the Philippines. The benchmarks consist of 7,074 new challenge pairs resulting from our cultural adaptation of English bias evaluation datasets, a process that we document in detail to guide similar forthcoming efforts. We apply the Filipino benchmarks on masked and causal multilingual models, including those pretrained on Southeast Asian data, and find that they contain considerable amounts of bias. We also find that for multilingual models, the extent of bias learned for a particular language is influenced by how much pretraining data in that language a model was exposed to. Our benchmarks and insights can serve as a foundation for future work analyzing and mitigating bias in multilingual models.

翻译：针对多语言模型的偏见研究证实，在处理高自然语言处理资源语言的掩码模型中存在与性别相关的刻板印象。我们通过引入菲律宾语CrowS-Pairs和菲律宾语WinoQueer扩展了该研究方向：这两个基准用于评估处理菲律宾低资源语言文本的预训练语言模型中存在的性别歧视与反酷儿偏见。该基准包含7,074个新挑战对，源自我们对英语偏见评估数据集的文化适应性改造过程，我们详细记录了该流程以指导未来类似工作。我们将菲律宾语基准应用于掩码和因果多语言模型（包括在东南亚数据上预训练的模型），发现这些模型存在显著偏见。研究还表明，对于多语言模型，特定语言所习得偏见的程度受该语言预训练数据量的影响。我们的基准与见解可为未来分析和缓解多语言模型偏见的研究奠定基础。