Sentiment analysis (SA) systems are used in many products and hundreds of languages. Gender and racial biases are well-studied in English SA systems, but understudied in other languages, with few resources for such studies. To remedy this, we build a counterfactual evaluation corpus for gender and racial/migrant bias in four languages. We demonstrate its usefulness by answering a simple but important question that an engineer might need to answer when deploying a system: What biases do systems import from pre-trained models when compared to a baseline with no pre-training? Our evaluation corpus, by virtue of being counterfactual, not only reveals which models have less bias, but also pinpoints changes in model bias behaviour, which enables more targeted mitigation strategies. We release our code and evaluation corpora to facilitate future research.

翻译：情感分析系统被广泛应用于众多产品及数百种语言中。性别和种族偏见在英语情感分析系统中已得到充分研究，但在其他语言中却研究不足，且缺乏相关研究资源。为解决这一问题，我们构建了一个针对四种语言的性别与种族/移民偏见的反事实评估语料库。通过回答工程师在部署系统时可能需要回答的一个简单但重要的问题，我们展示了该语料库的实用性：与未经过预训练的基线相比，系统从预训练模型中引入了哪些偏见？我们的评估语料库因其反事实特性，不仅能揭示哪些模型的偏见较少，还能精准定位模型偏见行为的变化，从而支持更有针对性的缓解策略。我们公开发布代码和评估语料库以促进未来研究。