We investigate five English NLP benchmark datasets (on the superGLUE leaderboard) and two Swedish datasets for bias, along multiple axes. The datasets are the following: Boolean Question (Boolq), CommitmentBank (CB), Winograd Schema Challenge (WSC), Wino-gender diagnostic (AXg), Recognising Textual Entailment (RTE), Swedish CB, and SWEDN. Bias can be harmful and it is known to be common in data, which ML models learn from. In order to mitigate bias in data, it is crucial to be able to estimate it objectively. We use bipol, a novel multi-axes bias metric with explainability, to estimate and explain how much bias exists in these datasets. Multilingual, multi-axes bias evaluation is not very common. Hence, we also contribute a new, large Swedish bias-labelled dataset (of 2 million samples), translated from the English version and train the SotA mT5 model on it. In addition, we contribute new multi-axes lexica for bias detection in Swedish. We make the codes, model, and new dataset publicly available.

翻译：我们对五个英文NLP基准数据集（来自SuperGLUE排行榜）和两个瑞典数据集进行多轴偏差研究。数据集包括：布尔问题（BoolQ）、承诺库（CB）、维诺格拉德模式挑战赛（WSC）、Wino-性别诊断（AXg）、文本蕴含识别（RTE）、瑞典语CB及SWEDN。偏差可能造成危害，且已知在机器学习模型学习的数据中普遍存在。为减轻数据中的偏差，客观评估其程度至关重要。我们采用Bipol——一种新颖的、具备可解释性的多轴偏差度量方法——来评估并解释这些数据集中存在的偏差程度。多语言、多轴偏差评估目前尚不常见。为此，我们额外贡献了一个新的大规模瑞典语偏差标注数据集（含200万样本），该数据集由英文版本翻译而来，并基于此训练了最新SOTA mT5模型。此外，我们为瑞典语偏差检测提供了新的多轴词典。相关代码、模型及新数据集均已开源。