The new regulatory framework proposal on Artificial Intelligence (AI) published by the European Commission establishes a new risk-based legal approach. The proposal highlights the need to develop adequate risk assessments for the different uses of AI. This risk assessment should address, among others, the detection and mitigation of bias in AI. In this work we analyze statistical approaches to measure biases in automatic decision-making systems. We focus our experiments in face recognition technologies. We propose a novel way to measure the biases in machine learning models using a statistical approach based on the N-Sigma method. N-Sigma is a popular statistical approach used to validate hypotheses in general science such as physics and social areas and its application to machine learning is yet unexplored. In this work we study how to apply this methodology to develop new risk assessment frameworks based on bias analysis and we discuss the main advantages and drawbacks with respect to other popular statistical tests.

翻译：欧盟委员会发布的人工智能（AI）新监管框架提案确立了一种基于风险的新型法律方法。该提案强调需要针对AI的不同用途制定充分的风险评估，其中应特别关注AI偏差的检测与缓解。本研究分析了用于衡量自动决策系统中偏差的统计方法，实验重点聚焦于人脸识别技术。我们提出了一种基于N-Sigma方法的统计框架，用于测量机器学习模型中的偏差。N-Sigma是物理学及社会科学等通用科学领域中验证假设的常用统计方法，而其在机器学习领域的应用尚未被充分探索。本文研究了如何将该方法应用于构建基于偏差分析的新型风险评估框架，并深入讨论了该方法相较于其他主流统计检验的主要优势与局限性。