When analyzing the behavior of machine learning algorithms, it is important to identify specific data subgroups for which the considered algorithm shows different performance with respect to the entire dataset. The intervention of domain experts is normally required to identify relevant attributes that define these subgroups. We introduce the notion of divergence to measure this performance difference and we exploit it in the context of (i) classification models and (ii) ranking applications to automatically detect data subgroups showing a significant deviation in their behavior. Furthermore, we quantify the contribution of all attributes in the data subgroup to the divergent behavior by means of Shapley values, thus allowing the identification of the most impacting attributes.

翻译：在分析机器学习算法的行为时,重要的是要确定特定的数据分组,即所考虑的算法显示整个数据集不同性能的具体数据分组。通常需要域专家的干预来确定界定这些分组的相关属性。我们引入差异概念来测量这种性能差异,并在(一) 分类模型和(二) 排序应用中加以利用,以自动检测显示其行为有重大偏差的数据分组。此外,我们用Shapley 值量化数据分组中所有属性对不同行为的贡献,从而能够识别影响最大的属性。