Predictive student models are increasingly used in learning environments due to their ability to enhance educational outcomes and support stakeholders in making informed decisions. However, predictive models can be biased and produce unfair outcomes, leading to potential discrimination against some students and possible harmful long-term implications. This has prompted research on fairness metrics meant to capture and quantify such biases. Nonetheless, so far, existing fairness metrics used in education are predictive performance-oriented, focusing on assessing biased outcomes across groups of students, without considering the behaviors of the models nor the severity of the biases in the outcomes. Therefore, we propose a novel metric, the Model Absolute Density Distance (MADD), to analyze models' discriminatory behaviors independently from their predictive performance. We also provide a complementary visualization-based analysis to enable fine-grained human assessment of how the models discriminate between groups of students. We evaluate our approach on the common task of predicting student success in online courses, using several common predictive classification models on an open educational dataset. We also compare our metric to the only predictive performance-oriented fairness metric developed in education, ABROCA. Results on this dataset show that: (1) fair predictive performance does not guarantee fair models' behaviors and thus fair outcomes, (2) there is no direct relationship between data bias and predictive performance bias nor discriminatory behaviors bias, and (3) trained on the same data, models exhibit different discriminatory behaviors, according to different sensitive features too. We thus recommend using the MADD on models that show satisfying predictive performance, to gain a finer-grained understanding on how they behave and to refine models selection and their usage.

翻译：预测性学生模型因其提升教育成果和支持利益相关者做出明智决策的能力，正越来越多地被应用于学习环境中。然而，预测模型可能存在偏见并产生不公平的结果，导致对某些学生的潜在歧视以及可能有害的长期影响。这促使研究者开展关于公平性指标的研究，旨在捕捉并量化此类偏见。尽管如此，截至目前，教育领域现有的公平性指标多侧重于预测性能，专注于评估学生群体间的偏见结果，而未考虑模型的行为或结果中偏见的严重程度。因此，我们提出一种新指标——模型绝对密度距离（MADD），用于独立于预测性能分析模型的歧视性行为。我们还提供了一种基于可视化的补充分析，以实现对人类评估模型如何在学生群体间进行歧视的细粒度判断。我们在一个常见的任务——预测在线课程中学生成功——上评估了该方法，使用了多个常见的预测分类模型及一个开放教育数据集。同时，我们将我们的指标与教育领域唯一开发的、以预测性能为导向的公平性指标ABROCA进行了比较。在该数据集上的结果表明：（1）公平的预测性能并不能保证模型行为的公平，因此也无法保证结果的公平；（2）数据偏见与预测性能偏见或歧视性行为偏见之间不存在直接关系；（3）在相同数据上训练的模型，会根据不同的敏感特征展现出不同的歧视性行为。因此，我们建议在表现出满意预测性能的模型上使用MADD，以更细粒度地理解其行为，并优化模型的选择和使用。