Explicit finite-sample statistical guarantees on model performance are an important ingredient in responsible machine learning. Previous work has focused mainly on bounding either the expected loss of a predictor or the probability that an individual prediction will incur a loss value in a specified range. However, for many high-stakes applications, it is crucial to understand and control the dispersion of a loss distribution, or the extent to which different members of a population experience unequal effects of algorithmic decisions. We initiate the study of distribution-free control of statistical dispersion measures with societal implications and propose a simple yet flexible framework that allows us to handle a much richer class of statistical functionals beyond previous work. Our methods are verified through experiments in toxic comment detection, medical imaging, and film recommendation.

翻译：在负责任机器学习中，对模型性能提供明确的有限样本统计保证是一项关键要素。先前的工作主要集中于界定预测器的期望损失，或单个预测的损失值落在指定范围内的概率。然而，在许多高风险应用中，理解并控制损失分布的离散度（即人群中不同成员经历算法决策影响的差异程度）至关重要。我们开创性地研究了具有社会影响的统计离散度测量的免分布控制问题，并提出一个简洁而灵活的框架，使我们能够处理比先前工作更丰富的一类统计泛函。通过毒性评论检测、医学影像和电影推荐实验验证了该方法。