In many predictive contexts (e.g., credit lending), true outcomes are only observed for samples that were positively classified in the past. These past observations, in turn, form training datasets for classifiers that make future predictions. However, such training datasets lack information about the outcomes of samples that were (incorrectly) negatively classified in the past and can lead to erroneous classifiers. We present an approach that trains a classifier using available data and comes with a family of exploration strategies to collect outcome data about subpopulations that otherwise would have been ignored. For any exploration strategy, the approach comes with guarantees that (1) all sub-populations are explored, (2) the fraction of false positives is bounded, and (3) the trained classifier converges to a ``desired'' classifier. The right exploration strategy is context-dependent; it can be chosen to improve learning guarantees and encode context-specific group fairness properties. Evaluation on real-world datasets shows that this approach consistently boosts the quality of collected outcome data and improves the fraction of true positives for all groups, with only a small reduction in predictive utility.

翻译：在许多预测场景（如信贷发放）中，真实结果仅能通过过去被正向分类的样本观测得到。这些历史观测数据进而构成用于未来预测的分类器训练集。然而，此类训练集缺乏关于过去被（错误）负向分类样本的结果信息，可能导致分类器产生系统性偏差。本文提出一种方法：首先利用现有数据训练分类器，继而通过一系列探索策略收集那些原本可能被忽略的亚群结果数据。对于任意探索策略，该方法均能保证：（1）所有亚群均被充分探索；（2）假阳性比例受严格约束；（3）训练所得分类器收敛于“理想”分类器。最优探索策略需结合具体场景选择：可通过调整策略以提升学习保证效果，并编码特定场景的群体公平性属性。在真实数据集上的评估表明，该方法能持续提升所收集结果数据的质量，改善所有群体的真阳性比例，同时仅造成预测效用的微小下降。