In many predictive contexts (e.g., credit lending), true outcomes are only observed for samples that were positively classified in the past. These past observations, in turn, form training datasets for classifiers that make future predictions. However, such training datasets lack information about the outcomes of samples that were (incorrectly) negatively classified in the past and can lead to erroneous classifiers. We present an approach that trains a classifier using available data and comes with a family of exploration strategies to collect outcome data about subpopulations that otherwise would have been ignored. For any exploration strategy, the approach comes with guarantees that (1) all sub-populations are explored, (2) the fraction of false positives is bounded, and (3) the trained classifier converges to a "desired" classifier. The right exploration strategy is context-dependent; it can be chosen to improve learning guarantees and encode context-specific group fairness properties. Evaluation on real-world datasets shows that this approach consistently boosts the quality of collected outcome data and improves the fraction of true positives for all groups, with only a small reduction in predictive utility.

翻译：在许多预测场景（如信贷发放）中，真实结果仅能通过过去被正面分类的样本观测得到。这些历史观测数据进而构成训练数据集，用于训练对未来进行预测的分类器。然而，这类训练数据集缺乏关于过去被（错误）负面分类的样本结果信息，可能导致分类器产生错误。我们提出一种方法，利用现有数据训练分类器，并配备一系列探索策略来收集原本会被忽略的子群体结果数据。对于任何探索策略，该方法都能保证：（1）所有子群体均被探索；（2）假阳性比例有界；（3）训练后的分类器收敛至"期望"分类器。最优探索策略取决于具体情境，可通过改进学习保证并编码特定场景的群体公平性属性进行选择。在真实数据集上的评估表明，该方法能持续提升收集结果数据的质量，并在仅轻微降低预测效用的前提下，提高所有群体的真阳性比例。