Weak supervision overcomes the label bottleneck, enabling efficient development of training sets. Millions of models trained on such datasets have been deployed in the real world and interact with users on a daily basis. However, the techniques that make weak supervision attractive -- such as integrating any source of signal to estimate unknown labels -- also ensure that the pseudolabels it produces are highly biased. Surprisingly, given everyday use and the potential for increased bias, weak supervision has not been studied from the point of view of fairness. This work begins such a study. Our departure point is the observation that even when a fair model can be built from a dataset with access to ground-truth labels, the corresponding dataset labeled via weak supervision can be arbitrarily unfair. Fortunately, not all is lost: we propose and empirically validate a model for source unfairness in weak supervision, then introduce a simple counterfactual fairness-based technique that can mitigate these biases. Theoretically, we show that it is possible for our approach to simultaneously improve both accuracy and fairness metrics -- in contrast to standard fairness approaches that suffer from tradeoffs. Empirically, we show that our technique improves accuracy on weak supervision baselines by as much as 32% while reducing demographic parity gap by 82.5%.

翻译：弱监督克服了标签瓶颈，使得训练集的高效构建成为可能。基于此类数据集训练的模型已达数百万，它们被部署在现实世界中，每日与用户交互。然而，使得弱监督具有吸引力的技术——例如整合任意信号源来估计未知标签——也导致其生成的伪标签具有高度偏差。令人惊讶的是，鉴于其日常应用及潜在的偏差加剧风险，弱监督尚未从公平性角度得到研究。本文开启了这一研究方向。我们的出发点是观察到：即使基于真实标签的数据集能够构建出公平模型，但通过弱监督标注的对应数据集可能任意不公平。幸运的是，情况并非完全无解：我们提出并实证验证了一个弱监督中的源不公平性模型，随后引入了一种基于反事实公平的简便技术来缓解这些偏差。理论上，我们证明该方法能够同时提升准确性与公平性指标——这与面临权衡取舍的标准公平性方法形成对比。实验表明，我们的技术在弱监督基线上将准确率最多提升32%，同时将人口统计均等差距降低82.5%。