In many fairness and distribution robustness problems, one has access to labeled data from multiple source distributions yet the test data may come from an arbitrary member or a mixture of them. We study the problem of constructing a conformal prediction set that is uniformly valid across multiple, heterogeneous distributions, in the sense that no matter which distribution the test point is from, the coverage of the prediction set is guaranteed to exceed a pre-specified level. We first propose a max-p aggregation scheme that delivers finite-sample, multi-distribution coverage given any conformity scores associated with each distribution. Upon studying several efficiency optimization programs subject to uniform coverage, we prove the optimality and tightness of our aggregation scheme, and propose a general algorithm to learn conformity scores that lead to efficient prediction sets after the aggregation under standard conditions. We discuss how our framework relates to group-wise distributionally robust optimization, sub-population shift, fairness, and multi-source learning. In synthetic and real-data experiments, our method delivers valid worst-case coverage across multiple distributions while greatly reducing the set size compared with naively applying max-p aggregation to single-source conformity scores, and can be comparable in size to single-source prediction sets with popular, standard conformity scores.

翻译：在许多公平性和分布鲁棒性问题中，我们能够获取来自多个源分布的带标签数据，但测试数据可能来自其中任意一个分布或它们的混合。我们研究如何构建一个保形预测集，使其在多个异构分布上具有一致有效性，即无论测试点来自哪个分布，该预测集的覆盖率都保证超过预设水平。我们首先提出一种最大p值聚合方案，该方案在给定与每个分布相关的任何一致性分数时，能够提供有限样本下的多分布覆盖率保证。通过研究多个在一致覆盖率约束下的效率优化问题，我们证明了该聚合方案的最优性和紧致性，并提出一种通用算法来学习一致性分数，使得在标准条件下聚合后能得到高效的预测集。我们讨论了本框架与分组分布鲁棒优化、子群体偏移、公平性以及多源学习之间的联系。在合成数据与真实数据实验中，我们的方法在多个分布上实现了有效的、最坏情况下的覆盖率保证，同时相较于将最大p值聚合方案简单应用于单源一致性分数的方法，显著减小了预测集规模，并且其规模可与使用流行标准一致性分数的单源预测集相媲美。