Social and real-world considerations such as robustness, fairness, social welfare and multi-agent tradeoffs have given rise to multi-distribution learning paradigms, such as collaborative, group distributionally robust, and fair federated learning. In each of these settings, a learner seeks to minimize its worst-case loss over a set of $n$ predefined distributions, while using as few samples as possible. In this paper, we establish the optimal sample complexity of these learning paradigms and give algorithms that meet this sample complexity. Importantly, our sample complexity bounds exceed that of the sample complexity of learning a single distribution only by an additive factor of $n \log(n) / \epsilon^2$. These improve upon the best known sample complexity of agnostic federated learning by Mohri et al. by a multiplicative factor of $n$, the sample complexity of collaborative learning by Nguyen and Zakynthinou by a multiplicative factor $\log n / \epsilon^3$, and give the first sample complexity bounds for the group DRO objective of Sagawa et al. To achieve optimal sample complexity, our algorithms learn to sample and learn from distributions on demand. Our algorithm design and analysis is enabled by our extensions of stochastic optimization techniques for solving stochastic zero-sum games. In particular, we contribute variants of Stochastic Mirror Descent that can trade off between players' access to cheap one-off samples or more expensive reusable ones.

翻译：社会和现实世界中的考量，如鲁棒性、公平性、社会福利和多智能体权衡，催生了多分布学习范式，例如协作学习、组分布鲁棒学习和公平联邦学习。在这些设置中，学习者旨在最小化其在$n$个预定义分布上的最坏情况损失，同时使用尽可能少的样本。本文建立了这些学习范式的最优样本复杂度，并给出了达到此样本复杂度的算法。重要的是，我们的样本复杂度界限仅相比学习单个分布的样本复杂度多出一个加法项$n \log(n) / \epsilon^2$。这相比Mohri等人提出的不可知联邦学习的最佳已知样本复杂度改进了乘法因子$n$，相比Nguyen和Zakynthinou的协作学习样本复杂度改进了乘法因子$\log n / \epsilon^3$，并且首次给出了Sagawa等人提出的组DRO目标的样本复杂度界限。为了实现最优样本复杂度，我们的算法学习按需从分布中采样并学习。我们的算法设计和分析得益于我们对解决随机零和博弈的随机优化技术的扩展。特别是，我们贡献了随机镜像下降的变体，这些变体可以在玩家访问廉价的单次样本或更昂贵的可重复使用样本之间进行权衡。