Modern datasets span billions of samples, making training on all available data infeasible. Selecting a high quality subset helps in reducing training costs and enhancing model quality. Submodularity, a discrete analogue of convexity, is commonly used for solving such subset selection problems. However, existing algorithms for optimizing submodular functions are sequential, and the prior distributed methods require at least one central machine to fit the target subset in DRAM. At billion datapoint scale, even the subset may not fit a single machine, and the sequential algorithms are prohibitively slow. In this paper, we relax the requirement of having a central machine for the target subset by proposing a novel distributed bounding algorithm with provable approximation guarantees. The algorithm iteratively bounds the minimum and maximum utility values to select high quality points and discard the unimportant ones. When bounding does not find the complete subset, we use a multi-round, partition-based distributed greedy algorithm to identify the remaining subset. We discuss how to implement these algorithms in a distributed data processing framework and empirically analyze different configurations. We find high quality subsets on CIFAR-100 and ImageNet with marginal or no loss in quality compared to centralized methods, and scale to a dataset with 13 billion points.

翻译：现代数据集包含数十亿样本，使得在所有可用数据上进行训练变得不可行。选择高质量子集有助于降低训练成本并提升模型质量。次模性作为凸性的离散类比，通常用于解决此类子集选择问题。然而，现有优化次模函数的算法均为串行方法，且先前的分布式方法要求至少一台中心机器能将目标子集存入DRAM。在十亿级数据点规模下，即使子集也可能无法单机容纳，而串行算法的速度则慢得难以接受。本文通过提出一种具有可证明近似保证的新型分布式定界算法，放宽了对目标子集需由中心机器存储的要求。该算法通过迭代界定最小与最大效用值来选取高质量数据点并剔除不重要点。当定界法无法找到完整子集时，我们采用基于多轮分区的分布式贪心算法识别剩余子集。我们探讨了如何在分布式数据处理框架中实现这些算法，并对不同配置进行了实证分析。在CIFAR-100和ImageNet数据集上，我们获得了与集中式方法相比质量损失微乎其微甚至无损的高质量子集，并将算法扩展至包含130亿数据点的数据集。