As the size of datasets used in statistical learning continues to grow, distributed training of models has attracted increasing attention. These methods partition the data and exploit parallelism to reduce memory and runtime, but suffer increasingly from communication costs as the data size or the number of iterations grows. Recent work on linear models has shown that a surrogate likelihood can be optimized locally to iteratively improve on an initial solution in a communication-efficient manner. However, existing versions of these methods experience multiple shortcomings as the data size becomes massive, including diverging updates and efficiently handling sparsity. In this work we develop solutions to these problems which enable us to learn a communication-efficient distributed logistic regression model even beyond millions of features. In our experiments we demonstrate a large improvement in accuracy over distributed algorithms with only a few distributed update steps needed, and similar or faster runtimes. Our code is available at \url{https://github.com/FutureComputing4AI/ProxCSL}.

翻译：随着统计学习所用数据集规模的持续增长，分布式模型训练已引起越来越多的关注。这些方法通过数据划分并利用并行性来减少内存占用和运行时间，但随着数据规模或迭代次数的增加，通信开销问题日益凸显。近期在线性模型上的研究表明，可以通过局部优化代理似然函数，以通信高效的方式迭代改进初始解。然而，当数据规模变得极其庞大时，现有方法的多个缺陷逐渐显现，包括更新发散问题以及对稀疏性的高效处理难题。本研究针对这些问题提出了解决方案，使我们能够训练通信高效的分布式逻辑回归模型，其处理特征规模甚至可超越百万级别。实验结果表明，与现有分布式算法相比，我们的方法仅需少量分布式更新步骤即可实现精度的大幅提升，同时保持相当或更快的运行速度。代码已发布于 \url{https://github.com/FutureComputing4AI/ProxCSL}。