Distributed stochastic optimization methods based on Newton's method offer significant advantages over first-order methods by leveraging curvature information for improved performance. However, the practical applicability of Newton's method is hindered in large-scale and heterogeneous learning environments due to challenges such as high computation and communication costs associated with the Hessian matrix, sub-model diversity, staleness in training, and data heterogeneity. To address these challenges, this paper introduces a novel and efficient algorithm called RANL, which overcomes the limitations of Newton's method by employing a simple Hessian initialization and adaptive assignments of training regions. The algorithm demonstrates impressive convergence properties, which are rigorously analyzed under standard assumptions in stochastic optimization. The theoretical analysis establishes that RANL achieves a linear convergence rate while effectively adapting to available resources and maintaining high efficiency. Unlike traditional first-order methods, RANL exhibits remarkable independence from the condition number of the problem and eliminates the need for complex parameter tuning. These advantages make RANL a promising approach for distributed stochastic optimization in practical scenarios.

翻译：基于牛顿法的分布式随机优化方法通过利用曲率信息提升性能，相比一阶方法具有显著优势。然而，在大规模和异构学习环境中，海森矩阵的高计算与通信开销、子模型多样性、训练滞后性及数据异构性等问题严重制约了牛顿法的实际应用。针对这些挑战，本文提出一种新型高效算法RANL，通过采用简单的海森矩阵初始化与训练区域自适应分配策略，突破了牛顿法的局限性。该算法展现出卓越的收敛特性，并在随机优化的标准假设下进行了严格分析。理论分析表明，RANL在实现线性收敛速率的同时，能够有效适配可用资源并保持高效性。与传统一阶方法不同，RANL显著降低了对问题条件数的依赖性，且无需复杂的参数调优。这些优势使得RANL成为面向实际场景中分布式随机优化的极具前景的解决方案。