Decentralized optimization is critical for solving large-scale machine learning problems over distributed networks, where multiple nodes collaborate through local communication. In practice, the variances of stochastic gradient estimators often differ across nodes, yet their impact on algorithm design and complexity remains unclear. To address this issue, we propose D-NSS, a decentralized algorithm with node-specific sampling, and establish its sample complexity depending on the arithmetic mean of local standard deviations, achieving tighter bounds than existing methods that rely on the worst-case or quadratic mean. We further derive a matching sample complexity lower bound under heterogeneous variance, thereby proving the optimality of this dependence. Moreover, we extend the framework with a variance reduction technique and develop D-NSS-VR, which under the mean-squared smoothness assumption attains an improved sample complexity bound while preserving the arithmetic-mean dependence. Finally, numerical experiments validate the theoretical results and demonstrate the effectiveness of the proposed algorithms.

翻译：去中心化优化对于解决分布式网络上的大规模机器学习问题至关重要，其中多个节点通过局部通信进行协作。在实践中，随机梯度估计器的方差通常在节点间存在差异，然而其对算法设计和复杂度的影响尚不明确。为解决此问题，我们提出了D-NSS，一种具有节点特定采样的去中心化算法，并建立了其样本复杂度依赖于局部标准差算术平均的结论，获得了比依赖最坏情况或二次平均的现有方法更紧的界。我们进一步推导了在异质方差下的匹配样本复杂度下界，从而证明了这种依赖关系的最优性。此外，我们利用方差缩减技术扩展了该框架，并开发了D-NSS-VR，其在均方光滑性假设下获得了改进的样本复杂度界，同时保持了算术平均依赖关系。最后，数值实验验证了理论结果，并证明了所提算法的有效性。