Data explosion and an increase in model size drive the remarkable advances in large-scale machine learning, but also make model training time-consuming and model storage difficult. To address the above issues in the distributed model training setting which has high computation efficiency and less device limitation, there are still two main difficulties. On one hand, the communication costs for exchanging information, e.g., stochastic gradients among different workers, is a key bottleneck for distributed training efficiency. On the other hand, less parameter model is easy for storage and communication, but the risk of damaging the model performance. To balance the communication costs, model capacity and model performance simultaneously, we propose quantized composite mirror descent adaptive subgradient (QCMD adagrad) and quantized regularized dual average adaptive subgradient (QRDA adagrad) for distributed training. To be specific, we explore the combination of gradient quantization and sparse model to reduce the communication cost per iteration in distributed training. A quantized gradient-based adaptive learning rate matrix is constructed to achieve a balance between communication costs, accuracy, and model sparsity. Moreover, we theoretically find that a large quantization error brings in extra noise, which influences the convergence and sparsity of the model. Therefore, a threshold quantization strategy with a relatively small error is adopted in QCMD adagrad and QRDA adagrad to improve the signal-to-noise ratio and preserve the sparsity of the model. Both theoretical analyses and empirical results demonstrate the efficacy and efficiency of the proposed algorithms.

翻译：数据爆炸和模型规模的增加促使大规模机器学习取得显著进步,但也使模型培训耗时和模型存储困难。为了解决分布式模型培训设置中的上述问题,因为计算效率高,设备限制少,因此仍然有两个主要困难。一方面,信息交流的通信成本,例如不同工人的随机梯度梯度,是分布式培训效率的关键瓶颈。另一方面,存储和通信容易采用较低的参数模型,但模型性能有受损的风险。为了平衡通信成本、模型能力和模型性能,我们建议对分布式培训的分布式模型培训进行定量化综合镜面下调效率亚梯度调整(QMD adgrad),对标准化的双均调子梯度调整(QRDA agrad)进行定量化。具体地说,我们探讨梯度倾斜度和分散式培训模式的组合,以降低分布式培训中的频率的通信成本。基于梯度的基于梯度的适应率调整率率矩阵,以平衡通信成本、准确性和模型性能和模型性能。此外,我们从理论上看,成本级缩缩缩缩缩缩缩缩度战略,在Q级模型中,将一个相对的缩缩缩缩缩缩成。