Dual Coordinate Descent (DCD) and Block Dual Coordinate Descent (BDCD) are important iterative methods for solving convex optimization problems. In this work, we develop scalable DCD and BDCD methods for the kernel support vector machines (K-SVM) and kernel ridge regression (K-RR) problems. On distributed-memory parallel machines the scalability of these methods is limited by the need to communicate every iteration. On modern hardware where communication is orders of magnitude more expensive, the running time of the DCD and BDCD methods is dominated by communication cost. We address this communication bottleneck by deriving $s$-step variants of DCD and BDCD for solving the K-SVM and K-RR problems, respectively. The $s$-step variants reduce the frequency of communication by a tunable factor of $s$ at the expense of additional bandwidth and computation. The $s$-step variants compute the same solution as the existing methods in exact arithmetic. We perform numerical experiments to illustrate that the $s$-step variants are also numerically stable in finite-arithmetic, even for large values of $s$. We perform theoretical analysis to bound the computation and communication costs of the newly designed variants, up to leading order. Finally, we develop high performance implementations written in C and MPI and present scaling experiments performed on a Cray EX cluster. The new $s$-step variants achieved strong scaling speedups of up to $9.8\times$ over existing methods using up to $512$ cores.

翻译：对偶坐标下降（DCD）与块对偶坐标下降（BDCD）是求解凸优化问题的重要迭代方法。本文针对核支持向量机（K-SVM）与核岭回归（K-RR）问题，提出了可扩展的DCD与BDCD方法。在分布式内存并行计算环境中，这些方法的可扩展性受限于每轮迭代所需的通信开销。在现代硬件架构中，通信成本比计算成本高出数个数量级，导致DCD与BDCD方法的运行时间主要受通信开销支配。为突破此通信瓶颈，我们分别推导出求解K-SVM与K-RR问题的$s$步DCD与$s$步BDCD变体算法。该$s$步变体算法通过可调参数$s$降低通信频率，代价是增加额外带宽与计算开销。在精确算术运算下，$s$步变体算法与现有方法具有相同的解。数值实验表明，即使在大$s$值情况下，$s$步变体算法在有限精度算术运算中仍保持数值稳定性。我们通过理论分析界定了新设计变体算法的计算与通信成本（精确至主阶项）。最后，我们开发了基于C语言与MPI的高性能实现，并在Cray EX集群上进行了扩展性实验。实验表明，在使用多达512个核心时，新型$s$步变体算法相比现有方法实现了最高达$9.8$倍的强扩展加速比。