Coordinate Descent (CD) methods have gained significant attention in machine learning due to their effectiveness in solving high-dimensional problems and their ability to decompose complex optimization tasks. However, classical CD methods were neither designed nor analyzed with data privacy in mind, a critical concern when handling sensitive information. This has led to the development of differentially private CD methods, such as DP-CD (Differentially Private Coordinate Descent) proposed by Mangold et al. (ICML 2022), yet a disparity remains between non-private CD and DP-CD methods. In our work, we propose a differentially private random block coordinate descent method that selects multiple coordinates with varying probabilities in each iteration using sketch matrices. Our algorithm generalizes both DP-CD and the classical DP-SGD (Differentially Private Stochastic Gradient Descent), while preserving the same utility guarantees. Furthermore, we demonstrate that better utility can be achieved through importance sampling, as our method takes advantage of the heterogeneity in coordinate-wise smoothness constants, leading to improved convergence rates.

翻译：坐标下降法因其在解决高维问题上的有效性及分解复杂优化任务的能力，在机器学习领域获得了广泛关注。然而，经典坐标下降法在设计之初并未考虑数据隐私性这一处理敏感信息时的关键问题。这推动了差分隐私坐标下降法的发展，例如Mangold等人（ICML 2022）提出的DP-CD方法，但非隐私坐标下降法与差分隐私坐标下降法之间仍存在性能差距。本研究提出一种差分隐私随机块坐标下降法，该方法通过草图矩阵在每次迭代中以不同概率选择多个坐标。我们的算法同时推广了DP-CD与经典DP-SGD方法，并保持了相同的效用保证。此外，我们证明通过重要性采样可获得更优的效用，因为该方法利用了坐标方向平滑常数的异质性，从而实现了更快的收敛速率。