The Kaczmarz method (KZ) and its variants, which are types of stochastic gradient descent (SGD) methods, have been extensively studied due to their simplicity and efficiency in solving linear equation systems. The iterative thresholding (IHT) method has gained popularity in various research fields, including compressed sensing or sparse linear regression, machine learning with additional structure, and optimization with nonconvex constraints. Recently, a hybrid method called Kaczmarz-based IHT (KZIHT) has been proposed, combining the benefits of both approaches, but its theoretical guarantees are missing. In this paper, we provide the first theoretical convergence guarantees for KZIHT by showing that it converges linearly to the solution of a system with sparsity constraints up to optimal statistical bias when the reshuffling data sampling scheme is used. We also propose the Kaczmarz with periodic thresholding (KZPT) method, which generalizes KZIHT by applying the thresholding operation for every certain number of KZ iterations and by employing two different types of step sizes. We establish a linear convergence guarantee for KZPT for randomly subsampled bounded orthonormal systems (BOS) and mean-zero isotropic sub-Gaussian random matrices, which are most commonly used models in compressed sensing, dimension reduction, matrix sketching, and many inverse problems in neural networks. Our analysis shows that KZPT with an optimal thresholding period outperforms KZIHT. To support our theory, we include several numerical experiments.

翻译：Kaczmarz方法（KZ）及其变体作为随机梯度下降（SGD）方法的一种，因其在求解线性方程组中的简洁性和高效性而得到广泛研究。迭代阈值法（IHT）在压缩感知或稀疏线性回归、带附加结构的机器学习以及非凸约束优化等多个研究领域逐渐流行。最近，一种名为基于Kaczmarz的迭代阈值法（KZIHT）的混合方法被提出，它结合了两种方法的优势，但缺乏理论保证。本文首次为KZIHT提供了理论收敛保证，证明了当采用重排数据采样方案时，该方法能线性收敛到具有稀疏约束的方程组的解，并达到最优统计偏差。我们还提出了周期性阈值Kaczmarz方法（KZPT），该方法通过每进行若干次KZ迭代后执行阈值操作，并采用两种不同类型的步长，从而推广了KZIHT。对于随机子采样有界正交系统（BOS）和零均值各向同性次高斯随机矩阵——这些是压缩感知、降维、矩阵草图以及神经网络中众多逆问题中最常用的模型——我们建立了KZPT的线性收敛保证。分析表明，采用最优阈值周期的KZPT优于KZIHT。为支持理论，我们进行了多项数值实验。