In this paper, we design a regularization-free algorithm for high-dimensional support vector machines (SVMs) by integrating over-parameterization with Nesterov's smoothing method, and provide theoretical guarantees for the induced implicit regularization phenomenon. In particular, we construct an over-parameterized hinge loss function and estimate the true parameters by leveraging regularization-free gradient descent on this loss function. The utilization of Nesterov's method enhances the computational efficiency of our algorithm, especially in terms of determining the stopping criterion and reducing computational complexity. With appropriate choices of initialization, step size, and smoothness parameter, we demonstrate that unregularized gradient descent achieves a near-oracle statistical convergence rate. Additionally, we verify our theoretical findings through a variety of numerical experiments and compare the proposed method with explicit regularization. Our results illustrate the advantages of employing implicit regularization via gradient descent in conjunction with over-parameterization in sparse SVMs.

翻译：本文通过将过参数化与Nesterov光滑方法相结合，设计了一种适用于高维支持向量机(SVM)的无正则化算法，并为其诱导的隐式正则化现象提供了理论保证。具体而言，我们构建了一个过参数化的铰链损失函数，并利用该损失函数上的无正则化梯度下降法估计真实参数。Nesterov方法的使用提升了算法的计算效率，尤其在确定停止准则和降低计算复杂度方面。通过合理选择初始化参数、步长和光滑参数，我们证明了无正则化梯度下降能够达到近乎最优的统计收敛速度。此外，我们通过一系列数值实验验证了理论结果，并将所提方法与显式正则化进行了比较。实验结果揭示了在稀疏SVM中结合过参数化采用梯度下降隐式正则化的优势。