Kernel ridge regression, KRR, is a generalization of linear ridge regression that is non-linear in the data, but linear in the parameters. Here, we introduce an equivalent formulation of the objective function of KRR, opening up both for using penalties other than the ridge penalty and for studying kernel ridge regression from the perspective of gradient descent. Using a continuous-time perspective, we derive a closed-form solution for solving kernel regression with gradient descent, something we refer to as kernel gradient flow, KGF, and theoretically bound the differences between KRR and KGF, where, for the latter, regularization is obtained through early stopping. We also generalize KRR by replacing the ridge penalty with the $\ell_1$ and $\ell_\infty$ penalties, respectively, and use the fact that analogous to the similarities between KGF and KRR, $\ell_1$ regularization and forward stagewise regression (also known as coordinate descent), and $\ell_\infty$ regularization and sign gradient descent, follow similar solution paths. We can thus alleviate the need for computationally heavy algorithms based on proximal gradient descent. We show theoretically and empirically how the $\ell_1$ and $\ell_\infty$ penalties, and the corresponding gradient-based optimization algorithms, produce sparse and robust kernel regression solutions, respectively.

翻译：核岭回归（KRR）是线性岭回归的推广，它在数据上呈非线性，但在参数上呈线性。本文中，我们引入了KRR目标函数的一种等价形式，这使得我们既可以使用岭惩罚以外的其他惩罚方法，也可以从梯度下降的角度研究核岭回归。利用连续时间视角，我们推导出用梯度下降求解核回归的闭式解（称为核梯度流，KGF），并从理论上界定了KRR与KGF之间的差异——对于后者，正则化通过早停实现。我们还将KRR推广至分别使用$\ell_1$和$\ell_\infty$惩罚替代岭惩罚，并利用以下事实：类似于KGF与KRR之间的相似性，$\ell_1$正则化与前向分段回归（亦称坐标下降法）、$\ell_\infty$正则化与符号梯度下降遵循相似的求解路径。因此，我们可避免依赖基于近端梯度下降的高计算量算法。我们从理论和实验两方面证明，$\ell_1$和$\ell_\infty$惩罚及其对应的基于梯度的优化算法，分别能产生稀疏和稳健的核回归解。