Kernel ridge regression, KRR, is a generalization of linear ridge regression that is non-linear in the data, but linear in the parameters. Here, we introduce an equivalent formulation of the objective function of KRR, opening up both for using penalties other than the ridge penalty and for studying kernel ridge regression from the perspective of gradient descent. Using a continuous-time perspective, we derive a closed-form solution for solving kernel regression with gradient descent, something we refer to as kernel gradient flow, KGF, and theoretically bound the differences between KRR and KGF, where, for the latter, regularization is obtained through early stopping. We also generalize KRR by replacing the ridge penalty with the $\ell_1$ and $\ell_\infty$ penalties, respectively, and use the fact that analogous to the similarities between KGF and KRR, $\ell_1$ regularization and forward stagewise regression (also known as coordinate descent), and $\ell_\infty$ regularization and sign gradient descent, follow similar solution paths. We can thus alleviate the need for computationally heavy algorithms based on proximal gradient descent. We show theoretically and empirically how the $\ell_1$ and $\ell_\infty$ penalties, and the corresponding gradient-based optimization algorithms, produce sparse and robust kernel regression solutions, respectively.

翻译：核岭回归（KRR）是线性岭回归的推广，它在数据上呈现非线性，但在参数上保持线性。本文引入核岭回归目标函数的等价形式，既允许使用除岭惩罚之外的其他惩罚项，也支持从梯度下降视角研究核岭回归。通过连续时间视角，我们推导出核岭回归梯度下降求解的闭式解（称为核梯度流，KGF），并从理论上约束核岭回归与核梯度流之间的差异——其中后者通过提前停止实现正则化。我们通过分别用$\ell_1$和$\ell_\infty$惩罚替代岭惩罚推广核岭回归，并利用以下事实：类似于核梯度流与核岭回归的相似性，$\ell_1$正则化与前向逐步回归（也称坐标下降）、$\ell_\infty$正则化与符号梯度下降遵循相似的解路径。由此可规避对基于近端梯度下降的高计算复杂度算法的需求。我们从理论和实证两方面证明，$\ell_1$和$\ell_\infty$惩罚及其对应的基于梯度的优化算法分别能够产生稀疏和鲁棒的核回归解。