Kernel ridge regression, KRR, is a generalization of linear ridge regression that is non-linear in the data, but linear in the parameters. Here, we introduce an equivalent formulation of the objective function of KRR, opening up both for using penalties other than the ridge penalty and for studying kernel ridge regression from the perspective of gradient descent. Using a continuous-time perspective, we derive a closed-form solution for solving kernel regression with gradient descent, something we refer to as kernel gradient flow, KGF, and theoretically bound the differences between KRR and KGF, where, for the latter, regularization is obtained through early stopping. We also generalize KRR by replacing the ridge penalty with the $\ell_1$ and $\ell_\infty$ penalties, respectively, and use the fact that analogous to the similarities between KGF and KRR, $\ell_1$ regularization and forward stagewise regression (also known as coordinate descent), and $\ell_\infty$ regularization and sign gradient descent, follow similar solution paths. We can thus alleviate the need for computationally heavy algorithms based on proximal gradient descent. We show theoretically and empirically how the $\ell_1$ and $\ell_\infty$ penalties, and the corresponding gradient-based optimization algorithms, produce sparse and robust kernel regression solutions, respectively.

翻译：核岭回归（KRR）是线性岭回归的推广，它在数据上呈现非线性，但在参数上保持线性。本文提出核岭回归目标函数的等价形式，既允许使用岭惩罚以外的其他惩罚项，也为从梯度下降视角研究核岭回归提供了可能。通过连续时间视角，我们推导出采用梯度下降求解核回归的闭式解（称为核梯度流KGF），并从理论上界定了KRR与KGF的差异——其中后者通过早停实现正则化。我们进一步推广KRR，分别用$\ell_1$和$\ell_\infty$惩罚替换岭惩罚，并利用以下事实：类似于KGF与KRR的相似性，$\ell_1$正则化与前向逐步回归（即坐标下降法）、$\ell_\infty$正则化与符号梯度下降法遵循相似的解路径。因此，我们可避免基于近端梯度下降的计算密集型算法。理论分析与实验证明，$\ell_1$和$\ell_\infty$惩罚及其对应的梯度优化方法，能够分别产生稀疏性和鲁棒性的核回归解。