Kernel ridge regression, KRR, is a non-linear generalization of linear ridge regression. Here, we introduce an equivalent formulation of the objective function of KRR, opening up both for using other penalties than the ridge penalty and for studying kernel ridge regression from the perspective of gradient descent. Using a continuous-time perspective, we derive a closed-form solution, kernel gradient flow, KGF, with regularization through early stopping, which allows us to theoretically bound the differences between KGF and KRR. We generalize KRR by replacing the ridge penalty with the $\ell_1$ and $\ell_\infty$ penalties and utilize the fact that analogously to the similarities between KGF and KRR, the solutions obtained when using these penalties are very similar to those obtained from forward stagewise regression (also known as coordinate descent) and sign gradient descent in combination with early stopping. Thus the need for computationally heavy proximal gradient descent algorithms can be alleviated. We show theoretically and empirically how these penalties, and corresponding gradient-based optimization algorithms, produce signal-driven and robust regression solutions, respectively. We also investigate kernel gradient descent where the kernel is allowed to change during training, and theoretically address the effects this has on generalization. Based on our findings, we propose an update scheme for the bandwidth of translational-invariant kernels, where we let the bandwidth decrease to zero during training, thus circumventing the need for hyper-parameter selection. We demonstrate on real and synthetic data how decreasing the bandwidth during training outperforms using a constant bandwidth, selected by cross-validation and marginal likelihood maximization. We also show that using a decreasing bandwidth, we are able to achieve both zero training error and a double descent behavior.

翻译：核岭回归（KRR）是线性岭回归的非线性推广。本文引入KRR目标函数的等价形式，既允许使用岭惩罚以外的其他惩罚项，也为从梯度下降视角研究核岭回归开辟了新途径。通过连续时间视角，我们推导出带早停正则化的闭式解——核梯度流（KGF），并由此在理论上界定了KGF与KRR的差异。通过将岭惩罚替换为ℓ₁和ℓ∞惩罚，我们推广了KRR，并利用与KGF和KRR相似性类似的事实：使用这些惩罚项获得的解与前向逐步回归（也称为坐标下降）及结合早停的符号梯度下降所得解高度相似，从而缓解了对计算量大的近端梯度下降算法的需求。我们从理论和实验两方面证明了这些惩罚项及相应的基于梯度的优化算法分别如何产生信号驱动型回归解和稳健回归解。我们还研究了在训练过程中允许核函数变化的核梯度下降，并从理论上分析了这种变化对泛化性能的影响。基于研究发现，我们提出了一种平移不变核带宽的更新方案：在训练过程中使带宽逐渐减小至零，从而避免了超参数选择问题。在真实和合成数据上的实验表明，与通过交叉验证和边际似然最大化选择的固定带宽相比，训练过程中递减带宽的方法性能更优。我们还证明，采用递减带宽可实现零训练误差与双重下降现象。