This paper presents Sparse Gradient Descent as a solution for variable selection in convex piecewise linear regression, where the model is given as the maximum of $k$-affine functions $ x \mapsto \max_{j \in [k]} \langle a_j^\star, x \rangle + b_j^\star$ for $j = 1,\dots,k$. Here, $\{ a_j^\star\}_{j=1}^k$ and $\{b_j^\star\}_{j=1}^k$ denote the ground-truth weight vectors and intercepts. A non-asymptotic local convergence analysis is provided for Sp-GD under sub-Gaussian noise when the covariate distribution satisfies the sub-Gaussianity and anti-concentration properties. When the model order and parameters are fixed, Sp-GD provides an $ε$-accurate estimate given $\mathcal{O}(\max(ε^{-2}σ_z^2,1)s\log(d/s))$ observations where $σ_z^2$ denotes the noise variance. This also implies the exact parameter recovery by Sp-GD from $\mathcal{O}(s\log(d/s))$ noise-free observations. The proposed initialization scheme uses sparse principal component analysis to estimate the subspace spanned by $\{ a_j^\star\}_{j=1}^k$, then applies an $r$-covering search to estimate the model parameters. A non-asymptotic analysis is presented for this initialization scheme when the covariates and noise samples follow Gaussian distributions. When the model order and parameters are fixed, this initialization scheme provides an $ε$-accurate estimate given $\mathcal{O}(ε^{-2}\max(σ_z^4,σ_z^2,1)s^2\log^4(d))$ observations. A new transformation named Real Maslov Dequantization (RMD) is proposed to transform sparse generalized polynomials into sparse max-affine models. The error decay rate of RMD is shown to be exponentially small in its temperature parameter. Furthermore, theoretical guarantees for Sp-GD are extended to the bounded noise model induced by RMD. Numerical Monte Carlo results corroborate theoretical findings for Sp-GD and the initialization scheme.

翻译：本文提出稀疏梯度下降法作为凸分段线性回归中变量选择的解决方案，其中模型表示为$k$个仿射函数的最大值：$ x \mapsto \max_{j \in [k]} \langle a_j^\star, x \rangle + b_j^\star$，$j = 1,\dots,k$。这里$\{ a_j^\star\}_{j=1}^k$和$\{b_j^\star\}_{j=1}^k$分别表示真实权重向量和截距项。在协变量分布满足次高斯性和反集中性质且噪声服从次高斯分布的条件下，本文提供了Sp-GD的非渐近局部收敛性分析。当模型阶次和参数固定时，Sp-GD在给定$\mathcal{O}(\max(ε^{-2}σ_z^2,1)s\log(d/s))$个观测值时可获得$ε$精度估计，其中$σ_z^2$表示噪声方差。这同时表明Sp-GD在无噪声场景下可通过$\mathcal{O}(s\log(d/s))$个观测值实现精确参数恢复。所提出的初始化方案通过稀疏主成分分析估计$\{ a_j^\star\}_{j=1}^k$张成子空间，然后利用$r$覆盖搜索估计模型参数。本文在协变量和噪声样本服从高斯分布时给出了该初始化方案的非渐近分析。当模型阶次和参数固定时，该初始化方案在$\mathcal{O}(ε^{-2}\max(σ_z^4,σ_z^2,1)s^2\log^4(d))$个观测值下可获得$ε$精度估计。本文提出名为实数Maslov逆量子化(RMD)的新变换，可将稀疏广义多项式转化为稀疏最大仿射模型。RMD的误差衰减率在其温度参数上呈指数级减小。此外，Sp-GD的理论保证被扩展至RMD诱导的有界噪声模型。数值蒙特卡洛结果验证了Sp-GD及初始化方案的理论发现。