LASSO regularization is a popular regression tool to enhance the prediction accuracy of statistical models by performing variable selection through the $\ell_1$ penalty, initially formulated for the linear model and its variants. In this paper, the territory of LASSO is extended to two-layer ReLU neural networks, a fashionable and powerful nonlinear regression model. Specifically, given a neural network whose output $y$ depends only on a small subset of input $\boldsymbol{x}$, denoted by $\mathcal{S}^{\star}$, we prove that the LASSO estimator can stably reconstruct the neural network and identify $\mathcal{S}^{\star}$ when the number of samples scales logarithmically with the input dimension. This challenging regime has been well understood for linear models while barely studied for neural networks. Our theory lies in an extended Restricted Isometry Property (RIP)-based analysis framework for two-layer ReLU neural networks, which may be of independent interest to other LASSO or neural network settings. Based on the result, we advocate a neural network-based variable selection method. Experiments on simulated and real-world datasets show promising performance of the variable selection approach compared with existing techniques.

翻译：LASSO正则化是一种流行的回归工具，通过ℓ1罚函数进行变量选择以提升统计模型的预测精度，最初针对线性模型及其变种提出。本文将LASSO的应用范围扩展至两层ReLU神经网络——一种时尚且强大的非线性回归模型。具体而言，给定一个输出y仅依赖于输入x的某个小规模子集（记为S*）的神经网络，我们证明当样本数量随输入维度呈对数增长时，LASSO估计量能够稳定重构该神经网络并识别出S*。这一具有挑战性的场景在线性模型中已得到充分理解，但在神经网络中鲜有研究。我们的理论基于扩展的受限等距性质（RIP）分析框架，适用于两层ReLU神经网络，该框架可能对LASSO或神经网络的其他设定具有独立研究价值。基于此结果，我们提出一种基于神经网络的变量选择方法。在模拟和真实数据集上的实验表明，与现有技术相比，该变量选择方法展现出有前景的性能。