We consider the well-studied problem of learning a linear combination of $k$ ReLU activations with respect to a Gaussian distribution on inputs in $d$ dimensions. We give the first polynomial-time algorithm that succeeds whenever $k$ is a constant. All prior polynomial-time learners require additional assumptions on the network, such as positive combining coefficients or the matrix of hidden weight vectors being well-conditioned. Our approach is based on analyzing random contractions of higher-order moment tensors. We use a multi-scale analysis to argue that sufficiently close neurons can be collapsed together, sidestepping the conditioning issues present in prior work. This allows us to design an iterative procedure to discover individual neurons.

翻译：我们研究了在$d$维输入服从高斯分布的条件下，学习$k$个ReLU激活函数的线性组合这一已被充分研究的问题。我们首次给出了当$k$为常数时总能成功的多项式时间算法。所有先前的多项式时间学习器都需要对网络施加额外假设，例如正组合系数或隐层权重向量矩阵具有良好条件数。我们的方法基于分析高阶矩张量的随机收缩。通过使用多尺度分析，我们论证了足够接近的神经元可以被合并，从而规避了先前工作中存在的条件数问题。这使得我们能够设计一种迭代过程来发现单个神经元。