Understanding the fundamental principles behind the success of deep neural networks is one of the most important open questions in the current literature. To this end, we study the training problem of deep neural networks and introduce an analytic approach to unveil hidden convexity in the optimization landscape. We consider a deep parallel ReLU network architecture, which also includes standard deep networks and ResNets as its special cases. We then show that pathwise regularized training problems can be represented as an exact convex optimization problem. We further prove that the equivalent convex problem is regularized via a group sparsity inducing norm. Thus, a path regularized parallel ReLU network can be viewed as a parsimonious convex model in high dimensions. More importantly, since the original training problem may not be trainable in polynomial-time, we propose an approximate algorithm with a fully polynomial-time complexity in all data dimensions. Then, we prove strong global optimality guarantees for this algorithm. We also provide experiments corroborating our theory.

翻译：理解深度神经网络成功背后的基本原理是当前文献中最重要的开放性问题之一。为此，我们研究深度神经网络的训练问题，并引入一种分析方法来揭示优化景观中的隐藏凸性。我们考虑一种深度并行ReLU网络架构，该架构也将标准深度网络和ResNet作为其特例。然后我们证明，路径正则化训练问题可以表示为精确的凸优化问题。我们进一步证明，等价的凸问题通过群稀疏诱导范数进行正则化。因此，路径正则化的并行ReLU网络在高维中可以视为一种简约的凸模型。更重要的是，由于原始训练问题可能无法在多项式时间内可训练，我们提出了一种在所有数据维度上具有完全多项式时间复杂度的近似算法。然后，我们证明该算法具有强全局最优性保证。我们还提供了验证我们理论的实验。