We describe an algorithm that learns two-layer residual units with rectified linear unit (ReLU) activation: suppose the input $\mathbf{x}$ is from a distribution with support space $\mathbb{R}^d$ and the ground-truth generative model is such a residual unit, given by \[\mathbf{y}= \boldsymbol{B}^\ast\left[\left(\boldsymbol{A}^\ast\mathbf{x}\right)^+ + \mathbf{x}\right]\text{,}\] where ground-truth network parameters $\boldsymbol{A}^\ast \in \mathbb{R}^{d\times d}$ is a nonnegative full-rank matrix and $\boldsymbol{B}^\ast \in \mathbb{R}^{m\times d}$ is full-rank with $m \geq d$ and for $\mathbf{c} \in \mathbb{R}^d$, $[\mathbf{c}^{+}]_i = \max\{0, c_i\}$. We design layer-wise objectives as functionals whose analytic minimizers express the exact ground-truth network in terms of its parameters and nonlinearities. Following this objective landscape, learning residual units from finite samples can be formulated using convex optimization of a nonparametric function: for each layer, we first formulate the corresponding empirical risk minimization (ERM) as a positive semi-definite quadratic program (QP), then we show the solution space of the QP can be equivalently determined by a set of linear inequalities, which can then be efficiently solved by linear programming (LP). We further prove the statistical strong consistency of our algorithm, and demonstrate the robustness and sample efficiency of our algorithm by experiments.

翻译：我们描述一个以校正线性单位( ReLU) 激活来学习两层剩余单位的算法: 假设输入 $\ mathbf{x} $ 是来自支持空间的分布 $\ mathb{R\\ d$, 地面真相基因模型是这样一个剩余单位, 由\\\ mathbf{\\\\ boldsymb{B\\\\\\\\\boldsybol} B\\\\\\\\ left[\ left( boldsymbol{ A} P ⁇ st\\ pathb{x}xright} =+\ mathblationf{xright}\ text{ $\\ text_ glodreauth servol= a nevy- blational- blational_ deal_ maxnational a.