We study the gradients of a maxout network with respect to inputs and parameters and obtain bounds for the moments depending on the architecture and the parameter distribution. We observe that the distribution of the input-output Jacobian depends on the input, which complicates a stable parameter initialization. Based on the moments of the gradients, we formulate parameter initialization strategies that avoid vanishing and exploding gradients in wide networks. Experiments with deep fully-connected and convolutional networks show that this strategy improves SGD and Adam training of deep maxout networks. In addition, we obtain refined bounds on the expected number of linear regions, results on the expected curve length distortion, and results on the NTK.

翻译：我们研究了maxout网络关于输入和参数的梯度，获得了依赖于网络架构和参数分布的矩估计。观察到输入-输出雅可比矩阵的分布取决于输入本身，这给稳定的参数初始化带来了困难。基于梯度的矩估计，我们提出了避免宽网络中梯度消失和爆炸的参数初始化策略。在深度全连接网络和卷积网络上的实验表明，该策略能够改进深度maxout网络的SGD和Adam训练。此外，我们还获得了线性区域期望数量的精细上界、期望曲线长度畸变的结果，以及神经正切核（NTK）的相关结论。