The edge-of-chaos dynamics of wide randomly initialized low-rank feedforward networks are analyzed. Formulae for the optimal weight and bias variances are extended from the full-rank to low-rank setting and are shown to follow from multiplicative scaling. The principle second order effect, the variance of the input-output Jacobian, is derived and shown to increase as the rank to width ratio decreases. These results inform practitioners how to randomly initialize feedforward networks with a reduced number of learnable parameters while in the same ambient dimension, allowing reductions in the computational cost and memory constraints of the associated network.

翻译：宽随机初始化低秩前馈网络的边缘混沌动力学被分析。权重和偏差最优方差的公式从满秩场景扩展到低秩设置，并证明其遵循乘法缩放。主要二阶效应——输入-输出雅可比矩阵的方差——被推导出，并证明其随秩宽比减小而增加。这些结果指导从业者如何在保持相同环境维度的条件下，随机初始化具有减少可学习参数数量的前馈网络，从而降低相关网络的计算成本和内存约束。