We establish that randomly initialized neural networks, with large width and a natural choice of hyperparameters, have nearly independent outputs exactly when their activation function is nonlinear with zero mean under the Gaussian measure: $\mathbb{E}_{z \sim \mathcal{N}(0,1)}[σ(z)]=0$. For example, this includes ReLU and GeLU with an additive shift, as well as tanh, but not ReLU or GeLU by themselves. Because of their nearly independent outputs, we propose neural networks with zero-mean activation functions as a promising candidate for the Alignment Research Center's computational no-coincidence conjecture -- a conjecture that aims to measure the limits of AI interpretability.

翻译：我们证明，随机初始化的神经网络在宽度较大且超参数选择自然的情况下，当其激活函数在高斯测度下具有零均值非线性特性时，其输出近乎相互独立：$\mathbb{E}_{z \sim \mathcal{N}(0,1)}[σ(z)]=0$。例如，这包括带有附加偏移的ReLU和GeLU，以及tanh函数，但不包括原始ReLU或GeLU。基于其近乎独立的输出特性，我们提出采用零均值激活函数的神经网络作为对齐研究中心"计算无巧合猜想"的理想候选模型——该猜想旨在衡量人工智能可解释性的理论极限。