Why does weight decay work? We prove that, in any fixed-precision regime, the smallest weight norm of a looped neural network outputting a binary string equals the Kolmogorov complexity of that string, up to a logarithmic factor. This implies that weight decay induces a prior matching Solomonoff's universal prior, the optimal prior over computable functions, up to a polynomial factor. The result is norm-agnostic: in fixed precision, every weight norm collapses to the non-zero parameter count up to constants, so the same sandwich bound holds for any norm used as a regulariser. The proof has two short reductions: any program for a universal Turing machine can be encoded into neural weights at unit cost per program bit, and any fixed-precision network can be described by enumerating its non-zero parameters with logarithmic addressing overhead. Both bounds are tight up to constants, with the logarithmic factor realised by permutation encodings: a network whose parameters encode a permutation produces a string whose Kolmogorov complexity is the non-zero parameter count times its logarithm. The fixed-precision assumption is essential: with infinite precision, neural networks can encode non-computable functions and the weight norm loses its relevance.

翻译：权重衰减为何有效？我们证明，在任意固定精度体制下，输出二进制字符串的循环神经网络的权重最小范数，等于该字符串的柯尔莫戈洛夫复杂度，仅差一个对数因子。这意味着权重衰减所诱导的先验，与所罗门诺夫的通用先验（可计算函数的最优先验）相匹配，仅差一个多项式因子。该结果与范数类型无关：在固定精度下，所有权重范数均退化为非零参数计数（仅差常数项），因此任何用作正则化项的范数均满足相同的夹逼界。证明包含两个简洁的归约：通用图灵机的任意程序可以以每个程序比特单位代价编码为神经权重；任何固定精度网络可通过枚举其非零参数并附加对数寻址开销来描述。两个界限在常数意义上都是紧的，其中对数因子通过排列编码实现：参数编码排列的网络，其输出的字符串的柯尔莫戈洛夫复杂度等于非零参数计数乘以该计数的对数。固定精度假设至关重要：在无限精度下，神经网络可编码非可计算函数，此时权重范数失去其相关性。