Grokking is the delayed onset of generalization in neural networks, arising long after they fit the training data. Whether the weight norm causes this delay is disputed: some studies report a critical norm at the transition, others observe grokking with no fixed norm at all. We settle this by intervening on the norm during training rather than only observing it. Under free training with weight decay, networks grok when the weight norm reaches a value Wc that varies little across seeds and learning rates (CV 1 to 2 percent) and grows with the modular base as a power law. When we instead clamp the norm to a fixed multiple rho of Wc and hold it there, the network still groks, but the delay follows T_grok proportional to exp(alpha rho). One exponent, alpha near 7.5, fits this delay across four moduli (R^2 = 0.996). Over the swept ranges the held norm moves the delay by about 19x and the learning rate by only about 2x, and holding the norm above Wc slows grokking rather than preventing it. A final LayerNorm removes the dependence by decoupling weight scale from the network function; without it the exponential law returns. This pinned-norm delay is the exponential counterpart to the logarithmic delay predicted for a freely contracting norm.

翻译：“顿悟”是神经网络中泛化能力的延迟涌现现象，会在网络拟合训练数据很久之后才出现。权重范数是否导致这种延迟存在争议：一些研究报告了转变时刻存在临界范数，而另一些观察则表明网络“顿悟”时根本不存在固定范数。我们通过干预训练过程中的范数（而非仅观察范数）解决了这一争议。在自由训练（含权重衰减）条件下，当权重范数达到某个值Wc时，网络会“顿悟”，该值在不同随机种子和学习率下变化很小（变异系数1%至2%），并随模数基底呈幂律增长。当我们转而将范数钳制为Wc的固定倍数ρ并保持该值时，网络仍会“顿悟”，但延迟时间T_grok与exp(α ρ)成正比。一个指数α（约7.5）在四个模数下拟合了此延迟（R²=0.996）。在所扫描范围内，固定范数使延迟变化约19倍，而学习率仅使其变化约2倍；将范数保持在Wc以上会减缓而非阻止“顿悟”。增加最终层归一化可通过解耦权重尺度与网络功能消除该依赖性；若无归一化，指数定律依然成立。这种固定范数下的延迟是指数形式的，与自由收缩范数所预测的对数延迟形成对应。