Despite a known dependence between grokking and model parameter count, how model capacity shapes grokking remains an open question. Focusing on small Transformers and modular arithmetic, we give a quantitative account of the relationship between model capacity and grokking as a competition between two timescales: generalisation ($T_{\text{gen}}$) and memorisation ($T_{\text{mem}}$). Building on prior formalisms for the empirical information capacity of Transformers, we define and measure $T_{\text{gen}}$ on modular arithmetic datasets and $T_{\text{mem}}$ on random-label datasets of equivalent complexity, and show that their intersection correlates with model sizes where consistent grokking begins. We also observe empirical laws for $T_{\text{gen}}$ and $T_{\text{mem}}$ as models and datasets scale, motivating their forms theoretically and finding that our model of $T_{\text{mem}}$ is consistent with prior observations that larger models tend to memorise faster. Moreover, extrapolating our empirical timescale laws predicted the existence of datasets for which no model of any size would consistently grok, a pattern supported on the modular arithmetic datasets of larger primes, demonstrating that our framework can make qualitative predictions out-of-distribution.
翻译:暂无翻译