We track the developmental trajectory of attention-head circuit formation across three 1B-class language models spanning two architecture families (dense transformer, mixture-of-experts) and two pretraining corpora (The Pile, DCLM): Pythia 1B, OLMo 1B-0724-hf, and OLMoE 1B-7B-0924. At each of 10 log-spaced revisions per model -- 30 mechanistic-interpretability runs in total -- we apply a participation-ratio (PR) spectral signal and an all-head capability-specific selectivity screen to track induction, previous-token, and BOS-attractor heads as they emerge. Five findings. (F1) Layers 0 and 1 produce zero BOS-classified heads at every revision in every model: the L0/L1 zero-BOS floor is an architectural property, not a learned outcome. (F2) The whole-model BOS-attractor fraction follows three distinct emergence shapes -- a gradual ramp in Pythia 1B, a sharp phase transition in OLMo 1B (7% to 70% between adjacent checkpoints), and a gradual ramp in OLMoE 1B-7B. (F3) In DCLM models, induction-circuit formation precedes BOS-attractor formation by 10-20x in tokens; capability-circuit formation and attention-sink formation are two transitions, not one. (F4) The capability-specific screen converges to the final induction circuit within 0.3-2% of total training tokens -- circuit identification does not require the final model. (F5) For every final-checkpoint induction head sampled across all three models, per-head PR is elevated at or before the first revision at which that head crosses its capability-selectivity threshold. The results refine the induction-phase-transition framing: in 1B-class models trained on DCLM, the induction transition and the attention-sink transition are separated by an order of magnitude in tokens and have qualitatively different shapes.

翻译：我们追踪了三种1B级语言模型中注意头电路形成的发展轨迹，这些模型涵盖两个架构家族（密集Transformer、混合专家模型）和两个预训练语料库（The Pile、DCLM）：Pythia 1B、OLMo 1B-0724-hf和OLMoE 1B-7B-0924。在每种模型的10个对数间隔检查点（总共30次机械可解释性运行）上，我们应用参与比（PR）频谱信号和全头能力特定选择性筛查，以追踪归纳头、前词头和BOS吸引头在涌现过程中的变化。研究发现如下：（F1）在每种模型的每个修订版本中，第0层和第1层均未产生任何BOS分类头：L0/L1零BOS层是架构属性，而非学习结果。（F2）整体模型的BOS吸引头比例呈现三种不同的涌现形态——Pythia 1B中的渐进斜坡、OLMo 1B中的陡峭相变（相邻检查点间从7%跃升至70%），以及OLMoE 1B-7B中的渐进斜坡。（F3）在DCLM模型中，归纳电路形成比BOS吸引形成早10-20倍（按词元计）；能力电路形成与注意力汇聚形成是两个不同的转变，而非单一过程。（F4）能力特定筛查在总训练词元的0.3%-2%内收敛至最终归纳电路——电路识别无需依赖于最终模型。（F5）对于所有三个模型中采样的每个最终检查点归纳头，其单个头PR值在头跨越能力选择性阈值的第一个修订版本时或之前即已升高。这些结果细化了归纳相变框架：在基于DCLM训练的1B级模型中，归纳转变与注意力汇聚转变按词元计相差一个数量级，且呈现定性不同的形态。