We study planning site formation in language models -- where internal representations of structurally-constrained future tokens form during the forward pass, and whether they causally drive generation. Using rhyming-couplet completion as a clean test of forward-looking constraint, we apply two lightweight methods (linear probing and activation patching) across Qwen3, Gemma-3, and Llama-3 at more than ten scales. Probing shows that future-rhyme information is linearly decodable at the line boundary, with signal that strengthens with scale in all three families. Activation patching reveals that only Gemma-3-27B causally relies on this encoding, exhibiting a handoff in which the causal driver migrates from the rhyme word to the line boundary around layer 30. Every other model we test conditions on the rhyme word throughout generation, with near-zero causal effect at the line boundary despite strong probe signal. We localize the Gemma-3-27B handoff to five attention heads through two-stage path patching that recover ~90% of the rhyme-routing capacity at the newline.

翻译：摘要：我们研究了语言模型中的规划位点形成机制——即前向传播过程中结构受限的未来词元的内部表征如何形成，以及这些表征是否对生成过程具有因果驱动作用。以押韵对句完成为前向约束的洁净测试任务，我们采用线性探针和激活修补两种轻量级方法，在Qwen3、Gemma-3和Llama-3三个系列共十余种参数规模的模型上展开研究。探针实验表明，未来押韵信息在行边界处呈线性可解码性，且该信号强度随模型规模增大而增强（三个系列均呈现此规律）。激活修补实验揭示，仅Gemma-3-27B模型因果性地依赖该编码，展现出一种关键转交机制：因果驱动者约在第30层从押韵词迁移至行边界。其他所有测试模型在整个生成过程中均以押韵词为条件，尽管行边界处存在强探针信号，但其因果效应近乎为零。通过两阶段路径修补，我们将Gemma-3-27B模型的关键转交定位至五个注意力头，在换行符处恢复了约90%的押韵路由能力。