Language models often show a preference for using information from specific positions in the input regardless of semantic relevance. While positional bias has been studied in various contexts, from attention sinks to task performance degradation in long-context settings, prior work has not established how these biases evolve across individual layers and input positions, or how they vary independent of task complexity. We introduce an attribution-based framework to analyze positional effects in short-context language modeling. Using layer conductance with a sliding-window approach, we quantify how each layer distributes importance across input positions, yielding layer-wise positional importance profiles. We find that these profiles are architecture-specific, stable across inputs, and invariant to lexical scrambling. Characterizing these profiles, we find prominent recency bias that increases with depth and subtle primacy bias that diminishes through model depth. Beyond positional structure, we also show that early layers preferentially weight content words over function words across all positions, while later layers lose this word-type differentiation.

翻译：语言模型往往倾向于使用输入中特定位置的信息，而无论其语义相关性如何。尽管位置偏置已在多种情境中得到研究——从注意力沉没到长上下文设置中的任务性能退化——但先前工作尚未阐明这些偏置如何随各独立层和输入位置演变，或如何独立于任务复杂度而变化。我们引入了一种基于归因的框架来分析短上下文语言建模中的位置效应。通过采用滑动窗口方法的层传导度，我们量化了每一层如何在输入位置间分配重要性，从而得到逐层位置重要性分布图。我们发现这些分布图具有架构特异性，在不同输入间保持稳定，且对词汇置乱具有不变性。通过刻画这些分布图，我们观察到显著的近因偏置随深度增加而增强，以及微弱的首因偏置随模型深度增加而减弱。除了位置结构之外，我们还发现早期层在所有位置上均优先加权实词而非虚词，而后期层则丧失了这种词类区分能力。