Large Language Models (LLMs) are trained with next-token prediction, implemented in autoregressive Transformers via causal masking for parallelism. This creates a subtle misalignment: residual connections tie activations to the current token, while supervision targets the next token, potentially propagating mismatched information if the current token is not the most informative for prediction. In this work, we empirically localize this input-output alignment shift in pretrained LLMs, using decoding trajectories over tied embedding spaces and similarity-based metrics. Our experiments reveal that the hidden token representations switch from input alignment to output alignment deep within the network. Motivated by this observation, we propose a lightweight residual-path mitigation based on residual attenuation, implemented either as a fixed-layer intervention or as a learnable gating mechanism. Experiments on multiple benchmarks show that these strategies alleviate the representation misalignment and yield improvements, providing an efficient and general architectural enhancement for autoregressive Transformers.

翻译：大型语言模型（LLMs）通过下一词预测进行训练，在自回归Transformer中通过因果掩码实现并行化。这导致了一种微妙的错位：残差连接将激活值与当前词元绑定，而监督目标针对下一词元，若当前词元对预测并非最具信息性，则可能传播不匹配的信息。在本工作中，我们利用绑定嵌入空间上的解码轨迹和基于相似性的度量，对预训练LLMs中的这种输入-输出对齐偏移进行了实证定位。实验表明，隐藏词元表征在网络深层从输入对齐切换为输出对齐。受此观察启发，我们提出一种基于残差衰减的轻量级残差路径缓解方法，可通过固定层干预或可学习门控机制实现。在多个基准测试上的实验表明，这些策略能缓解表征错位并带来性能提升，为自回归Transformer提供了一种高效且通用的架构增强方案。