Self-attention selects information freely across the sequence, but across depth, Transformers merely add each layer's output to the residual stream, so later layers cannot selectively reuse earlier-layer representations. Recent cross-layer methods improve this flow but operate on hidden states outside attention, adding state beyond the key-value cache at inference--a cost that becomes increasingly salient as modern LLMs compress the cache with grouped-query and multi-head latent attention. We introduce Depth-Attention, which performs this selection inside the attention module itself: before a layer attends over the sequence, its query attends over the keys of earlier layers at the same token position and mixes their values into the value that self-attention then reads. Because Depth-Attention reuses the standard attention queries, keys, and value-cache slots, storing depth-mixed values in place of the original values, it adds no parameters and introduces no persistent inference state beyond the standard key-value cache--the same cache size as a vanilla decoder and less than hidden-state-based cross-layer methods. On Qwen3-style decoders at 1.5B and 3B parameters, Depth-Attention attains the lowest perplexity and the highest average downstream accuracy, improving over the vanilla Transformer by up to 2.3 accuracy points and surpassing strong cross-layer baselines in perplexity and average accuracy, while adding under 0.01% extra arithmetic FLOPs and no additional persistent inference state. The gains hold from 360M to 3B parameters and extend to looped Transformers.

翻译：自注意力机制使得信息可以在序列各位置间自由选择，但在深度维度上，Transformer仅将每层输出添加到残差流中，导致后续层无法选择性重用先前层的表示。近期跨层方法虽改善了信息流动，但其操作对象为注意力外的隐藏状态，从而在推理时向键值缓存之外引入了额外状态——随着现代大语言模型采用分组查询和多头潜注意力压缩缓存，这一开销日益显著。我们提出深度注意力（Depth-Attention），在注意力模块内部实现选择性交互：当某层对序列执行注意力计算前，其查询（query）会与同位置更早层的键（key）进行匹配，并将这些层的值（value）混合至自注意力即将读取的值中。由于深度注意力复用标准注意力的查询、键以及值缓存槽位，并通过原位存储深度混合值取代原始值，因此既不增加参数，也不会在标准键值缓存之外引入持久化推理状态——其缓存大小与原生解码器相同，且小于基于隐藏状态的跨层方法。在基于Qwen3架构的1.5B和3B参数解码器上，深度注意力实现了最低困惑度与最高平均下游任务准确率，相比原始Transformer提升高达2.3个准确率百分点，同时在困惑度与平均准确率上超越强跨层基线方法。其带来的额外计算量不足0.01%算术FLOPs，且不增加持久化推理状态。该优势在360M至3B参数范围内保持稳定，并可扩展至循环Transformer架构。