Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought. Existing sparse attention methods often face a practical efficiency-quality trade-off. Structured block sparse methods typically provide stronger acceleration but incur noticeable quality loss, while token sparse methods are usually more accurate yet deliver limited end-to-end speedup because top-k routing over the full cache remains expensive. In this work, we propose cross-layer sparse attention (CLSA), which is built on top of KV-sharing architectures such as YOCO. The core idea is to share not only the KV cache across cross-decoder layers, but also the routing index. A single indexer computes token-level top-k selection once and reuses the resulting index across layers, thereby preserving the fine-grained selectivity of token sparse attention while amortizing the routing overhead. The resulting architecture improves all major inference bottlenecks jointly, including pre-filling, KV-cache storage, and long-context decoding. Experiments across short-context and long-context benchmarks show that CLSA is both accurate and efficient, achieving up to 7.6x decoding speedup and 17.1x overall throughput improvement at 128K context. These results suggest a more complete architectural solution for long-context LLMs that jointly advances model quality and inference efficiency.

翻译：现代大语言模型中的长上下文推理越来越受到解码效率的制约，尤其在推理密集型场景下，模型会生成长篇中间思维链。现有稀疏注意力方法通常面临实际效率与质量之间的权衡：结构化块稀疏方法虽能提供更强的加速效果，但会带来显著的质量损失；而词元稀疏方法虽通常更精确，但由于在完整缓存上进行top-k路由计算的成本高昂，端到端加速效果有限。本文提出跨层稀疏注意力（CLSA），该架构基于如YOCO等KV共享架构构建。核心思想不仅在于共享跨解码器层的KV缓存，还共享路由索引。单个索引器计算一次词元级top-k选择，并将所得索引在多个层间复用，从而在保持词元稀疏注意力细粒度选择性的同时分摊路由开销。该架构协同优化了所有主要推理瓶颈，包括预填充、KV缓存存储及长上下文解码。短上下文与长上下文基准测试结果表明，CLSA兼具准确性与高效性：在128K上下文长度下，实现最高7.6倍解码加速与17.1倍整体吞吐量提升。这些结果揭示了长上下文大语言模型更完整的架构解决方案，可同步提升模型质量与推理效率。